Manual

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 282

Download
Open PDF In Browser	View PDF

The Pencil Code:
A High-Order MPI code for MHD Turbulence
User’s and Reference Manual

May 10, 2018
http://www.nordita.org/software/pencil-code/
https://github.com/pencil-code/pencil-code

The P ENCIL C ODE: multi-purpose and multi-user maintained
http://www.nordita.org/~brandenb/talks/misc/PencilCode04.htm

Figure 1: Check-in patterns as a function of time for different subroutines. The different users are marked
by different symbols and different colors.

ii

Contributors to the code
(in inverse alphabetical order according to their user name)
An up to date list of Pencil Code contributors can be found at GitHub.
wladimir.lyra
weezy
wdobler
vpariev
torkel
tavo.buk
thomas.gastine
theine
tarek
sven.bingert
steveb
snod
pkapyla
nils.e.haugen
ngrs
NBabkovskaia
mreinhardt
mkorpi
miikkavaisala
mee
mcmillan
mattias
koenkemel
karlsson
joishi
joern.warnecke
Iomsn1
fadiesis
dorch
boris.dintrans
dhruba.mitra
ccyang
christer
Bourdin.KIS
AxelBrandenburg
apichat
amjed
alex.i.hubbard
michiel.lambrechts
anders
mppiyali

Wladimir Lyra
S. Louise Wilkin
Wolfgang Dobler
Vladimir Pariev
Ulf Torkelsson
Gustavo Guerrero
Thomas Gastine
Tobias (Tobi) Heinemann
Tarek A. Yousef
Sven Bingert
Steve Berukoff
Andrew Snodin
Petri Käpylä
Nils Erland L. Haugen
Graeme R. Sarson
Natalia Babkovskaia
Matthias Rheinhardt
Maarit J. Käpylä (née Korpi, Mantere)
Miikka Väisälä
Antony (tOnY) Mee
David McMillan
Mattias Christensson
Koen Kemel
Torgny Karlsson
Jeff S. Oishi
Jörn Warnecke
Simon Candelaresi
Fabio Del Sordo
Bertil Dorch
Boris Dintrans
Dhrubaditya Mitra
Chao-Chin Yang
Christer Sandin
Philippe Bourdin
Axel Brandenburg
Apichat Neamvonk
Amjed Mohammed
Alex Hubbard
Michiel Lambrechts
Anders Johansen
Piyali Chatterjee

California State University/JPL
University of Newcastle
Potsdam
University of Rochester
Chalmers University
Stanford University
MPI for Solar System Research
IAS Princeton
University of Trondheim
MPI for Solar System Research
UCLA
University of Newcastle
University of Helsinki
SINTEF
University of Newcastle
University of Helsinki
University of Helsinki
University of Helsinki
University of Helsinki
University of Newcastle
York University, Toronto
formerly at Nordita
Nordita, Stockholm
Nordita
Kavli Institute for Particle Astrop
MPI for Solar System Research, L
University of Dundee, Dundee
Nordita, Stockholm
University of Copenhagen
Observatoire Midi-Pyrénées, Tou
Nordita, Stockholm
Lund Observatory
University of Uppsala
MPI for Solar System Research
Nordita
University of Newcastle
University of Oldenburg
Am. Museum Nat. History
Lund Observatory, Lund Univers
Lund Observatory, Lund Univers
University of Oslo

Copyright c 2001–2017 Wolfgang Dobler & Axel Brandenburg
Permission is granted to make and distribute verbatim copies of this manual provided
the copyright notice and this permission notice are preserved on all copies.
iii

Permission is granted to copy and distribute modified versions of this manual under
the conditions for verbatim copying, provided that the entire resulting derived work is
distributed under the terms of a permission notice identical to this one.

iv

License agreement and giving credit
The content of all files under :pserver:$USER@svn.nordita.org:/var/cvs/brandenb are
under the GNU General Public License (http://www.gnu.org/licenses/gpl.html).
We, the P ENCIL C ODE community, ask that in publications and presentations the use of the code (or parts of it) be acknowledged with reference to
the web site http://www.nordita.org/software/pencil-code/ or (equivalently) to.
https://github.com/pencil-code/pencil-code. As a courtesy to the people involved in
developing particularly important parts of the program (use svn annotate src/*.f90 to
find out who did what!) we suggest to give appropriate reference to one or several of the
following (or other appropriate) papers (listed here in temporal order):
Dobler, W., Haugen, N. E. L., Yousef, T. A., & Brandenburg, A.: 2003, “Bottleneck effect in three-dimensional turbulence simulations,” Phys. Rev. E 68, 026304, 1-8
(astro-ph/0303324)
Haugen, N. E. L., Brandenburg, A., & Dobler, W.: 2003, “Is nonhelical hydromagnetic turbulence peaked at small scales?” Astrophys. J. Lett. 597, L141-L144
(astro-ph/0303372)
Brandenburg, A., Käpylä, P., & Mohammed, A.: 2004, “Non-Fickian diffusion and
tau-approximation from numerical turbulence,” Phys. Fluids 16, 1020-1027
(astro-ph/0306521)
Johansen, A., Andersen, A. C., & Brandenburg, A.: 2004, “Simulations of dusttrapping vortices in protoplanetary discs,” Astron. Astrophys. 417, 361-371
(astro-ph/0310059)
Haugen, N. E. L., Brandenburg, A., & Mee, A. J.: 2004, “Mach number dependence
of the onset of dynamo action,” Monthly Notices Roy. Astron. Soc. 353, 947-952
(astro-ph/0405453)
Brandenburg, A., & Multamäki, T.: 2004, “How long can left and right handed life forms
coexist?” Int. J. Astrobiol. 3, 209-219 (q-bio/0407008)
McMillan, D. G., & Sarson, G. R.: 2005, “Dynamo simulations in a spherical shell of
ideal gas using a high-order Cartesian magnetohydrodynamics code,” Phys. Earth
Planet. Int.153, 124-135
Heinemann, T., Dobler, W., Nordlund, Å., & Brandenburg, A.: 2006, “Radiative transfer
in decomposed domains,” Astron. Astrophys. 448, 731-737 (astro-ph/0503510)
Dobler, W., Stix, M., & Brandenburg, A.: 2006, “Convection and magnetic field generation in fully convective spheres,” Astrophys. J. 638, 336-347 (astro-ph/0410645)
Snodin, A. P., Brandenburg, A., Mee, A. J., & Shukurov, A.: 2006, “Simulating fieldaligned diffusion of a cosmic ray gas,” Monthly Notices Roy. Astron. Soc. 373, 643652 (astro-ph/0507176)
Johansen, A., Klahr, H., & Henning, T.: 2006, “Dust sedimentation and self-sustained
Kelvin-Helmholtz turbulence in protoplanetary disc mid-planes,” Astrophys. J.
636, 1121-1134 (astro-ph/0512272)
de Val-Borro, M. and 22 coauthors (incl. Lyra, W.): 2006, “A comparative study
of disc-planet interaction,” Monthly Notices Roy. Astron. Soc. 370, 529-558
(astro-ph/0605237)
Johansen, A., Oishi, J. S., Mac Low, M. M., Klahr, H., Henning, T., & Youdin, A.: 2007,
“Rapid planetesimal formation in turbulent circumstellar disks,” Nature 448,
1022–1025 (arXiv/0708.3890)
Lyra, W., Johansen, A., Klahr, H., & Piskunov, N.: 2008, “Global magnetohydrodynamical models of turbulence in protoplanetary disks I. A cylindrical potential
v

on a Cartesian grid and transport of solids,” Astron. Astrophys. 479, 883-901
(arXiv/0705.4090)
Brandenburg, A., Rädler, K.-H., Rheinhardt, M., & Käpylä, P. J.: 2008, “Magnetic diffusivity tensor and dynamo effects in rotating and shearing turbulence,” Astrophys.
J. 676, 740-751 (arXiv/0710.4059)
Lyra, W., Johansen, A., Klahr, H., & Piskunov, N.: 2008, “Embryos grown in the dead
zone. Assembling the first protoplanetary cores in low-mass selfgravitating circumstellar disks of gas and solids,” Astron. Astrophys. 491, L41-L44
Lyra, W., Johansen, A., Klahr, H., & Piskunov, N.: 2009, “Standing on the shoulders of
giants. Trojan Earths and vortex trapping in low-mass selfgravitating protoplanetary disks of gas and solids,” Astron. Astrophys. 493, 1125-1139
Lyra, W., Johansen, A., Zsom, A., Klahr, H., & Piskunov, N.: 2009, “Planet formation
bursts at the borders of the dead zone in 2D numerical simulations of circumstellar disks,” Astron. Astrophys. 497, 869-888 (arXiv/0901.1638)
Mitra, D., Tavakol, R., Brandenburg, A., & Moss, D.: 2009, “Turbulent dynamos in spherical shell segments of varying geometrical extent,” Astrophys. J. 697, 923-933
(arXiv/0812.3106)
Haugen, N. E. L., & Kragset, S.: 2010, “Particle impaction on a cylinder in a crossflow
as function of Stokes and Reynolds numbers,” J. Fluid Mech. 661, 239-261
Rheinhardt, M., & Brandenburg, A.: 2010, “Test-field method for mean-field coefficients
with MHD background,” Astron. Astrophys. 520, A28 (arXiv/1004.0689)
Babkovskaia, N., Haugen, N. E. L., Brandenburg, A.: 2011, “A high-order public domain
code for direct numerical simulations of turbulent combustion,” J. Comp. Phys.
230, 1-12 (arXiv/1005.5301)
Johansen, A., Klahr, H., & Henning, Th.: 2011, “High-resolution simulations of planetesimal formation in turbulent protoplanetary discs,” Astron. Astrophys. 529, A62
Johansen, A., Youdin, A. N., & Lithwick, Y.: 2012, “Adding particle collisions to the formation of asteroids and Kuiper belt objects via streaming instabilities,” Astron.
Astrophys. 537, A125
Lyra, W. & Kuchner, W. : 2013, “Formation of sharp eccentric rings in debris disks with
gas but without planets,” Nature 499, 184–187
Yang, C.-C., & Johansen, A.: 2016, “Integration of Particle-Gas Systems with Stiff Mutual Drag Interaction,” Astrophys. J. Suppl. Series 224, 39
This list is not always up-to-date. We therefore ask the developers to check in new relevant papers, avoiding however redundancies.

vi

Foreword
This code was originally developed at the Turbulence Summer School of the Helmholtz
Institute in Potsdam (2001). While some SPH and PPM codes for hydrodynamics and
magnetohydrodynamics are publicly available, this does not generally seem to be the
case for higher order finite-difference or spectral codes. Having been approached by people interested in using our code, we decided to make it as flexible as possible and as
user-friendly as seems reasonable, and to put it onto a public CVS repository. Since
21 September 2008 it is distributed via https://github.com/pencil-code/pencil-code.
The code can certainly not be treated as a black box (no code can), and in order to solve
a new problem in an optimal way, users will need to find their own optimal set of parameters. In particular, you need to be careful in choosing the right values of viscosity,
magnetic diffusivity, and radiative conductivity.
The P ENCIL C ODE is primarily designed to deal with weakly compressible turbulent
flows, which is why we use high-order first and second derivatives. To achieve good parallelization, we use explicit (as opposed to compact) finite differences. Typical scientific
targets include driven MHD turbulence in a periodic box, convection in a slab with nonperiodic upper and lower boundaries, a convective star embedded in a fully nonperiodic
box, accretion disc turbulence in the shearing sheet approximation, etc. Furthermore,
nonlocal radiation transport, inertial particles, dust coagulation, self-gravity, chemical
reaction networks, and several other physical components are installed, but this number increases steadily. In addition to Cartesian coordinates, the code can also deal with
spherical and cylindrical polar coordinates.
Magnetic fields are implemented in terms of the magnetic vector potential to ensure
that the field remains solenoidal (divergence-free). At the same time, having the magnetic vector potential readily available is a big advantage if one wants to monitor the
magnetic helicity, for example. The code is therefore particularly well suited for all kinds
of dynamo problems.
The code is normally non-conservative; thus, conserved quantities should only be conserved up to the discretization error of the scheme (not to machine accuracy). There is
no guarantee that a conservative code is more accurate with respect to quantities that
are not explicitly conserved, such as entropy. Another important quantity that is (to
our knowledge) not strictly conserved by ordinary flux conserving schemes is magnetic
helicity .
There are currently no plans to implement adaptive mesh refinement into the code,
which would cause major technical complications. Given that turbulence is generically
space-filling, local refinement to smaller scales would often not be very useful anyway.
On the other hand, in some geometries turbulence may well be confined to certain regions in space, so one could indeed gain by solving the outer regions with fewer points.
In order to be cache-efficient, we solve the equations along pencils in the x direction.
One very convenient side-effect is that auxiliary and derived variables use very little
memory, as they are only ever defined on one pencil. The domain can be tiled in the y
and z directions. On multiprocessor computers, the code can use MPI (Message Passing Interface) calls to communicate between processors. An easy switching mechanism
allows the user to run the code on a machine without MPI libraries (e.g. a notebook
computer). Ghost zones are used to implement boundary conditions on physical and
processor boundaries.
vii

A high level of flexibility is achieved by encapsulating individual physical processes
and variables in individual modules , which can be switched on or off in the file
‘Makefile.local’ in the local ‘src’ directory. This approach avoids the use of difficultto-read preprocessor directives, at the price of requiring one dummy module for each
physics module. For nonmagnetic hydrodynamics, for example, one will use the module
‘nomagnetic.f90’ and specifies
MAGNETIC = nomagnetic
in ‘Makefile.local’, while for MHD simulations, ‘magnetic.f90’ will be used:
MAGNETIC = magnetic
Note that the term module as used here is only loosely related to Fortran modules: both
‘magnetic.f90’ and ‘nomagnetic.f90’ define an F90 module named Magnetic — this is the
basis of the switching mechanism we are using.
Input parameters (which are set in the files ‘start.in’, ‘run.in’) can be changed without
recompilation. Furthermore, one can change the list of variables for monitoring (diagnostic) output on the fly, and there are mechanisms for making the code reload new
parameters or exit gracefully at runtime. You may want to check for correctness of these
files with the command pc_configtest.
The requirements for using the Pencil-MPI code are modest: you can use it on any Linux
or Unix system with a F95 and C compiler suite, like GNU gcc and gfortran , together
with the shell CSH , and the Perl interpreter are mandatory requirements.
Although the P ENCIL C ODE is mainly designed to run on supercomputers, more than
50% of the users run their code also on Macs, and the other half uses either directly
Linux on their laptops or they use VirtualBox on their Windows machine on which they
install Ubuntu Linux. If you have IDL as well, you will be able to visualize the results (a number of sample procedures are provided), but other tools such as Python , DX
(OpenDX, data explorer) can also be used and some relevant tools and routines come
with the P ENCIL C ODE.
If you want to make creative use of the code, this manual will contain far too little information. Its major aim is to give you an idea of the way the code is organized, so you
can more efficiently read the source code, which contains a reasonable amount of comments. You might want to read through the various sample directories that are checked
in. Choose one that is closest to your application and start modifying. For further enhancements that you may want to add to the code, you can take as an example the lines
in the code that deal with related variables, functions, diagnostics, equations etc., which
have already been implemented. Just remember: grep is one of your best friends when
you want to understand how certain variables or functions are used in the code.
We will be happy to include user-supplied changes and updates to the code in future
releases and welcome any feedback.
Potsdam
Stockholm

wdobler@gmail.com
AxelBrandenburg@gmail.com

viii

Acknowledgments
Many people have contributed in different ways to the development of this code. We
thank first of all Åke Nordlund (Copenhagen Observatory) and Bob Stein (University of
Michigan) who introduced us to the idea of using high-order schemes in compressible
flows and who taught us a lot about simulations in general.
The calculation of the power spectra, structure functions, the remeshing procedures,
routines for changing the number of processors, as well as the shearing sheet approximation and the flux-limited diffusion approximation for radiative transfer were implemented by Nils Erland L. Haugen (University of Trondheim). Tobi Heinemann added
the long characteristics method for radiative transfer as well as hydrogen ionization.
He also added and/or improved shock diffusion for other variables and improved the
resulting timestep control. Anders Johansen, Wladimir (Wlad) Lyra, and Jeff Oishi contributed to the implementation of the dust equations (which now comprises an array of
different components). Antony (Tony) Mee (University of Newcastle) implemented shock
viscosity and added the interstellar module together with Graeme R. Sarson (also University of Newcastle), who also implemented the geodynamo set-up together with David
McMillan (currently also at the University of Newcastle). Tony also included a method
for outputting auxiliary variables and enhanced the overall functionality of the code
and related idl and dx procedures. He also added, together with Andrew Snodin, the
evolution equations for the cosmic ray energy density. Vladimir Pariev (University of
Rochester) contributed to the development and testing of the potential field boundary
condition at an early stage. The implementation of spherical and cylindrical coordinates
is due to Dhrubaditya (Dhruba) Mitra and Wladimir Lyra. Wlad also implemented the
global set-up for protoplanetary disks (as opposed to the local shearing sheet formalism).
He also added a N -body code (based on the particle module coded by Anders Johansen
and Tony), and implemented the coupled evolution equations of neutrals and ions for
two-fluid models of ambipolar diffusion. Boris Dintrans is in charge of implementing the
anelastic and Boussinesq modules. Philippe-A. Bourdin implemented HDF5 support and
wrote the optional IO-modules for high-performance computing featuring various communication strategies. He also contributed to the solar-corona module and worked on
the IDL GUI, including the IDL routines for reading and working with large amounts of
data. Again, this list contains other recent items that are not yet fully documented and
acknowledged.
Use of the PPARC supported supercomputers in St Andrews (Mhd) and Leicester (Ukaff)
is acknowledged. We also acknowledge the Danish Center for Scientific Computing for
granting time on Horseshoe, which is a 512+140 processor Beowulf cluster in Odense
(Horseshoe).

ix

Contents

I

Using the P ENCIL C ODE

1

1 System requirements

1

2 Obtaining the code
2.1 Obtaining the code via git or svn
2.2 Updating via svn or git . . . . . .
2.3 Getting the last validated version
2.4 Getting older versions . . . . . .

.
.
.
.

2
2
2
3
4

.
.
.
.
.
.
.
.

5
5
5
6
6
6
7
7
9

.
.
.
.
.
.
.
.
.
.
.

11
11
12
12
13
14
14
14
14
14
14
15

.
.
.
.
.
.
.
.
.
.
.
.
.

17
17
17
18
20
20
20
21
22
23
25
25
25
26

.
.
.
.

.
.
.
.

.
.
.
.

3 Getting started
3.1 Setup . . . . . . . . . . . . . . . . . . .
3.1.1 Environment settings . . . . . .
3.1.2 Linking scripts and source files
3.1.3 Adapting ‘Makefile.src’ . . . .
3.1.4 Running make . . . . . . . . . .
3.1.5 Choosing a data directory . . .
3.1.6 Running the code . . . . . . . .
3.2 Further tests . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

4 Code structure
4.1 Directory tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Data access in pencils . . . . . . . . . . . . . . . . . . . . .
4.2.2 Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Files in the run directories . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 ‘start.in’, ‘run.in’, ‘print.in’ . . . . . . . . . . . . . . . . .
4.3.2 ‘datadir.in’ . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.3 ‘reference.out’ . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.4 ‘start.csh’, ‘run.csh’, ‘getconf.csh’ [obsolete; see Sect. 5.1]
4.3.5 ‘src/ ’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.6 ‘data/ ’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Using the code
5.1 Configuring the code to compile and run on your computer
5.1.1 Locating the configuration file . . . . . . . . . . . . .
5.1.2 Structure of configuration files . . . . . . . . . . . .
5.1.3 Compiling the code . . . . . . . . . . . . . . . . . . .
5.1.4 Running the code . . . . . . . . . . . . . . . . . . . .
5.1.5 Testing the code . . . . . . . . . . . . . . . . . . . . .
5.2 Adapting ‘Makefile.src’ [obsolete; see Sect. 5.1] . . . . . .
5.3 Changing the resolution . . . . . . . . . . . . . . . . . . . .
5.4 Using a non-equidistant grid . . . . . . . . . . . . . . . . .
5.5 Diagnostic output . . . . . . . . . . . . . . . . . . . . . . . .
5.6 Data files . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6.1 Snapshot files . . . . . . . . . . . . . . . . . . . . . .
5.7 Video files and slices . . . . . . . . . . . . . . . . . . . . . .
x

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

5.8 Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.8.1 One-dimensional output averaged in two dimensions
5.8.2 Two-dimensional output averaged in one dimension .
5.8.3 Azimuthal averages . . . . . . . . . . . . . . . . . . . .
5.8.4 Time averages . . . . . . . . . . . . . . . . . . . . . . .
5.9 Helper scripts . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.10 RELOAD and STOP files . . . . . . . . . . . . . . . . . . . . .
5.11 RERUN and NEWDIR files . . . . . . . . . . . . . . . . . . .
5.12 Start and run parameters . . . . . . . . . . . . . . . . . . . .
5.13 Physical units . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.14 Minimum amount of viscosity . . . . . . . . . . . . . . . . . .
5.15 The time step . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.15.1 The usual RK-2N time step . . . . . . . . . . . . . . .
5.15.2 The Runge-Kutta-Fehlberg time step . . . . . . . . . .
5.16 Boundary conditions . . . . . . . . . . . . . . . . . . . . . . .
5.16.1 Where to specify boundary conditions . . . . . . . . .
5.16.2 How to specify boundary conditions . . . . . . . . . .
5.17 Restarting a simulation . . . . . . . . . . . . . . . . . . . . .
5.18 One- and two-dimensional runs . . . . . . . . . . . . . . . . .
5.19 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.19.1 Gnuplot . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.19.2 Data explorer . . . . . . . . . . . . . . . . . . . . . . .
5.19.3 GDL . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.19.4 IDL . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.19.5 Python . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.20 Running on multi-processor computers . . . . . . . . . . . . .
5.20.1 How to run a sample problem in parallel . . . . . . .
5.20.2 Hierarchical networks (e.g. on Beowulf clusters) . . .
5.20.3 Extra workload caused by the ghost zones . . . . . . .
5.21 Running in double-precision . . . . . . . . . . . . . . . . . . .
5.22 Power spectrum . . . . . . . . . . . . . . . . . . . . . . . . . .
5.23 Structure functions . . . . . . . . . . . . . . . . . . . . . . . .
5.24 Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.24.1 Particles in parallel . . . . . . . . . . . . . . . . . . . .
5.24.2 Large number of particles . . . . . . . . . . . . . . . .
5.24.3 Random number generator . . . . . . . . . . . . . . .
5.25 Non-cartesian coordinate systems . . . . . . . . . . . . . . . .
6 The equations
6.1 Continuity equation . . . . . . . . . . . .
6.2 Equation of motion . . . . . . . . . . . .
6.3 Induction equation . . . . . . . . . . . .
6.4 Entropy equation . . . . . . . . . . . . .
6.4.1 Viscous heating . . . . . . . . . .
6.4.2 Alternative description . . . . . .
6.5 Transport equation for a passive scalar
6.6 Bulk viscosity . . . . . . . . . . . . . . .
6.6.1 Shock viscosity . . . . . . . . . .
6.7 Equation of state . . . . . . . . . . . . .
6.8 Ionization . . . . . . . . . . . . . . . . . .
xi

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

29
29
29
29
30
31
33
34
34
36
37
38
38
38
39
39
39
40
41
41
41
41
42
43
46
49
49
50
50
51
52
54
55
56
58
58
59

.
.
.
.
.
.
.
.
.
.
.

60
60
60
61
61
62
62
63
63
63
63
64

.
.
.
.
.
.
.
.
.
.
.
.

65
66
67
67
67
68
69
69
69
70
71
71

7 Troubleshooting / Frequently Asked Questions
7.1 Download and setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.1.1 Download forbidden . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.1.2 Shell gives error message when sourcing ‘sourceme.X’ . . . . . . . .
7.2 Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.1 Problems compiling syscalls . . . . . . . . . . . . . . . . . . . . . . .
7.2.2 Unable to open include file: chemistry.h . . . . . . . . . . . . . . . .
7.2.3 Compiling with ifc under Linux . . . . . . . . . . . . . . . . . . . . .
7.2.4 Segmentation fault with ifort 8.0 under Linux . . . . . . . . . . . .
7.2.5 The underscore problem: linking with MPI . . . . . . . . . . . . . .
7.2.6 Compilation stops with the cryptic error message: . . . . . . . . . .
7.2.7 The code doesn’t compile, . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.8 Some samples don’t even compile, . . . . . . . . . . . . . . . . . . .
7.2.9 Internal compiler error with Compaq/Dec F90 . . . . . . . . . . . .
7.2.10 Assertion failure under SunOS . . . . . . . . . . . . . . . . . . . . .
7.2.11 After some dirty tricks I got pencil code to compile with MPI, ... . .
7.2.12 Error: Symbol ’mpi comm world’ at (1) has no IMPLICIT type . . .
7.2.13 Error: Can’t open included file ’mpif.h’ . . . . . . . . . . . . . . . . .
7.3 Pencil check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.1 The pencil check complains for no reason. . . . . . . . . . . . . . . .
7.3.2 The pencil check reports MISSING PENCILS and quits . . . . . .
7.3.3 The pencil check reports unnecessary pencils . . . . . . . . . . . . .
7.3.4 The pencil check reports that most or all pencils are missing . . . .
7.3.5 Running the pencil check triggers mathematical errors in the code
7.3.6 The pencil check still complains . . . . . . . . . . . . . . . . . . . . .
7.3.7 The pencil check is annoying so I turned it off . . . . . . . . . . . .
7.4 Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.1 Periodic boundary conditions in ‘start.x’ . . . . . . . . . . . . . . .
7.4.2 csh problem? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.3 ‘run.csh’ doesn’t work: . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.4 Code crashes after restarting . . . . . . . . . . . . . . . . . . . . . .
7.4.5 auto-test gone mad...? . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4.6 Can I restart with a different number of cpus? . . . . . . . . . . . .
7.4.7 Can I restart with a different number of cpus? . . . . . . . . . . . .
7.4.8 fft xyz parallel 3D: nygrid needs to be an integer multiple... . . . .
7.4.9 Unit-agnostic calculations? . . . . . . . . . . . . . . . . . . . . . . .
7.5 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75
75
75
75
76
76
76
76
77
77
77
78
78
79
79
80
80
81
81
81
81
81
81
82
82
82
82
82
82
83
83
83
84
84
84
85
86

6.9
6.10
6.11
6.12
6.13
6.14

6.15
6.16
6.17

6.8.1 Ambipolar diffusion . . . . . . . . . . . . .
Radiative transfer . . . . . . . . . . . . . . . . . .
Self-gravity . . . . . . . . . . . . . . . . . . . . . .
Incompressible and anelastic equations . . . . .
Dust equations . . . . . . . . . . . . . . . . . . . .
Cosmic ray pressure in diffusion approximation
Particles . . . . . . . . . . . . . . . . . . . . . . .
6.14.1 Tracer particles . . . . . . . . . . . . . . .
6.14.2 Dust particles . . . . . . . . . . . . . . . .
N -body solver . . . . . . . . . . . . . . . . . . . .
Test-field equations . . . . . . . . . . . . . . . . .
Gravitational wave equations . . . . . . . . . . .

xii

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

7.5.1 ‘start.pro’ doesn’t work: . . . . . . . . . . . . . . . . . . .
7.5.2 ‘start.pro’ doesn’t work: . . . . . . . . . . . . . . . . . . .
7.5.3 Something about tag name undefined: . . . . . . . . . . .
7.5.4 Something INC in start.pro . . . . . . . . . . . . . . . . .
7.5.5 nl2idl problem when reading param2.nml . . . . . . . . .
7.5.6 Spurious dots in the time series file . . . . . . . . . . . .
7.5.7 Problems with pc_varcontent.pro . . . . . . . . . . . . .
7.6 General questions . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.6.1 “Installation” procedure . . . . . . . . . . . . . . . . . . .
7.6.2 Small numbers in the code . . . . . . . . . . . . . . . . . .
7.6.3 Why do we need a /lphysics/ namelist in the first place?
7.6.4 Can I run the code on a Mac? . . . . . . . . . . . . . . . .
7.6.5 Pencil Code discussion forum . . . . . . . . . . . . . . . .
7.6.6 The manual . . . . . . . . . . . . . . . . . . . . . . . . . .

II

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.

Programming the P ENCIL C ODE

86
86
86
86
87
87
87
88
88
88
89
90
90
90

91

8 Understanding the code
8.1 Example: how is the continuity equation being solved? . . . . . . . . . . .
9 Adapting the code
9.1 The P ENCIL C ODE coding standard . . . . . . . . . . . . . .
9.2 Adding new output diagnostics . . . . . . . . . . . . . . . .
9.3 The f-array . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.4 The df-array . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.5 The fp-array . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.6 The pencil case . . . . . . . . . . . . . . . . . . . . . . . . . .
9.6.1 Pencil check . . . . . . . . . . . . . . . . . . . . . . .
9.6.2 Adding new pencils . . . . . . . . . . . . . . . . . . .
9.7 Adding new physics: the Special module . . . . . . . . . . .
9.8 Adding switchable modules . . . . . . . . . . . . . . . . . .
9.9 Adding your initial conditions: the InitialCondition module

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

95
95
97
97
98
100
100
101
101
102
103
103
104
104

10 Testing the code
106
10.1 How to set up periodic tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
11 Useful internals
108
11.1 Global variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
11.2 Subroutines and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

III Appendix
A Timings
A.1 Test case . . . . .
A.2 Running the code
A.3 Triolith . . . . . .
A.4 Lindgren . . . . .

111
.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

B Coding standard

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

111
118
119
119
119
122

xiii

B.1 File naming conventions . . . . .
B.2 Fortran Code . . . . . . . . . . . .
B.2.1 Indenting and whitespace
B.2.2 Comments . . . . . . . . .
B.2.3 Module names . . . . . . .
B.2.4 Variable names . . . . . .
B.2.5 Emacs settings . . . . . .
B.3 Other best practices . . . . . . . .
B.4 General changes to the code . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

C Some specific initial conditions
C.1 Random velocity or magnetic fields . . . . . . . . .
C.2 Turbulent initial with given spectrum . . . . . . .
C.3 Beltrami fields . . . . . . . . . . . . . . . . . . . . .
C.4 Magnetic flux rings: initaa=’fluxrings’ . . . . .
C.5 Vertical stratification . . . . . . . . . . . . . . . . .
C.5.1 Isothermal atmosphere . . . . . . . . . . . .
C.5.2 Polytropic atmosphere . . . . . . . . . . . .
C.5.3 Changing the stratification . . . . . . . . .
C.5.4 The Rayleigh number . . . . . . . . . . . . .
C.5.5 Entropy boundary condition . . . . . . . . .
C.5.6 Temperature boundary condition at the top
C.6 Potential-field boundary condition . . . . . . . . .
C.7 Planet solution in the shearing box . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

122
122
122
123
124
124
125
126
126

.
.
.
.
.
.
.
.
.
.
.
.
.

127
127
127
128
128
129
129
130
131
131
132
132
132
134

D Some specific boundary conditions
135
D.1 Perfect-conductor boundary condition . . . . . . . . . . . . . . . . . . . . . 135
D.2 Stress-free boundary condition . . . . . . . . . . . . . . . . . . . . . . . . . 135
D.3 Normal-field-radial boundary condition . . . . . . . . . . . . . . . . . . . . 136
E High-frequency filters
E.1 Conservative hyperdissipation . . . . .
E.2 Hyperviscosity . . . . . . . . . . . . . .
E.2.1 Conservative case . . . . . . . .
E.2.2 Non-conservative cases . . . . .
E.2.3 Choosing the coefficient . . . .
E.2.4 Turbulence with hyperviscosity
E.3 Anisotropic hyperdissipation . . . . . .
E.4 Hyperviscosity in Burgers shock . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

137
137
139
139
140
141
141
142
142

F Special techniques
144
F.1 After changing REAL PRECISION . . . . . . . . . . . . . . . . . . . . . . . 144
F.2 Remeshing (regridding) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
F.3 Restarting from a run with less physics . . . . . . . . . . . . . . . . . . . . 145
G Runs and reference data
G.1 Shock tests . . . . . . . . . . . .
G.1.1 Sod shock tube problem
G.1.2 Temperature jump . . .
G.2 Random forcing function . . . .
G.3 Three-layered convection model

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

xiv

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

147
147
147
147
147
148

G.4 Magnetic helicity in the shearing sheet . . . . . . . . . . . . . . . . . . . . 149
H Numerical methods
H.1 Sixth-order spatial derivatives . . . . . . . . .
H.2 Upwind derivatives to avoid ‘wiggles’ . . . . . .
H.3 The bidiagonal scheme for cross-derivatives . .
H.4 The 2N-scheme for time-stepping . . . . . . . .
H.5 Diffusive error from the time-stepping . . . . .
H.6 Ionization . . . . . . . . . . . . . . . . . . . . . .
H.7 Radiative transfer . . . . . . . . . . . . . . . . .
H.7.1 Solving the radiative transfer equation
H.7.2 Angular integration . . . . . . . . . . . .
I

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

Switchable modules

153
153
154
155
156
157
158
159
159
160
163

J Startup and run-time parameters
J.1 Startup parameters for ‘start.in’ . . . .
J.2 Runtime parameters for ‘run.in’ . . . .
J.3 Parameters for ‘print.in’ . . . . . . . . .
J.4 Parameters for ‘video.in’ . . . . . . . . .
J.5 Parameters for ‘phiaver.in’ . . . . . . .
J.6 Parameters for ‘xyaver.in’ . . . . . . . .
J.7 Parameters for ‘xzaver.in’ . . . . . . . .
J.8 Parameters for ‘yzaver.in’ . . . . . . . .
J.9 Parameters for ‘yaver.in’ . . . . . . . . .
J.10 Parameters for ‘zaver.in’ . . . . . . . . .
J.11 Boundary conditions . . . . . . . . . . .
J.11.1 Boundary condition bcx . . . . .
J.11.2 Boundary condition bcy . . . . .
J.11.3 Boundary condition bcz . . . . .
J.12 Initial condition parameter dependence

IV

.
.
.
.
.
.
.
.
.

Indexes

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

163
163
171
178
214
216
217
223
224
227
229
232
232
235
237
241

245

xv

xvi

1

Part I

Using the P ENCIL C ODE
1 System requirements
To use the code, you will need the following:
1. Absolutely needed:
• F95 compiler
• C compiler
2. Used heavily (if you don’t have one of these, you will need to adjust many things
manually):
• a Unix /Linux -type system with make and csh
• Perl (remember: if it doesn’t run Perl, it’s not a computer)
3. The following are dispensable, but enhance functionality in one way or the other:
• an MPI implementation (for parallelization on multiprocessor systems)
• DX alias OpenDX or data explorer (for 3-D visualization of results)
• IDL (for visualization of results; the 7-minute demo license will do for many
applications)

2

T HE P ENCIL C ODE

2

Obtaining the code

The code is now distributed via https://github.com/pencil-code/pencil-code, where
you can either download a tarball, or, preferably, download it via svn or git . In Iran and
some other countries, GitHub is not currently available. To alleviate this problem, we
have made a recent copy available on http://www.nordita.org/software/pencil-code/.
If you want us to update this tarball, please contact us.
To ensure at least some level of stability of the svn/git versions, a set of test problems
(listed in ‘$PENCIL_HOME/bin/auto-test’) are routinely tested. This includes all problems
in ‘$PENCIL_HOME/samples’. See Sect. 10 for details.

2.1

Obtaining the code via git or svn

1. Many machines have svn installed (try svn -v or which svn). On Ubuntu, for example, svn comes under the package name subversion.
2. The code is now saved under Github, git can be obtained in Linux by typing sudo
apt-get install git
3. Unless you are a privileged users with write access, you can download the code
with the command
git clone https://github.com/pencil-code/pencil-code.git
or
svn checkout https://github.com/pencil-code/pencil-code/trunk/ ...\\
pencil-code --username MY_GITHUB_USERNAME
In order to push your changes to the repository, you have to ask the maintainer of
pencil code for push access (to become a contributor), or put a pull request to the
maintainer of the code.
Be sure to run auto-test before you check anything back in again. It can be very
annoying for someone else to figure out what’s wrong, especially if you are just up
to something else. At the very least, you should do
pc_auto-test --level=0 --no-pencil-check -C
This allows you to run just 2 of the most essential tests starting with all the nomodules and then most-modules.

2.2

Updating via svn or git

Independent of how you installed the code in the first place (from tarball or via svn/git ),
you can update your version using svn/git . If you have done nontrivial alterations to
your version of the code, you ought to be careful about upgrading: although svn/git is an
excellent tool for distributed programming, conflicts are quite possible, since many of us
are going to touch many parts of the code while we develop it further. Thus, despite the

2.3

Getting the last validated version

3

fact that the code is under svn/git , you should probably back up your important changes
before upgrading.
Here is the upgrading procedure for git :
1. Perform a git update of the tree:
unix>

git pull

2. Fix any conflicts you encounter and make sure the examples in the directory
‘samples/’ are still working.
Here is the upgrading procedure for svn :
1. Perform a svn update of the tree:
unix>

pc_svnup

2. Fix any conflicts you encounter and make sure the examples in the directory
‘samples/’ are still working.
If you have made useful changes, please contact one of the (currently) 10 “Contributors”
(listed under https://github.com/pencil-code/pencil-code) who can give you push or
check-in permission. Be sure to have sufficient comments in the code and please follow
our standard coding conventions explained in Section 9.1. There is also a script to check
and fix the most common stylebreaks, pc codingstyle.

2.3

Getting the last validated version

The script pc_svnup accepts arguments -val or -validated, which means that the current
changes on a user’s machine will be merged into the last working version. This way
every user can be sure that any problems with the code must be due to the current
changes done by this user since the last check-in.
Examples:
unix>

pc_svnup -src -s -validated

brings all files in ‘$PENCIL_HOME/src’ to the last validated status, and merges all your
changes into this version. This allows you to work with this, but in order to check in
your changes you have to update everything to the most recent status first, i.e.
unix>

pc_svnup -src

Your own changes will be merged into this latest version as before.
NOTE: The functionality of the head of the trunk should be preserved at all times. However, accidents do happen. For the benefit of all other developers, any errors should
be corrected within 1-2 hours. This is the reason why the code comes with a file
‘pencil-code/license/developers.txt’, which should contain contact details of all developers. The pc_svnup -val option allows all other people to stay away from any trouble.

4

2.4

T HE P ENCIL C ODE

Getting older versions

You may find that the latest svn version of the code produces errors. If you have reasons
to believe that this is due to changes introduced on 27 November 2008 (to give an example), you can check out the version prior to this by specifying a revision number with
svn update -r #####. One reason why one cannot always reproduce exactly the same
situation too far back in time is connected with the fact that processor architecture and
the compiler were different, resulting e.g. in different rounding errors.

3. Getting started

5

3 Getting started
To get yourself started, you should run one or several examples which are provided in
one of the ‘samples/’ subdirectories. Note that you will only be able to fully assess the
numerical solutions if you visualize them with IDL , DX or other tools (see Sect. 5.19).
3.1

Setup

3.1.1 Environment settings
The functionality of helper scripts and IDL routines relies on a few environment variables being set correctly. The simplest way to achieve this is to go to the top directory of
the code and source one of the two scripts ‘sourceme.csh’ or ‘sourceme.sh’ (depending on
the type of shell you are using):
csh>
csh>

cd pencil-code
source ./sourceme.csh

for tcsh or csh users; or
sh>
sh>

cd pencil-code
. ./sourceme.sh

for users of bash , Bourne shell , or similar shells. You should get output similar to
PENCIL_HOME = 
Adding /home/dobler/f90/pencil-code/bin to PATH
Apart from the PATH variable, the environment variable IDL_PATH is set to something
like ./idl:../idl:+$PENCIL_HOME/idl:./data: .
Note 1 The  mechanism does not work for IDL versions 5.2 or older. In
this case, you will have to edit the path manually, or adapt the ‘sourceme’ scripts.
Note 2 If you don’t want to rely on the ‘sourceme’ scripts’ (quite heuristic) ability to correctly identify the code’s main directory, you can set the environment variable PENCIL_HOME explicitly before you run the source command.
Note 3 Do not just source the ‘sourceme’ script from your shell startup file (‘~/.cshrc’
or ‘~/.bashrc’, because it outputs a few lines of diagnostics for each sub-shell, which will
break many applications. To suppress all output, follow the instructions given in the
header documentation of ‘sourceme.csh’ and ‘sourceme.sh’. Likewise, output from other
files invoked by source should also be suppressed.
Note 4 The second time you source ‘sourceme’, it will not add anything to your PATH
variable. This is on purpose to avoid cluttering of your environment: you can source the
file as often as you like (in your shell startup script, then manually and in addition in
some script you have written), without thinking twice. If, however, at the first sourcing,
the setting of PENCIL_HOME was wrong, this mechanism would keep you from ever adding

6

T HE P ENCIL C ODE

the right directory to your PATH. In this case, you need to first undefine the environment
variable PENCIL_HOME:
csh> unsetenv PENCIL_HOME
csh> source ./sourceme.csh
or
sh> unset PENCIL_HOME
sh> . ./sourceme.sh
3.1.2 Linking scripts and source files
With your environment set up correctly, you can now go to the directory you want to
work in and set up subdirectories and links. This is accomplished by the script ‘pc_setupsrc’, which is located in ‘$PENCIL_HOME/bin’ and is thus now in your executable
path.
For concreteness, let us assume you want to use ‘samples/conv-slab’ as your run directory , i.e. you want to run a three-layer slab model of solar convection. You then do the
following:
unix> cd samples/conv-slab
unix> pc_setupsrc
src already exists
2 files already exist in src
The script has linked a number of scripts from ‘$PENCIL_HOME/bin’, generated a directory
‘src’ for the source code and linked the Fortran source files (plus a few more files) from
‘$PENCIL_HOME/src’ to that directory:
unix> ls -F
reference.out
start.csh@
start.in

src/
run.csh@
run.in

getconf.csh@
print.in

3.1.3 Adapting ‘Makefile.src’
This step requires some input from you, but you only have to do this once for each
machine you want to run the code on. See Sect. 5.2 for a description of the steps you
need to take here.
Note: If you are lucky and use compilers similar to the ones we have, you may be able
to skip this step; but blame yourself if things don’t compile, then. If not, you can run
make with explicit flags, see Sect. 5.2 and in particular Table 1.
3.1.4 Running make
Next, you run make in the ‘src’ subdirectory of your run directory. Since you are
using one of the predefined test problems, the settings in ‘src/Makefile.local’ and
‘src/cparam.local’ are all reasonable, and you just do

3.1

unix>

Setup

7

make

If you have set up the compiler flags correctly, compilation should complete successfully.
3.1.5 Choosing a data directory
The code will by default write data like snapshot files to the subdirectory ‘data’ of the
run directory. Since this will involve a large volume of IO-operations (at least for large
grid sizes), one will normally try to avoid writing the data via NFS. The recommended
way to set up a ‘data’ data directory is to generate a corresponding directory on the local
disc of the computer you are running on and (soft-)link it to ‘./data’. Even if the link is
part of an NFS directory, all the IO operations will be local. For example, if you have a
local disc ‘/scratch’, you can do the following:
unix>
unix>

mkdir -p /scratch/$USER/pencil-data/samples/conv-slab
ln -s /scratch/$USER/pencil-data/samples/conv-slab ./data

This is done automatically by the pc_mkdatadir command which, in turn, is invoked
when making a new run directory with the pc_newrun command, for example.
Even if you don’t have an NFS-mounted directory (say, on your notebook computer), it
is probably still a good idea to have code and data well separated by a scheme like the
one described above.
An alternative to symbolic links, is to provide a file called ‘datadir.in’ in the root of
the run directory. This file should contain one line of text specifying the absolute or
relative data directory path to use. This facility is useful if one wishes to switch one run
directory between different data directories. It is suggested that in such cases symbolic
links are again made in the run directory to the various locations, then the ‘datadir.in’
need contain only a short relative path.
3.1.6 Running the code
You are now ready to start the code:
unix>

start.csh

Linux cincinnatus 2.4.18-4GB #1 Wed Mar 27 13:57:05 UTC 2002 i686 unknown
Non-MPI version
datadir = data
Fri Aug 8 21:36:43 CEST 2003
src/start.x
CVS: io_dist.f90
v. 1.61
(brandenb ) 2003/08/03 09:26:55
[. . . ]
CVS: start.in
v. 1.4
(dobler
) 2002/10/02 20:11:14
nxgrid,nygrid,nzgrid=
32
32
32
thermodynamics: assume cp=1
uu: up-down
piecewise polytropic vertical stratification (lnrho)
init_lnrho: cs2bot,cs2top=
1.450000
0.3333330
e.g. for ionization runs: cs2bot,cs2top not yet set
piecewise polytropic vertical stratification (ss)

8

T HE P ENCIL C ODE

start.x has completed successfully
0.070u 0.020s 0:00.14 64.2%
Fri Aug

0+0k 0+0io 180pf+0w

8 21:36:43 CEST 2003

This runs ‘src/start.x’ to construct an initial condition based on the parameters
set in ‘start.in’. This initial condition is stored in ‘data/proc0/var.dat’ (and in
‘data/proc1/var.dat’, etc. if you run the multiprocessor version). It is fair to say that
this is now a rather primitive routine; see ‘pencil-code/idl/read’ for various reading
routines. You can then visualize the data using standard idl language.
If you visualize the profiles using IDL (see below), the result should bear some resemblance to Fig. 2, but with different values in the ghost zones (the correct values are set
at run-time only) and a simpler velocity profile.
Now we run the code:
unix>

run.csh

This executes ‘src/run.x’ and carries out nt time steps, where nt and other run-time
parameters are specified in ‘run.in’. On a decent PC (1.7 GHz), 50 time steps take about
10 seconds.
The relevant part of the code’s output looks like
--it----t-------dt-------urms----umax----rhom------ssm-----dtc----dtu---dtnu---dtchi0
0.34 6.792E-03 0.0060 0.0452 14.4708 -0.4478 0.978 0.013 0.207 0.346
10
0.41 6.787E-03 0.0062 0.0440 14.4707 -0.4480 0.978 0.013 0.207 0.345
20
0.48 6.781E-03 0.0064 0.0429 14.4705 -0.4481 0.977 0.012 0.207 0.345
30
0.54 6.777E-03 0.0067 0.0408 14.4703 -0.4482 0.977 0.012 0.207 0.345
40
0.61 6.776E-03 0.0069 0.0381 14.4702 -0.4482 0.977 0.011 0.207 0.346

and lists
1. the number it of the current time step;
2. the time, t ;
3. the time step, dt ;
4. the rms velocity, urms =

p
hu2 i;

5. the maximum velocity, umax = max |u|;
6. the mean density, rhom = hρi;
7. the mean entropy, ssm = hsi /cp ;
8. the time step in units of the acoustic Courant step, dtc = δt/(cs0 δxmin );
9. the time step in units of the advective time step, dtu = δt/(cδt δx/ max |u|);
10. the time step in units of viscous time step, dtnu = δt/(cδt,v δx2 /νmax );
11. the time step in units of the conductive time step, dtchi = δt/(cδt,v δx2 /χmax ).
The entries in this list can be added, removed or reformatted in the file ‘print.in’, see
Sects 5.5 and J.3. The output is also saved in ‘data/time_series.dat’ and should be
identical to the content of ‘reference.out’.

3.2
ln ρ

Further tests

uz

9
Entropy s

Temperature T

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5
z

1.5

z

1.5

z

1.5

z

1.5

0.0

0.0

0.0

0.0

−0.5

−0.5

−0.5

−0.5

0

1

2

3

−0.02 0.00 0.02 0.04

−0.6−0.4−0.2 0.0 0.2

0.5

1.0

1.5

2.0

Figure 2: Stratification of the three-layer convection model in ‘samples/conv-slab’ after 50 timesteps
(t = 0.428). Shown are (from left to right) density ρ, vertical velocity uz , entropy s/cp and temperature
T as functions of the vertical coordinate z for about ten different vertical lines in the computational box.
The dashed lines denote domain boundaries: z < −0.68 is the lower ghost zone (points have no physical
significance); −0.68 < z < 0 is a stably stratified layer (ds/dz > 0); 0 < z < 1 is the unstable layer
(ds/dz < 0); 1 < z < 1.32 is the isothermal top layer; z > 1.32 is the upper ghost zone (points have no
physical significance).

If you have IDL , you can visualize the stratification with (see Sect. 5.19.4 for details)
unix >
IDL >
IDL >

idl
pc_read_var,obj=var,/trimall
tvscl,var,uu(*,*,0,0)

which shows ux in the xy plane through the first meshpoint in the z direction. There
have been some now outdates specific routines that produce results like that shown in
Fig. 2.
Note: If you want to run the code with MPI , you will probably need to adapt
‘getconf.csh’, which defines the commands and flags used to run MPI jobs (and which
is sourced by the scripts ‘start.csh’ and ‘run.csh’). Try
csh -v getconf.csh
or
csh -x getconf.csh
to see how ‘getconf.csh’ makes its decisions. You would add a section for the host name
of your machine with the particular settings. Since ‘getconf.csh’ is linked from the central directory ‘pencil-code/bin’, your changes will be useful for all your other runs too.

3.2

Further tests

There is a number of other tests in the ‘samples/’ directory. You can use the script
‘bin/auto-test’ to automatically run these tests and have the output compared to refer-

10
ence results.

T HE P ENCIL C ODE

4. Code structure

11

4 Code structure
4.1

Directory tree

pencil-code

src

idl | dx

doc

bin

Makefile

conv-slab

start, run

src

samples

interlocked-fluxrings

...

Makefile.local
cparam.local

timestep, deriv

equ

hydro | nohydro
density | nodensity
entropy | noentropy
grav_z | grav_r | nograv
magnetic | nomagnetic

Figure 3: The basic structure of the code

The overall directory structure of the code is shown in Fig. 3. Under ‘pencil-code’, there
are currently the following files and directories:
bin/
bugs/

config/ doc/ idl/
dx/
lib/ misc/

license/ perl/
samples/
README sourceme.csh src/

sourceme.sh
www/

utils/

Almost all of the source code is contained in the directory ‘src/’, but in order to encapsulate individual applications, the code is compiled separately for each run in a local
directory ‘src’ below the individual run directory, like e. g. ‘samples/conv-slab/src’.
It may be a good idea to keep your own runs also under svn or cvs (which is older than
but similar to svn ), but this would normally be a different repository. On the machine
where you are running the code, you may want to check them out into a subdirectory of

12

T HE P ENCIL C ODE

‘pencil-code/’. For example, we have our own runs in a repository called ‘pencil-runs’,
so we do
unix>
unix>

cd $PENCIL_HOME
svn co runs pencil-runs

In this case, ‘runs’ contains individual run directories, grouped in classes (like ‘spher’ for
spherical calculations, or ‘kinematic’ for kinematic dynamo simulations). The current
list of classes in our own ‘pencil-runs’ repository is
1d-tests/
2d-tests/
3d-tests/
buoy_tube/
convstar/

disc/
discont/
discussion/
forced/
interstellar/

kinematic/
Misc/
OLD/
pass_only/
radiation/

rings/
slab_conv/
test/

The directory ‘forced/’ contains some forced turbulence runs (both magnetic and nonmagnetic); ‘gravz/’ contains runs with vertical gravity; ‘rings/’ contains decaying MHD
problems (interlocked flux rings as initial condition, for example); and ‘kinematic/’ contains kinematic dynamo problems where the hydrodynamics is turned off entirely. The
file ‘samples/README’ should contain an up-to-date list and short description of the individual classes.1
The subdirectory ‘src’ of each run directory contains a few local configuration files (currently these are ‘Makefile.local’ and ‘cparam.local’) and possibly ‘ctimeavg.local’. To
compile the samples, links the files ‘.f90’, ‘.c’ and ‘Makefile.src’ need to be linked from
the top file[src/]src directory to the local directory ‘./src’. These links are set up by the
script pc_setupsrc) when used in the root of a run directory.
General-purpose visualization routines for IDL or DX are in the directories ‘idl’ and ‘dx’,
respectively. There are additional and more specialized IDL directories in the different
branches under ‘pencil-runs’.
The directory ‘doc’ contains this manual; ‘bin’ contains a number of utility scripts
(mostly written in csh and Perl ), and in particular the ‘start.csh’, ‘run.csh’, and
‘getconf.csh’ scripts. The ‘.svn’ directory is used (you guessed it) by .svn , and is not
normally directly accessed by the user; ‘bugs’, finally is used by us for internal purposes.
The files ‘sourceme.csh’ and ‘sourceme.sh’ will set up some environment variables — in
particular PATH — and aliases/shell functions for your convenience. If you do not want
to source one of these files, you need to make sure your IDL path is set appropriately
(provided you want to use IDL ) and you will need to address the scripts from ‘bin’ with
their explicit path name, or adjust your PATH manually.
4.2

Basic concepts

4.2.1 Data access in pencils
Unlike the CRAY computers that dominated supercomputing in the 80s and early 90s,
all modern computers have a cache that constitutes a significant bottleneck for many
1

Our ‘pencil-runs’ directory also contains runs that were done some time ago. Occasionally, we try to
update these, especially if we have changed names or other input conventions.

4.2

Basic concepts

13

codes. This is the case if large three-dimensional arrays are constantly used within each
time step, which has the obvious advantage of working on long arrays and allows vectorization of elementary machine operations. This approach also implies conceptual simplicity of the code and allows extensive use of the intuitive F90 array syntax. However,
a more cache-efficient way of coding is to calculate an entire time step (or substep of a
multi-stage time-stepping scheme) only along a one-dimensional pencil of data within
the numerical grid. This technique is more efficient for modern RISC processors: on
Linux PCs and SGI workstations, for example, we have found a speed-up by about 60%
in some cases. An additional advantage is a drastic reduction in temporary storage for
auxiliary variables within each time step.

4.2.2 Modularity
Each run directory has a file ‘src/Makefile.local’ in which you choose certain modules 2 ,
which tell the code whether or not entropy, magnetic fields, hydrodynamics, forcing, etc.
should be invoked. For example, the settings for forced turbulent MHD simulations are
HYDRO
DENSITY
ENTROPY
MAGNETIC
GRAVITY
FORCING

=
hydro
=
density
= noentropy
=
magnetic
= nogravity
=
forcing

MPICOMM
GLOBAL
IO
FOURIER

= nompicomm
= noglobal
=
io_dist
= nofourier

This file will be processed by make and the settings are thus assignments of make
variables. Apart from the physics modules (equation of motion: yes, density [pressure]:
yes, entropy equation: no, magnetic fields: yes, gravity: no, forcing: yes), a few technical
modules can also be used or deactivated; in the example above, these are MPI (switched
off), additional global variables (none), input/output (distributed), and FFT (not used).
The table in Sect. I in the Appendix lists all currently available modules.
Note that most of these make variables must be set, but they will normally obtain reasonable default values in ‘Makefile’ (so you only need to set the non-standard ones in
‘Makefile.local’). It is by using this switching mechanism through make that we achieve
high flexibility without resorting to excessive amounts of cryptic preprocessor directives
or other switches within the code.
Many possible combinations of modules have already been tested and examples are part
of the distribution, but you may be interested in a combination which was never tried
before and which may not work yet, since the modules are not fully orthogonal. In such
cases, we depend on user feedback for fixing problems and documenting the changes for
others.
2

We stress once more that we are not talking about F90 modules here, although there is some connection, as most of our modules define F90 modules: For example each of the modules gravity simple , grav r
and nogravity defines a Fortran module Gravity .

14

4.3

T HE P ENCIL C ODE

Files in the run directories

4.3.1 ‘start.in’, ‘run.in’, ‘print.in’
These files specify the startup and runtime parameters (see Sects. 5.12 and J.2), and the
list of diagnostic variables to print (see 5.5). They specify the setup of a given simulation
and are kept under svn in the individual ‘samples’ directories.
You may want to check for the correctness of these configuration files by issuing the
command pc_configtest.

4.3.2 ‘datadir.in’
If this file exists, it must contain the name of an existing directory, which will be used as
data directory , i. e. the directory where all results are written. If ‘datadir.in’ does not
exist, the data directory is ‘data/’.

4.3.3 ‘reference.out’
If present, ‘reference.out’ contains the output you should obtain in the given run directory, provided you have not changed any parameters. To see whether the results of your
run are OK, compare ‘time_series.dat’ to ‘reference.out’:
unix>

diff data/time_series.dat reference.out

4.3.4 ‘start.csh’, ‘run.csh’, ‘getconf.csh’ [obsolete; see Sect. 5.1]
These are links to ‘$PENCIL_HOME/bin’. You will be constantly using the scripts
‘start.csh’ and ‘run.csh’ to initialize the code. Things that are needed by both (like the
name of the mpirun executable, MPI options, or the number of processors) are located in
‘getconf.csh’, which is never directly invoked.

4.3.5 ‘src/ ’
The ‘src’ directory contains two local files, ‘src/Makefile.local’ and ‘src/cparam.local’,
which allow the user to choose individual modules (see 4.2.2) and to set parameters like
the grid size and the number of processors for each direction. These two files are part
of the setup of a given simulation and are kept under svn in the individual ‘samples’
directories.
The file ‘src/cparam.inc’ is automatically generated by the script ‘mkcparam’ and contains
the number of fundamental variables for a given setup.
All other files in ‘src/’ are either links to source files (and ‘Makefile.src’) in the
‘$PENCIL_HOME/src’ directory, or object and module files generated by the compiler.

4.3

Files in the run directories

15

4.3.6 ‘data/ ’
This directory (the name of which will actually be overwritten by the first line of
‘datadir.in’, if that file is present; see §4.3.2) contains the output from the code:
‘data/dim.dat’ The global array dimensions.
‘data/legend.dat’ The header line specifying the names of the diagnostic variables in
‘time_series.dat’.
‘data/time_series.dat’ Time series of diagnostic variables (also printed to stdout). You
can use this file directly for plotting with Gnuplot , IDL , Xmgrace or similar tools (see
also §5.19).
‘data/tsnap.dat’, ‘data/tvid.dat’ Time when the next snapshot ‘VARN ’ or animation
slice should be taken.
‘data/params.log’ Keeps a log of all your parameters: ‘start.x’ writes the startup parameters to this file, ‘run.x’ appends the runtime parameters and appends them anew,
each time you use the ‘RELOAD’ mechanism (see §5.10).
‘data/param.nml’ Complete set of startup parameters, printed as Fortran namelist.
This file is read in by ‘run.x’ (this is how values of startup parameters are propagated
to ‘run.x’) and by IDL (if you use it).
‘data/param2.nml’ Complete set of runtime parameters, printed as Fortran namelist.
This file is read by IDL (if you use it).
‘data/index.pro’ Can be used as include file in IDL and contains the column in which
certain variables appear in the diagnostics file (‘time_series.dat’). It also contains the
positions of variables in the ‘VARN ’ files. These positions depend on whether entropy or
noentropy , etc, are invoked. This is a temporary solution and the file may disappear in
future releases.
‘data/interstellar.dat’ Unformatted file containing the time at which the next supernova event will occur, under certain supernova schemes. (Only needed by the interstellar module.)
‘data/proc0’, ‘data/proc1’, . . . These are the directories containing data from the individual processors. So after running an MPI job on two processors, you will have the
two directories ‘data/proc0’ and ‘data/proc1’. Each of the directories can contain the
following files:
‘var.dat’ binary file containing the latest snapshot;

16

T HE P ENCIL C ODE

‘VARN ’ binary file containing individual snapshot number N ;
‘dim.dat’ ASCII file containing the array dimensions as seen by the given processor;
‘time.dat’ ASCII file containing the time corresponding to ‘var.dat’ (not actually used
by the code, unless you use the io mpiodist.f90 module);
‘grid.dat’ binary file containing the part of the grid seen by the given processor;
‘seed.dat’ the random seed for the next time step (saved for reasons of reproducibility).
For multi-processor runs with velocity forcing, the files ‘procN /seed.dat’ must all
contain the same numbers, because globally coherent waves of given wavenumber
are used;
‘X.xy’, ‘X.xz’, ‘X.yz’ two-dimensional sections of variable X, where X stands for the
corresponding variable. The current list includes
bx.xy
ss.xz

bx.xz
ux.xy

by.xy
ux.xz

by.xz
uz.xy

bz.xy
uz.xz

bz.xz

divu.xy

lnrho.xz

Each processor writes its own slice, so these need to be reassembled if one wants
to plot a full slice.

5. Using the code

17

5 Using the code
5.1

Configuring the code to compile and run on your computer

Note: We recommend to use the procedure described here, rather than the old method
described in Sects. 5.2 and 4.3.4.

Quick instructions: You may compile with a default compiler-specific configuration:
1. Single-processor using the GNU compiler collection:
unix>

pc_build -f GNU-GCC

2. Multi-processor using GNU with MPI support:
unix>

pc_build -f GNU-GCC_MPI

Many compilers are supported already, please refer to the available config files in
‘$PENCIL_HOME/config/compilers/*.conf’, e.g. ‘Intel.conf’ and ‘Intel_MPI.conf’.
If you have to set up some compiler options specific to a certain host system you work on,
or if you like to create a host-specific configuration file so that you can simply execute
pc_build without any options, you can clone an existing host-file, just include an existing
compiler configuration, and simply only add the options you need. A good example of
a host-file is ‘$PENCIL_HOME/config/hosts/IWF/host-andromeda-GNU_Linux-Linux.conf’.
You may save a clone under ‘$PENCIL_HOME/config/hosts/.conf’, where ‘’ is to
be replaced by the output of pc_build -i. This will be the new default for pc_build.
If you don’t know what this was all about, read on.
In essence, configuration, compiling and running the code work like this:
1. Create a configuration file for your computer’s host ID.
2. Compile the code using pc_build.
3. Run the code using pc_run
In the following, we will discuss the essentials of this scheme. Exhaustive documentation is available with the commands perldoc Pencil::ConfigFinder and perldoc
PENCIL::ConfigParser.

5.1.1 Locating the configuration file
Commands like pc_build and pc_run use the Perl module ‘Pencil::ConfigFinder’ to locate an appropriate configuration file and ‘Pencil::ConfigParser’ to read and interpret
it. When you use ‘ConfigFinder’ on a given computer, it constructs a host ID for the system it is running on, and looks for a file ‘host_ID.conf’ in any subdirectory of ‘$PENCIL_HOME/config/hosts’.
E.g., if the host ID is “workhorse.pencil.org”, ‘ConfigFinder’ would consider the file
‘$PENCIL_HOME/config/hosts/pencil.org/workhorse.pencil.org.conf’.

18

T HE P ENCIL C ODE

Note 1: The location in the tree under ‘hosts/’ is irrelevant, which allows you to group
related hosts by institution, owner, hardware, etc.
Note 2: ‘ConfigFinder’ actually uses the following search path:
1. ‘./config’
2. ‘$PENCIL_HOME/config-local’
3. ‘$HOME/.pencil/config-local’
4. ‘$PENCIL_HOME/config’
This allows you to override part of the ‘config/’ tree globally on the given file system,
or locally for a particular run directory, or for one given copy of the P ENCIL C ODE. This
search path is used both, for locating the configuration file for your host ID, and for
locating included files (see below).
The host ID is constructed based on information that is easily available for your system.
The algorithm is as follows:
1. Most commands using ‘ConfigFinder’ have a ‘--host-id’ (sometimes abbreviated
as ‘-H’) option to explicitly set the host ID.
2. If the environment variable PENCIL HOST ID is set, its value is used.
3. If any of the files ‘./host-ID’, ‘$PENCIL_HOME/host-ID’, ‘$HOME/.pencil/host-ID’, exists, its first line is used.
4. If ‘ConfigFinder’ can get hold of a fully qualified host name , that is used as host
ID.
5. Else, a combination of host name, operating system name and possibly some other
information characterizing the system is used.
6. If no config file for the host ID is found, the current operating system name is tried
as fallback host ID.
To see which host IDs are tried (up to the first one for which a configuration file is found),
run
unix>

pc_build --debug-config

This command will tell you the host-ID of the system that you are using:
unix>

pc_build -i

5.1.2 Structure of configuration files
It is strongly recommended to include in a users configuration file one of the preset
compiler suite configuration files. Then, only minor options need to be set by a user, e.g.
the optimization flags. One of those user configuration files looks rather simple:
# Simple config file. Most files don’t need more.
%include compilers/GNU-GCC
or if you prefer a different compiler:

5.1

Configuring the code to compile and run on your computer

19

# Simple Intel compiler suite config file.
%include compilers/Intel
A more complex file (using MPI with additional options) would look like this:
# More complex config file.
%include compilers/GNU-GCC_MPI
%section Makefile
MAKE_VAR1 = -j4
# joined compilation with four threads
FFLAGS += -O3 -Wall -fbacktrace
# don’t redefine, but append with ’+=’
%endsection Makefile
%section runtime
mpiexec = mpirun
%endsection runtime

# some MPI backends need a redefinition of mpiexec

%section environment
SCRATCH_DIR=/var/tmp/$USER
%endsection environment
Adding ” MPI” to a compiler suite’s name is usually sufficient to use MPI.
Note 3: We strongly advise not to mix Fortran- and C-compilers from different manufacturers or versions by manually including multiple separate compiler configurations.
Note 4: We strongly advise to use at maximum the optimization levels ’-O2’ for the
Intel compiler and ’-O3’ for all other compilers. Higher optimization levels implicate an
inadequate loss of precision.
The ‘.conf’ files consist of the following elements:
Comments: A # sign and any text following it on the same line are ignored.
Sections: There are three sections:
Makefile for setting make parameters
runtime for adding compiler flags used by pc_run
environment shell environment variables for compilation and running
Include statements: An %include ... statement is recursively replaced by the contents of the files it points to.3
The included path gets a .conf suffix appended. Included paths are typically “absolute”, e.g.
%include os/Unix
will include the file ‘os/Unix.conf’ in the search path listed above (typically from
‘$PENCIL_HOME/config’). However, if the included path starts with a dot, it is a relative path, so
3

However, if the include statement is inside a section, only the file’s contents inside that section are
inserted.

20

T HE P ENCIL C ODE

%include ./Unix
will only search in the directory where the including file is located.
Assignments: Statements like FFLAGS += -O3 or mpiexec=mpirun are assignments and
will set parameters that are used by pc_build/make or pc_run.
Lines ending with a backslash ‘\’ are continuation lines.
If possible, one should always use incremental assignments, indicated by using a
+= sign instead of =, instead of redefining certain flags.
Thus,
CFLAGS += -O3
CFLAGS += -I../include -Wall
is the same as
CFLAGS = $(CFLAGS) -O3 -I../include -Wall
5.1.3 Compiling the code
Use the pc_build command to compile the code:
unix>
unix>
unix>
unix>
unix>

pc_build
pc_build
pc_build
pc_build
pc_build

-f Intel_MPI
-f os/GNU_Linux,mpi/open-mpi
VAR=something
--cleanall

#
#
#
#
#

use default compiler suite
specify a compiler suite
explicitly specify config files
set variables for the makefile
remove generated files

The third example circumvents the whole host ID mechanism by explicitly instructing
pc_build which configuration files to use. The fourth example shows how to define extra
variables (VAR=something) for the execution of the Makefile.
See pc_build --help for a complete list of options.
5.1.4 Running the code
Use the pc_run command to run the code:
unix>
unix>
unix>

pc_run
pc_run start
pc_run run

# start if necessary, then run

unix>
unix>
unix>

pc_run start run^3
pc_run start run run run
pc_run ^3

# start, then run 3 times
# start, then run 3 times
# start if necessary, then run 3 times

See pc_run --help for a complete list of options.
5.1.5 Testing the code
The pc_auto-test command uses pc_build for compiling and pc_run for running the
tests. Here are a few useful options:

5.2

Adapting ‘Makefile.src’file]Makefile.src@Makefile.src [obsolete; see Sect. 5.1]21

unix>
unix>
unix>
unix>

pc_auto-test
pc_auto-test --no-pencil-check
pc_auto-test --max-level=1
pc_auto-test --time-limit=2m

# suppress pencil consistency check
# run only tests in level 0 and 1
# kill each test after 2 minutes

See pc_auto-test --help for a complete list of options.
The pencil-test script will use pc_auto-test if given the ‘--use-pc_auto-test’ or ‘-b’
option:
unix>
unix>
unix>

pencil-test --use-pc_auto-test
pencil-test -b
# ditto
pencil-test -b
-Wa,--max-level=1,--no-pencil-check

# quick penc

See pencil-test --help for a complete list of options.
5.2

Adapting ‘Makefile.src’ [obsolete; see Sect. 5.1]

By default, one should use the above described configuration mechanism for compilation. If for whatever reason one needs to work with a modified ‘Makefile’, there is a
mechanism for picking the right set of compiler flags based on the hostname.
To give you an idea of how to add your own machines, let us assume you have several
Linux boxes running a compiler f95 that needs the options ‘-O2 -u’, while one of them,
Janus , runs a compiler f90 which needs the flags ‘-O3’ and requires the additional options ‘-lmpi -llam’ for MPI .
The ‘Makefile.src’ you need will have the following section:
### Begin machine dependent
## IRIX64:
[. . . ] (leave as it is in the original Makefile)
## OSF1:
[. . . ] (leave as it is in the original Makefile)
## Linux:
[. . . ] (leave everything from original Makefile and add:)
#FC=f95
#FFLAGS=-O2 -u
#FC=f90
#(Janus)
#FFLAGS=-O3
#(Janus)
#LDMPI=-lmpi -llam #(Janus)
## SunOS:
[. . . ] (leave as it is in the original Makefile)
## UNICOS/mk:
[. . . ] (leave as it is in the original Makefile)
## HI-UX/MPP:
[. . . ] (leave as it is in the original Makefile)
## AIX:
[. . . ] (leave as it is in the original Makefile)

22

T HE P ENCIL C ODE

Table 1: Compiler flags for common compilers. Note that some combinations of OS and compiler require
much more elaborate settings; also, if you use MPI, you will have to set LDMPI.

Compiler

FC

FFLAGS

CC

CFLAGS

Unix/POSIX:
GNU
Intel
PGI
G95
Absoft
IBM XL

gfortran
ifort
pgf95
g95
f95
xlf95

-O3
-O2
-O3
-O3 -fno-second-underscore
-O3 -N15
-qsuffix=f=f90 -O3

gcc
icc
pgcc
gcc
gcc
xlc

-O3
-O3
-O3
-O3
-O3
-O3

outdated:
IRIX Mips
Compaq

f90
f90

-64 -O3 -mips4
-fast -O3

cc
cc

-O3 -64 -DFUNDERSC=1
-O3 -DFUNDERSC=1

-DFUNDERSC=1
-DFUNDERSC=1
-DFUNDERSC=1
-DFUNDERSC=1
-DFUNDERSC=1
-DFUNDERSC=1

### End machine dependent

Note 1 There is a script for adapting the Makefile: ‘adapt-mkfile’. In the example
above, #(Janus) is not a comment, but marks this line to be activated (uncommented) by
adapt-mkfile if your hostname (‘uname -n‘) matches ‘Janus’ or ‘janus’ (capitalization
is irrelevant). You can combine machine names with a vertical bar: a line containing
#(onsager|Janus) will be activated on both, Janus and Onsager .

Note 2 If you want to experiment with compiler flags, or if you want to get things
running without setting up the machine-dependent section of the ‘Makefile’, you can set
make variables at the command line in the usual manner:
src>

make FC=f90 FFLAGS=’-fast -u’

will use the compiler f90 and the flags ‘-fast -u’ for both compilation and linking. Table 1 summarizes flags we use for common compilers.

5.3

Changing the resolution

It is advisable to produce a new run directory each time you run a new case. (This does
not include restarts from an old run, of course.) If you have a 323 run in some directory
‘runA_32a’, then go to its parent directory, i.e.
runA_32a> cd ..
forced> pc_newrun runA_32a runA_64a
forced> cd runA_64a/src
forced> vi cparam.local
and edit the ‘cparam.local’ for the new resolution.
If you have ever wondered why we don’t do dynamic allocation of the main variable (f)
array, the main reason it that with static allocation the compiler can check whether we
are out of bounds.

5.4 Using a non-equidistant grid

5.4

23

Using a non-equidistant grid

We introduce a non-equidistant grid zi (by now, this is also implemented for the other
directions) as a function z(ζ) of an equidistant grid ζi with grid spacing ∆ζ = 1.
The way the parameters are handled, the box size and position are not changed when
you switch to a non-equidistant grid, i.e. they are still determined by xyz0 and Lxyz .
The first and second derivatives can be calculated by
df dζ
1
df
=
= ′ f ′ (ζ) ,
dz
dζ dz
z

d2 f
1 ′′
z ′′ ′
=
f
(ζ)
−
f (ζ) ,
dz 2
z ′2
z ′3

(1)

which can be written somewhat more compactly using the inverse function ζ(z):
d2 f
= ζ ′2 (z) f ′′ (ζ) + ζ ′′ (z)ζ ′ (z)f ′ (ζ) .
dz 2

df
= ζ ′ (z) f ′ (ζ) ,
dz

(2)

Internally, the code uses the quantities dz 1 ≡ 1/z ′ = ζ ′ (z) and dz tilde ≡ −z ′′ /z ′2 =
ζ ′′ /ζ ′ , and stores them in ‘data/procN /grid.dat’.
The parameters lequidist (a 3-element logical array), grid func , coeff grid (a
≥ 2-element real array) are used to choose a non-equidistant grid and define
the function z(ζ). So far, one can choose between grid_function=’sinh’, grid_function=’linear’ (which produces an equidistant grid for testing purposes), and
grid_function=’step-linear’.
The sinh profile: For grid_function=’sinh’, the function z(ζ) is given by
z(ζ) = z0 + Lz

sinh a(ζ−ζ∗ ) + sinh a(ζ∗ −ζ1 )
,
sinh a(ζ2 −ζ∗ ) + sinh a(ζ∗ −ζ1 )

(3)

where z0 and z0 +Lz are the lowest and uppermost levels, ζ1 , ζ2 are the ζ values representing those levels (normally ζ1 = 0, ζ2 = Nz − 1 for a grid of Nz vertical layers [excluding
ghost layers]), and ζ∗ is the ζ value of the inflection point of the sinh function. The z
coordinate and ζ value of the inflection point are related via
z ∗ = z 0 + Lz

sinh a(ζ∗ −ζ1 )
,
sinh a(ζ2 −ζ∗ ) + sinh a(ζ∗ −ζ1 )

which can be inverted (“after some algebra”) to



ζ1 +ζ2 1
z∗ −z0
ζ2 −ζ1
ζ∗ =
.
+ artanh 2
− 1 tanh a
2
a
Lz
2

(4)

(5)

General profile: For a general monotonic function ψ() instead of sinh we get,
z(ζ) = z0 + Lz

ψ[a(ζ−ζ∗ )] + ψ[a(ζ∗ −ζ1 )]
,
ψ[a(ζ2 −ζ∗ )] + ψ[a(ζ∗ −ζ1 )]

(6)

and the reference point ζ∗ is found by inverting
z ∗ = z 0 + Lz
numerically.

ψ[a(ζ∗ −ζ1 )]
,
ψ[a(ζ2 −ζ∗ )] + ψ[a(ζ∗ −ζ1 )]

(7)

24

T HE P ENCIL C ODE

Duct flow: The profile function grid_function=’duct’ generates a grid profile for turbulent flow in ducts. For a duct flow, most gradients are steepest close to the walls, and
hence very fine resolution is required near the walls. Here we follow the method of [21]
and use a Chebyshev-type grid with a cosine distribution of the grid points such that in
the y direction they are located at
yj = h cos θj ,
where
θj =

(Ny −j) π
,
Ny −1

j = 1, 2, . . . , Ny

(8)

(9)

and h = Ly /2 is the duct half-width.

Currently this method is only adapted for ducts where x is the stream-wise direction, z
is in the span-wise direction and the walls are at y = y0 and y = y0 + Ly .
In order to have fine enough resolution, the first grid point should be a distance δ =
0.05 lw from the wall, where
r
ν
τw
lw =
,
(10)
,
uτ =
uτ
ρ
and τw is the shear wall stress. This is accomplished by using at least
π
+1

δ
arccos 1 −
h r
"  #
r
3/2
h
π 2δ
δ
= π
+1−
+O
2δ
24 h
h

Ny ≥ Ny∗ =

(11)

(12)

grid points in the y-direction. After rounding up to the next integer value, we find that
the truncated condition
& r '
h
+1
(13)
Ny ≥ π
2δ
(where ⌈x⌉ denotes the ceiling function, i.e. the smallest integer equal to, or larger than,
x) gives practically identical results.
Example: To apply the sinh profile, you can set the following in ‘start.in’ (this example is from ‘samples/sound-spherical-noequi’):
&init_pars
[...]
xyz0 = -2., -2., -2.
Lxyz = 4., 4., 4.
lperi = F , F , F
lequidist = F, F, T
xyz_star
= , , -2.
grid_func = , , ’sinh’
coeff_grid = , , 0.5
/

!
!
!
!
!
!
!

first corner of box
box size
periodic direction?
non-equidistant grid in z
position of inflection point
sinh function: linear for small, but
exponential for large z

5.5

Diagnostic output

25

The parameter array coeff grid represents z∗ and a; the bottom height z0 and the total
box height Lz are taken from xyz0 and Lxyz as in the equidistant case. The inflection
point of the sinh profile (the part where it is linear) is not in the middle of the box,
because we have set xyz star(3) (i. e. z∗ ) to −2.
5.5

Diagnostic output

Every it1 time steps (it1 is a runtime parameter, see Sect. J.2), the code writes monitoring output to stdout and, parallel to this, to the file ‘data/time_series.dat’. The variables that appear in this listing and the output format are defined in the file ‘print.in’
and can be changed without touching the code (even while the code is running). A simple
example of ‘print.in’ may look like this:
t(F10.3)
urms(E13.4)
rhom(F10.5)
oum
which means that the output table will contain time t in the first column formatted as
1/2
(F10.3), followed by the mean squared velocity, urms (i.e. hu2 i ) in the second column
with format (E13.4), the average density rhom (hρi, which allows to monitor mass conservation) formatted (F10.5) and the kinetic helicity oum (that is hω · ui) in the last
column with the default format (E10.2).4 The corresponding diagnostic output will look
like
----t---------urms--------rhom------oum---0.000
4.9643E-03 14.42457 -8.62E-06
0.032
3.9423E-03 14.42446 -5.25E-06
0.063
6.8399E-03 14.42449 -3.50E-06
0.095
1.1437E-02 14.42455 -2.58E-06
0.126
1.6980E-02 14.42457 -1.93E-06
5.6

Data files

5.6.1 Snapshot files
Snapshot files contain the values of all evolving variables and are sufficient to restart a
run. In the case of an MHD simulation with entropy equation, for example, the snapshot
files will contain the values of velocity, logarithmic density, entropy and the magnetic
vector potential.
There are two kinds of snapshot files: the current snapshot and permanent snapshots,
both of which reside in the directory ‘data/procN /’. The parameter isav determines the
frequency at which the current snapshot ‘data/procN /var.dat’ is written. If you keep
this frequency too high, the code will spend a lot of time on I/O, in particular for large
jobs; too low a frequency makes it difficult to follow the evolution interactively during
test runs. There is also the ialive parameter. Setting this to 1 or 10 gives an updated
4

The format specifiers are like in Fortran, apart from the fact that the E format will use standard
scientific format, corresponding to the Fortran 1pE syntax. Seasoned Fortran IV programmers may use
formats like (0pE13.4) to enjoy nostalgic feelings, or (1pF10.5) if they depend on getting wrong numbers.

26

T HE P ENCIL C ODE

timestep in the files ‘data/proc*/alive.info’. You can put ialive=0 to turn this off to
limit the I/O on your machine.
The permanent snapshots ‘data/proc*/VARN ’ are written every dsnap time units. These
files are numbered consecutively from N = 1 upward and for long runs they can occupy
quite some disk space. On the other hand, if after a run you realize that some additional quantity q would have been important to print out, these files are the only way to
reconstruct the time evolution of q without re-running the code.
File structure Snapshot files consist of the following Fortran records 5 :
1. variable vector f [mx ×my ×mz ×nvar ]
2. time t [1], coordinate vectors x [mx ], y [my ], z [mz ], grid spacings δx [1], δy [1], δz
[1], shearing-box shift ∆y [1]
All numbers (apart from the record markers) are single precision (4-byte) floating point
numbers, unless you use double precision (see §5.21, in which case all numbers are 8byte floating point numbers, while the record markers remain 4-byte integers.
The script pc_tsnap allows you to determine the time t of a snapshot file:
unix> pc_tsnap data/proc0/var.dat
data/proc0/var.dat:
t = 8.32456
unix> pc_tsnap data/proc0/VAR2
data/proc0/VAR2:
t = 2.00603

5.7

Video files and slices

We use the terms video files and slice files interchangeably. These files contain a time
series of values of one variable in a given plane. The output frequency of these video
snapshots is set by the parameter dvid (in code time units).
When output to video files is activated by some settings in ‘run.in’ (see example below)
and the existence of ‘video.in’, slices are written for four planes:
1. x-z plane (y index iy ; file suffix .xz)
2. y-z plane (y index ix ; suffix .yz)
3. x-y plane (y index iz ; suffix .xy)
4. another slice parallel to the x-y plane (y index iz2 ; suffix .xy2)
You can specify the position of the slice planes by setting the parameters ix , iy , iz and
iz2 in the namelist run pars in ‘run.in’. Alternatively, you can set the input parameter
slice position to one of ’p’ (periphery of box) or ’m’ (middle of box). Or you can also
specify the z−position in terms of z using the tags zbot slice and ztop slice . In this case,
the zbot slice slice will have the suffix .xy and the ztop slice the suffix .xy2
In the file ‘video.in’ of your run directory, you can choose for which variables you want
to get video files; valid choices are listed in § J.4.
5

A Fortran record is marked by the 4-byte integer byte count of the data in the record at the beginning
and the end, i.e. has the form hNbytes , raw data, Nbytes i

5.7

Video files and slices

27

The slice files are written in each processor directory ‘data/proc*/’ and have a file name
indicating the individual variable (e. g. ‘slice_uu1.yz’ for a slice of ux in the y-z plane).
Before visualizing slices one normally wants to combine the sub-slices written by each
processor into one global slice (for each plane and variable). This is done by running
‘src/read_videofiles.x’, which will prompt for the variable name, read the individual
sub-slices and write global slices to ‘data/’ Once all global slices have been assembled
you may want to remove the local slices ‘data/proc*/slice*’.
To read all sub-slices demanded in ‘video.in’ at once use ‘src/read_all_videofiles.x’.
This program doesn’t expect any user input and can thus be submitted in computing
queues.
For visualization of slices, you can use the IDL routines ‘rvid_box.pro’, ‘rvid_plane.pro’, or ‘rvid_line.pro’ which allows the flag ‘/png’ for writing PNG images that
can then be combined into an MPEG movie using mpeg encode . Based on ‘rvid_box’,
you can write your own video routines in IDL .
An example Suppose you have set up a run using entropy.f90 and magnetic.f90 (most
probably together with hydro.f90 and other modules). In order to animate slices of entropy s and the z-component Bz of the magnetic field, in planes passing through the
center of the box, do the following:
1. Write the following lines to ‘video.in’ in your run directory:
ss
bb
divu
2. Edit the namelist run pars in the file ‘run.in’. Request slices by setting write slices
and set dvid and slice position to reasonable values, say
!lwrite_slices=T !(no longer works; write requested slices into video.in)
dvid=0.05
slice_position=’m’
3. Run the P ENCIL C ODE:
unix>
unix>

start.csh
run.csh

4. Run ‘src/read_videofiles.x’ to assemble global slice files from those scattered
across ‘data/proc*/’:
unix> src/read_videofiles.x
enter name of variable (lnrho, uu1, ..., bb3):
unix> src/read_videofiles.x
enter name of variable (lnrho, uu1, ..., bb3):
5. Start IDL and run ‘rvid_box’:
unix> idl
IDL> rvid_box,’bb3’
IDL> rvid_box,’ss’,min=-0.3,max=2.
etc.

ss
bb3

28

T HE P ENCIL C ODE

Another example Suppose you have set up a run using magnetic.f90 and some other
modules. This run studies some process in a “surface” layer inside the box. This “surface”
can represent a sharp change in density or turbulence. So you defined your box setting
the z = 0 point at the surface. Therefore, your ‘start.in’ file will look something similar
to:
&init_pars
lperi=T,T,F
bcz = ’s’,’s’,’a’,’hs’,’s’,’s’,’a’
xyz0 = -3.14159, -3.14159, -3.14159
Lxyz = 6.28319, 6.28319, 9.42478
Now you can analyze quickly the surface of interest and some other xy−slice setting
zbot slice and ztop slice in the ‘run.in’ file:
&run_pars
slice_position=’c’
zbot_slice=0.
ztop_slice=0.2
In this case, the slices with the suffix .xy will be at the “surface” and the ones with the
suffix .xy2 will be at the position z = 0.2 above the surface. And you can visualize this
slices by:
1. Write the following lines to ‘video.in’ in your run directory:
bb
2. Edit the namelist run pars in the file ‘run.in’ to include zbot slice and ztop slice .
3. Run the P ENCIL C ODE:
unix>
unix>

start.csh
run.csh

4. Run ‘src/read_videofiles.x’ to assemble global slice files from those scattered
across ‘data/proc*/’:
unix> src/read_videofiles.x
enter name of variable (lnrho, uu1, ..., bb3):

bb3

5. Start IDL , load the slices with ‘pc_read_video’ and plot them at some time:
unix> idl
IDL> pc_read_video, field=’bb3’,ob=bb3,nt=ntv
IDL> tvscl,bb3.xy(*,*,100)
IDL> tvscl,bb3.xy2(*,*,100)
etc.
File structure Slice files consist of one Fortran record (see footnote on page 26) for
each slice, which contains the data of the variable (without ghost zones), the time t of
the snapshot and the position of the sliced variable (e. g. the x position for a y-z slice):
1. data1 [nx ×ny ×nz ], time t1 [1], position1 [1]

5.8

Averages

29

2. data2 [nx ×ny ×nz ], time t2 [1], position2 [1]
3. data3 [nx ×ny ×nz ], time t3 [1], position3 [1]
etc.

5.8

Averages

5.8.1 One-dimensional output averaged in two dimensions
In the file ‘xyaver.in’, z-dependent (horizontal) averages are listed. They are written to
the file ‘data/xyaverages.dat’. A new line of averages is written every it1 th time steps.
There is the possibility to output two-dimensional averages. The result then depends on the remaining dimension. The averages are listed in the files ‘xyaver.in’,
‘xzaver.in’, and ‘yzaver.in’ where the first letters indicate the averaging directions.
The output is then stored to the files ‘data/xyaverages.dat’, ‘data/xzaverages.dat’, and
‘data/yzaverages.dat’. The output is written every it1d th time steps.
The rms values of the so defined mean magnetic fields are referred to as bmz , bmy and
bmx , respectively, and the rms values of the so defined mean velocity fields are referred
to as umz , umy , and umx . (The last letter indicates the direction on which the averaged
quantity still depends.)
See Sect. 9.2 on how to add new averages.
In idl such xy-averages can be read using the procedure ‘pc_read_xyaver’.

5.8.2 Two-dimensional output averaged in one dimension
There is the possibility to output one-dimensional averages. The result then depends
on the remaining two dimensions. The averages are listed in the files ‘yaver.in’,
‘zaver.in’, and ‘phiaver.in’ where the first letter indicates the averaging direction.
The output is then stored to the files ‘data/yaverages.dat’, ‘data/zaverages.dat’, and
‘data/phiaverages.dat’.
See Sect. 9.2 on how to add new averages.
Disadvantage: The output files, e.g. ‘data/zaverages.dat’, can be rather big because each
average is just appended to the file.

5.8.3 Azimuthal averages
Azimuthal averages are controlled by the file ‘phiaver.in’, which currently supports the
quantities listed in Sect. J.5. In addition, one needs to set lwrite phiaverages , lwrite yaverages , or lwrite zaverages to .true.. For example, if ‘phiaver.in’ contains the single
line
b2mphi
then you will get azimuthal averages of the squared magnetic field B 2 .

30

T HE P ENCIL C ODE

Azimuthal averages are written every d2davg time units to the files
‘data/averages/PHIAVGN ’. The file format of azimuthal-average files consists of the
following Fortran records :
1. number of radial points Nr,φ−avg [1], number of vertical points Nz,φ−avg [1], number
of variables Nvar,φ−avg [1], number of processors in z direction [1]
2. time t [1], positions of cylindrical radius rcyl [Nr,φ−avg ] and z [Nz,φ−avg ] for the grid,
radial spacing δrcyl [1], vertical spacing δz [1]
3. averaged data [Nr,φ−avg ×Nz,φ−avg ]
4. label length [1], labels of averaged variables [Nvar,φ−avg ]
All numbers are 4-byte numbers (floating-point numbers or integers), unless you use
double precision (see §5.21).
To read and visualize azimuthal averages in IDL , use ‘$PENCIL_HOME/idl/files/pc_read_phiavg.pro’
IDL>
IDL>

avg = pc_read_phiavg(’data/averages/PHIAVG1’)
contour, avg.b2mphi, avg.rcyl, avg.z, TITLE=’!17B!U2!N!X’

or have a look at ‘$PENCIL_HOME/idl/phiavg.pro’ for a more sophisticated example.
5.8.4 Time averages
Time averages need to be prepared in the file ‘src/ctimeavg.local’, since they use extra
memory. They are controlled by the averaging time τavg (set by the parameter tavg in
‘run.in’), and by the indices idx tavg of variables to average.
Currently, averaging is implemented as exponential (memory-less) average,6
hf it+δt = hf it +

δt
[f (t+δt) − hf it ] ,
τavg

(16)

which is equivalent to
hf it =

Zt

′

e−(t−t )/τavg f (t′ ) dt′ .

(17)

t0

Here t0 is the time of the snapshot the calculation started with, i.e. the snapshot read by
the last run.x command. Note that the implementation (16) will approximate Eq. (17)
only to first-order accuracy in δt. In practice, however, δt is small enough to make this
accuracy suffice.
6

At some point we may also implement the more straight-forward average
hf it+δt = hf it +

δt
[f (t+δt) − hf it ] ,
t−t0 +δt

(14)

which is equivalent to
1
hf it =
t − t0
but we do not expect large differences.

Zt

t0

f (t′ ) dt′ ,

(15)

5.9

Helper scripts

31

In ‘src/ctimeavg.local’, you need to set the number of slots used for time averages.
Each of these slots uses mx × my × mz floating-point numbers, i.e. half as much memory
as each fundamental variable.
For example, if you want to get time averages of all variables, set
integer, parameter :: mtavg=mvar
in ‘src/ctimeavg.local’, and don’t set idx tavg in ‘run.in’.
If you are only interested in averages of variables 1–3 and 6–8 (say, the velocity vector
and the magnetic vector potential in a run with ‘hydro.f90’, ‘density.f90’, ‘entropy.f90’
and ‘magnetic.f90’), then set
integer, parameter :: mtavg=6
in ‘src/ctimeavg.local’, and set
idx_tavg = 1,2,3,6,7,8

! time-average velocity and vector potential

in ‘run.in’.
Permanent snapshots of time averages are written every tavg time units to
the files ‘data/proc*/TAVN ’. The current time averages are saved periodically in
‘data/proc*/timeavg.dat’ whenever ‘data/proc*/var.dat’ is written. The file format for
time averages is equivalent to that of the snapshots; see § 5.6.1 above.
5.9

Helper scripts

The ‘bin’ directory contains a collection of utility scripts, some of which are discussed
elsewhere, Here is a list of the more important ones.
adapt-mkfile Activate the settings in a ‘Makefile’ that apply to the given computer,
see §5.2.
auto-test Verify that the code compiles and runs in a set of run directories and compare
the results to the reference output. These tests are carried out routinely to ensure
that the svn version of the code is in a usable state.
cleanf95 Can be use to clean up the output from the Intel x86 Fortran 95 compiler
(ifc).
copy-proc-to-proc Used for restarting in
copy-proc-to-proc seed.dat ../hydro256e.

a

different

directory.

Example

copy-snapshots Copy snapshots from a processor-local directory to the global directory. To be started in the background before ‘run.x’ is invoked. Used by ‘start.csh’
and ‘run.csh’ on network connected processors.
pc copyvar var1 var2 source dest Copies snapshot files from one directory (source)
to another (dest). See documentation in file.
pc copyvar v v dir Copies all ‘var.dat’ files from current directory to ‘var.dat’ in ’dir’
run directory. Used for restarting in a different directory.

32

T HE P ENCIL C ODE

pc copyvar N v Used to restart a run from a particular snapshot ‘VARN ’. Copies a
specified snapshot ‘VARN ’ to ‘var.dat’ where N and (optionally) the target run directory are given on the command line.
cvs-add-rundir Add the current run directory to the svn repository.
cvsci run Similar to cvs-add-rundir, but it also checks in the ‘*.in’ and ‘src/*.local’
files. It also checks in the files ‘data/time_series.dat’, ‘data/dim.dat’ and
‘data/index.pro’ for subsequent processing in IDL on another machine. This is
particularly useful if collaborators want to check each others’ runs.
dx * These script perform several data collection or reformatting exercises required to
read particular files into DX . They are called internally by some of the DX macros
in the ‘dx/macros/’ directory.
getconf.csh See § 4.3.4
gpgrowth Plot simple time evolution with Gnuplot’s ASCII graphics for fast orientation via a slow modem line.
local Materialize a symbolic link
mkcparam Based on ‘Makefile’ and ‘Makefile.local’, generate ‘src/cparam.inc’,
which specifies the number mvar of fundamental variables, and maux of auxiliary
variables. Called by the ‘Makefile’.
pc mkdatadir Creates a link to a data directory in a suitable workspace. By default
this is on ‘/var/tmp/’, but different locations are specified for different machines.
mkdotin Generate minimal ‘start.in’, ‘run.in’ files based on ‘Makefile’ and
‘Makefile.local’.
mkinpars Wrapper around ‘mkdotin’ — needs proper documentation.
mkproc-tree Generates a multi-processor(‘procN ’/), directory structure. Useful when
copying data files in a processor tree, such as slice files.
mkwww Generates a template HTML file for describing a run of the code, showing
input parameters and results.
move-slice Moves all the slice files from a processor tree structure, ‘procN /’, to a new
target tree creating directories where necessary.
nl2idl Transform a Fortran namelist (normally the files ‘param.nml’, ‘param2.nml’ written by the code) into an IDL structure. Generates an IDL file that can be sourced
from ‘start.pro’ or ‘run.pro’.
pacx-adapt-makefile Version of adapt-makefile for highly distributed runs using
PACX MPI.
pc newrun Generates a new run directory from an old one. The new one contains a
copy of the old ‘*.local’ files, runs pc_setupsrc, and makes also a copy of the old
‘*.in’ and ‘k.dat’ files.
pc newscan Generates a new scan directory from an old one. The new one contains
a copy of the old, e.g. the one given under ‘samples/parameter_scan’. Look in the
‘README’ file for details.

5.10

RELOAD and STOP files

33

pc inspectrun Check the execution of the current run: prints legend and the last few
lines of the ‘time_series.dat’ file. It also appends this result to a file called ‘SPEED’,
which contains also the current wall clock time, so you can work out the speed of
the code (without being affected by i/o time).
read videofiles.csh The script for running read videofiles.x.
remote-top Create a file ‘top.log’ in the relevant ‘procN /’ directory containing the
output of top for the appropriate processor. Used in batch scripts for multiprocessor runs.
run.csh The script for producing restart files with the initial condition; see § 4.3.4
scpdatadir Make a tarball of data directory, ‘data/’ and use scp to secure copy to copy
it to the specified destination.
pc setupsrc Link ‘start.csh’, ‘run.csh’ and ‘getconf.csh’ from ‘$PENCIL_HOME/bin’.
Generate ‘src/’ if necessary and link the source code files from ‘$PENCIL_HOME/src’
to that directory.
start.csh The script for initializing the code; see § 4.3.4
summarize-history Evaluate ‘params.log’ and print a history of changes.
timestr Generate a unique time string that can be appended to file names from shell
scripts through the backtick mechanism.
pc tsnap Extract time information from a snapshot file, ‘VARN ’.
There are several additional scripts on ‘pencil-code/utils’. Some are located in separate folders according to users. There could be redundancies, but it is often just as easy
to write your own new script than figuring out how something else works.

5.10 RELOAD and STOP files
The code periodically (every it time steps) checks for the existence of two files, ‘RELOAD’
and ‘STOP’, which can be used to trigger certain behavior.

Reloading run parameters In the directory where you started the code, create the file
‘RELOAD’ with
unix>

touch RELOAD

34

T HE P ENCIL C ODE

to force the code to re-read the runtime parameters from ‘run.in’. This will happen the
next time the code is writing monitoring output (the frequency of this happening is
controlled by the input parameter it , see Sect. 5.12).
Each time the parameters are reloaded, the new set of parameters is appended (in the
form of namelists ) to the file ‘data/params.log’ together with the time t, so you have a
full record of your changes. If ‘RELOAD’ contains any text, its first line will be written to
‘data/params.log’ as well, which allows you to annotate changes:
unix>

echo "Reduced eta to get fields growing" > RELOAD

Use the command summarize-history to print a history of changes.
Stopping the code In the directory where you started the code, create the file ‘STOP’
with
unix>

touch STOP

to stop the code in a controlled manner (it will write the latest snapshot). Again, the
action will happen the next time the code is writing monitoring output.
5.11

RERUN and NEWDIR files

After the code finishes (e.g., when the final timestep number is reached or when a ‘STOP’
file is found), the ‘run.csh’ script checks whether there is a ‘RERUN’ file. If so, the code will
simply run again, perhaps even after you have recompiled the code. This is useful in the
development phase when you changed something in the code, so you don’t need to wait
for a new slot in the queue!
Even more naughty, as Tony says, is the ‘NEWDIR’ file, where you can enter a new directory path (relative path is ok, e.g. ../conv-slab). If nothing is written in this file (e.g.
via touch NEWDIR) it stays in the same directory. On distributed machines, the ‘NEWDIR’
method will copy all the ‘VAR#’ and ‘var.dat’ files back to and from the sever. This can be
useful if you want to run with new data files, but you better do it in a separate directory,
because with ‘NEWDIR’ the latest data from the code are written back to the server before
running again.
Oh, by the way, if you want to be sure that you haven’t messed up the content of the pair
of ‘NEWDIR’ files, you may want to try out the pc_jobtransfer command. It writes the
decisive ‘STOP’ file only after the script has checked that the content of the two ‘NEWDIR’
files points to existing run directory paths, so if the new run crashes, the code returns
safely to the old run directory. I’m not sure what Tony would say now, but this is now
obviously extremely naughty.
5.12

Start and run parameters

All input parameters in ‘start.in’ and ‘run.in’ are grouped in Fortran namelists . This
allows arbitrary order of the parameters (within the given namelist; the namelists need
no longer be in the correct order), as well as enhanced readability through inserted Fortran comments and whitespace. One namelist (init pars / run pars ) contains general
parameters for initialization/running and is always read in. All other namelists are specific to individual modules and will only be read if the corresponding module is used.

5.12 Start and run parameters

35

The syntax of a namelist (in an input file like ‘start.in’) is
&init_pars
ip=5, Lxyz=2,4,2
/
— in this example, the name of the namelist is init pars , and we read just two variables
(all other variables in the namelist retain their previous value): ip , which is set to 5, and
Lxyz , which is a vector of length three and is set to (2, 4, 2).
While all parameters from the namelists can be set, in most cases reasonable default
values are preset. Thus, the typical file ‘start.in’ will only contain a minimum set of
variables or (if you are very minimalistic) none at all. If you want to run a particular
problem, it is best to start by modifying an existing example that is close to your application.
Before starting a simulation run, you may want to execute the command pc_configtest
in order to test the correctness of your changes to these configuration files.
As an example, we give here the start parameters for ‘samples/helical-MHDturb’
&init_pars
cvsid=’$Id:$’,
! identify version of start.in
xyz0 = -3.1416, -3.1416, -3.1416, ! first corner of box
Lxyz = 6.2832, 6.2832, 6.2832, ! box size
lperi = T
, T
, T
, ! periodic in x, y, z
random_gen=’nr_f90’
/
&hydro_init_pars
/
&density_init_pars
gamma=1.
/
&magnetic_init_pars
initaa=’gaussian-noise’, amplaa=1e-4
/
The three entries specifying the location, size and periodicity of the box are just given
for demonstration purposes here — in fact a periodic box from −π to −π in all three
directions is the default. In this run, for reproducibility, we use a random number generator from the Numerical Recipes [27], rather than the compiler’s built-in generator.
The adiabatic index γ is set explicitly to 1 (the default would have been 5/3) to achieve
an isothermal equation of state. The magnetic vector potential is initialized with uncorrelated, normally distributed random noise of amplitude 10−4 .
The run parameters for ‘samples/helical-MHDturb’ are
&run_pars
cvsid=’$Id:$’,
! identify version of start.in
nt=10, it1=2, cdt=0.4, cdtv=0.80, isave=10, itorder=3
dsnap=50, dvid=0.5
random_gen=’nr_f90’
/
&hydro_run_pars
/

36

T HE P ENCIL C ODE

&density_run_pars
/
&forcing_run_pars
iforce=’helical’, force=0.07, relhel=1.
/
&magnetic_run_pars
eta=5e-3
/
&viscosity_run_pars
nu=5e-3
/
Here we run for nt = 10 timesteps, every second step, we write a line of diagnostic output;
we require the time step to keep the advective Courant number ≤ 0.4 and the diffusive
Courant number ≤ 0.8, save ‘var.dat’ every 20 time steps, and use the 3-step timestepping scheme described in Appendix H.4 (the Euler scheme itorder = 1 is only useful
for tests). We write permanent snapshot file ‘VARN ’ every dsnap = 50 time units and 2d
slices for animation every dvid = 0.5 time units. Again, we use a deterministic random
number generator. Viscosity ν and magnetic diffusivity η are set to 5 × 10−3 (so the mesh
Reynolds number is about urms δx/ν = 0.3×(2π/32)/5×10−3 ≈ 12, which is in fact rather a
bit to high). The parameters in forcing run pars specify fully helical forcing of a certain
amplitude.
A full list of input parameters is given in Appendix J.

5.13

Physical units

Many calculations are unit-agnostic, in the sense that all results remain the same independent of the unit system in which you interpret the numbers. E. g. if you simulate
a simple hydrodynamical flow in a box of length L = 1. and get a maximum velocity of
umax = 0.5 after t = 3 time units, then you may interpret this as L = 1 m, umax = 0.5 m/s,
t = 3 s, or as L = 1 pc, umax = 0.5 pc/Myr, t = 3 Myr, depending on the physical system you
have in mind. The units you are using must of course be consistent, thus in the second
example above, the units for diffusivities would be pc2 /Myr, etc.
The units of magnetic field and temperature are determined by the values µ0 = 1 and
cp = 1 used internally by the code7 . This means that if your units for density and velocity
are [ρ] and [v], then magnetic fields will be in
[B] =
and temperatures are in
[T ] =

p
µ0 [ρ] [v]2 ,

γ−1 [v]2
[v]2
.
=
cp
γ R/µ

(18)

(19)

For some choices of density and velocity units, Table 2 shows the resulting units of
magnetic field and temperature.
7

Note that cp = 1 is only assumed if you use the module noionization.f90 . If you work with ionization.f90 , temperature units are specified by unit temperature as described below.

5.14 Minimum amount of viscosity

37

Table 2: Units of magnetic field and temperature for some choices of [ρ] and [v] according to Eqs. (18) and
(19). Values are for a monatomic gas (γ = 5/3) of mean atomic weight µ̄g = µ̄/1 g in grams.

[ρ]

[v]

[B]

1 kg/m3

1 m/s

1.12 mT = 11.2 G

1 g/cm3

1 cm/s

3.54 × 10−4 T = 3.54 G

1 g/cm3

1 km/s

35.4 T = 354 kG

1 g/cm3

10 km/s

354 T = 3.54 MG

[T ]
 µ̄ 
g

× 2.89 × 10−5 K
0.6
 µ̄ 
g
× 2.89 nK
0.6
 µ̄ 
g
× 28.9 K
0.6
 µ̄ 
g
× 2 890 K
0.6

On the other hand, as soon as material equations are used (e.g. one of the popular parameterizations for radiative losses, Kramers opacity, Spitzer conductivities or ionization, which implies well-defined ionization energies), the corresponding routines in the
code need to know the units you are using. This information is specified in ‘start.in’ or
‘run.in’ through the parameters unit system , unit length , unit velocity , unit density
and unit temperature 8 like e. g.
unit_system=’SI’,
unit_length=3.09e16, unit_velocity=978.

! [l]=1pc, [v]=1pc/Myr

Note: The default unit system is unit_system=’cgs’ which is a synonym for unit_system=’Babylonian cubits’.

5.14 Minimum amount of viscosity
We emphasize that, by default, the code works with constant diffusion coefficients (viscosity ν, thermal diffusivity χ, magnetic diffusivity η, or passive scalar diffusivity D).
If any of these numbers is too small, you would need to have more meshpoints to get
consistent numerical solutions; otherwise the code develops wiggles (‘ringing’) and will
eventually crash. A useful criterion is given by the mesh Reynolds number based on the
maximum velocity,
Remesh = max(|u|) max(δx, δy, δz)/ν,
(20)
which should not exceed a certain value which can be problem-dependent. Often the
largest possible value of Remesh is around 5. Similarly there exist mesh Péclet and mesh
magnetic Reynolds numbers that should not be too large.
Note that in some cases, ‘wiggles’ in ln ρ will develop despite sufficiently large diffusion
coefficients, essentially because the continuity equation contains no dissipative term.
For convection runs (but not only for these), we have found that this can often be prevented by upwinding , see Sect. H.2.
If the Mach number of the code approaches unity, i.e. if the rms velocity becomes comparable with the speed of sound, shocks may form. In such a case the mesh Reynolds
number should be smaller. In order to avoid excessive viscosity in the unshocked regions,
8

Note: the parameter unit temperature is currently only used in connection with ionization.f90 . If you
are working with noionization.f90 , the temperature unit is completely determined by Eq. (19) above.

38

T HE P ENCIL C ODE

one can use the so-called shock viscosity (Sect. 6.6.1) to concentrate the effects of a low
mesh Reynolds number to only those areas where it is necessary.
5.15

The time step

5.15.1 The usual RK-2N time step
RK-2N refers to the third order Runge-Kutta scheme by Williamson (1980) with a memory consumption of two chunks. Therefore the 2N in the name.
The time step is normally specified as Courant time step through the coefficients cdt
(cδt ), cdtv (cδt,v ) and cdts (cδt,s ). The resulting Courant step is given by
δxmin
δx2
1
δt = min cδt
, cδt,v min , cδt,s
Umax
Dmax
Hmax




,

(21)

where
δxmin ≡ min(δx, δy, δz) ;


q
2
Umax ≡ max |u| + c2s +vA ,

(22)

Dmax = max(ν, γχ, η, D),

(24)

(23)

cs and vA denoting sound speed and Alfvén speed, respectively;

where ν denotes kinematic viscosity, χ = K/(cp ρ) thermal diffusivity and η the magnetic
diffusivity; and


2νS2 + ζshock (∇ · u)2 + ...
,
(25)
Hmax = max
cv T
where dots indicate the presence of other terms on the rhs of the entropy equation.
To fix the time step δt to a value independent of velocities and diffusivities, explicitly set
the run parameter dt , rather than cdt or cdtv (see p. 172).
If the time step exceeds the viscous time step the simulation may actually run ok for
quite some time. Inspection of images usually helps to recognize the problem. An example is shown in Fig. 4.
Timestepping is accomplished using the Runge-Kutta 2N scheme. Regarding details of
this scheme see Sect. H.4.
5.15.2 The Runge-Kutta-Fehlberg time step
A fifth order Runge-Kutta-Fehlberg time stepping procedure is available. It is used
mostly for chemistry application, often together with the double precision option. In
order to make this work, you need to compile with
TIMESTEP

=

timestep_rkf

in ‘src/Makefile.local’. In addition, you must put itorder=5 in ‘run.in’. An example
application is ‘samples/1d-tests/H2_flamespeed’. This procedure is still experimental.

5.16 Boundary conditions

39

Figure 4: Example of a velocity slice from a run where the time step is too long. Note the spurious
checkerboard modulation in places, for example near x = −0.5 and −2.5 < y < −1.5. This is an example
of a hyperviscous turbulence simulations with 5123 meshpoints and a third order hyperviscosity of ν3 =
5 × 10−12 . Hyperviscosity is explained in the Appendix E.

5.16 Boundary conditions
5.16.1 Where to specify boundary conditions
In most tests that come with the P ENCIL C ODE, boundary conditions are set in ‘run.in’,
which is a natural choice. However, this may lead to unexpected initial data written by
‘start.x’, since when you start the code (via ‘start.csh’), the boundary conditions are
unknown and ‘start.x’ will then fill the ghost zones assuming periodicity (the default
boundary condition) in all three directions. These ghost data will never be used in a
calculation, as ‘run.x’ will apply the boundary conditions before using any ghost-zone
values.
To avoid these periodic conditions in the initial snapshot, you can set the boundary
conditions in ‘start.in’ already. In this case, they will be inherited by ‘run.x’, unless you
also explicitly set boundary conditions in ‘run.in’.
5.16.2 How to specify boundary conditions
Boundary conditions are implemented through three layers of ghost points on either
boundary, which is quite a natural choice for an MPI code that uses ghost zones for
representing values located on the neighboring processors anyway. The desired type of
boundary condition is set through the parameters bc{x,y,z} in ‘run.in’; the nomenclature
used is as follows. Set bc{x,y,z} to a sequence of letters like
bcx = ’p’,’p’,’p’, ’p’,

’p’

for periodic boundaries, or
bcz = ’s’,’s’,’a’,’a2’,’c1:c2’
for non-periodic ones. Each element corresponds to one of the variables, which are those
of the variables ux , uy , uz , ln ρ, s/cp , Ax , Ay , Az , ln c that are actually used in this order.
The following conditions are available:
‘p’ periodic boundary condition
‘a’ antisymmetric condition w. r. t. the boundary, i. e. vanishing value

40

T HE P ENCIL C ODE

‘s’ symmetric condition w. r. t. the boundary, i. e. vanishing first derivative
‘a2’ antisymmetry w. r. t. the arbitrary value on the boundary, i. e. vanishing second
derivative
‘c1’ special boundary condition for ln ρ and s: constant heat flux through the boundary
‘c2’ special boundary condition for s: constant temperature at the boundary — requires
boundary condition a2 for ln ρ
‘cT’ special boundary condition for s or ln T : constant temperature at the boundary (for
arbitrarily set ln ρ)
‘ce’ special boundary condition for s: set temperature in ghost points to value on boundary (for arbitrarily set ln ρ)
‘db’ low-order one-sided derivatives (“no boundary condition”) for density
‘she’ shearing-sheet boundary condition (default when the module Shear is used)
‘g’ force the value of the corresponding field on vertical boundaries (should be used
in combination with the force lower bound and force upper bound flags set in the
namelist init pars)
‘hs’ special boundary condition for ln ρ and s which enforces hydrostatic equilibrium on
vertical boundaries
The extended syntax a:b (e. g. ‘c1:c2’) means: use boundary condition a at the left/lower
boundary, but b at the right/upper one.
If you build a new ‘run.in’ file from another one with a different number of variables
(noentropy vs. entropy , for example), you need to remember to adjust the length of the
arrays bcx to bcz . The advantage of the present approach is that it is very easy to exchange all boundary conditions by a new set of conditions in a particular direction (for
example, make everything periodic, or switch off shearing sheet boundary conditions
and have stress-free instead).

5.17

Restarting a simulation

When a run stops at the end of a simulation, you can just resubmit the job again, and
it will start from the latest snapshot saved in ‘data/proc*/var.dat’. The value of the
latest time is saved in a separate file, ‘data/proc*/time.dat’. On parallel machines it is
possible that some (or just one) of the ‘var.dat’ are corrupt; for example after a system
crash. Check for file size and date, and restart from a good ‘VAR’N file instead.
If you want to run on a different machine, you just need to copy the ‘data/proc*/var.dat’
(and, just to be sure) ‘data/proc*/time.dat’) files into a new directory tree. You may also
need the ‘data/proc*/seed.dat’ files for the random number generator. The easiest way
to get all these other files is to run start.csh again on the new machine (or in a new
directory) and then to overwrite the ‘data/proc*/var.dat’ files with the correct ones.
For restarting from runs that didn’t have magnetic fields, passive scalar fields, or test
fields, see Sect. F.3.

5.18 One- and two-dimensional runs

41

5.18 One- and two-dimensional runs
If you want to run two-dimensional problems, set the number of mesh points in one
direction to unity, e.g. nygrid =1 or nzgrid =1 in ‘cparam.local’. Remember that the number of mesh points is still divisible by the number of processors. For 2D-runs, it is also
possible to write only 2D-snapshots (i.e. VAR files written only in the considered (x, y)
or (x, z) plane, with a size seven times smaller as we do not write the third unused direction). To do that, please add the logical flag ‘lwrite 2d=T’ in the namelist init pars in
‘start.in’.
Similarly, for one-dimensional problems, set, for example, nygrid =1 and nzgrid =1 in
‘cparam.local’. You can even do a zero-dimensional run, but then you better set dt
(rather than cdt ), because there is no Courant condition for the time step.
See 0d, 1d, 2d, and 3d tests with examples.

5.19 Visualization
5.19.1 Gnuplot
Simple visualization can easily be done using Gnuplot (http://www.gnuplot.info), an
open-source plotting program suitable for two-dimensional plots.
For example, suppose you have the variables
---it-----t-------dt-------urms-----umax-----rhom-----ssm------dtc--in ‘time_series.dat’ and want to plot urms (t). Just start gnuplot and type
gnuplot>

plot "data/time_series.dat" using 2:4 with lines

If you work over a slow line and want to see both urms (t) and umax (t), use ASCII graphics:
gnuplot>
gnuplot>
gnuplot>
gnuplot>

set term dump
set logscale y
plot "data/time_series.dat" using 2:4 title "urms", \
"data/time_series.dat" using 2:5 title "umax"

5.19.2 Data explorer

DX (data explorer ; http://www.opendx.org) is an open-source tool for visualization of
three-dimensional data.
The P ENCIL C ODE provides a few networks for DX . It is quite easy to read in a snapshot
file from DX (the only tricky thing is the four extra bytes at the beginning of the file,
representing a Fortran record marker), and whenever you run ‘start.x’, the code writes
a file ‘var.general’ that tells DX all it needs to know about the data structure.
As a starting point for developing your own DX programs or networks , you can use a
few generic DX scripts provided in the directory ‘dx/basic/’. From the run directory,
start DX with
unix>

dx -edit $PENCIL_HOME/dx/basic/lnrho

42

T HE P ENCIL C ODE
✄

to load the file ‘dx/basic/lnrho.net’, and execute it with ✂Ctl-o ✁ or Execute → Execute
Once. You will see a set of iso-surfaces✄ of logarithmic density. If the viewport does not
fit to your data, you can reset it with ✂Ctl-f ✁. To rotate the object, drag the mouse over
the Image window with the left or right mouse button pressed. Similar networks are
provided for entropy (‘ss.net’), velocity (‘uu.net’) and magnetic field (‘bb.net’).
When you expand these simple networks to much more elaborate ones, it is probably
a good idea to separate the different tasks (like Importing and Selecting, visualizing
velocity, visualizing entropy, and Rendering) onto separate pages through Edit → Page.
Note Currently, DX can only read in data files written by one single processor, so from
a multi-processor run, you can only visualize one subregion at a time.
5.19.3 GDL

GDL , also known as Gnu Data Language is a free visualization package that can be
found at http://gnudatalanguage.sourceforge.net/. It aims at replacing the very expensive IDL package (see S. 5.19.4). For the way we use IDL for the Pencil Code, compatibility is currently not completely sufficient, but you can use GDL for many of the
visualization tasks. If you get spurious “Error opening file” messages, you can normally
simply ignore them.
This section tells you how to get started with using GDL for visualization.
Setup As of GDL 0.9 – at least the version packed with Ubuntu Jaunty (9.10) – you
will need to add GDL’s ‘examples/pro/’ directory to your !PATH variable. So the first call
after starting GDL should be
GDL>

.run setup_gdl

Starting visualization There are mainly two possibilities for visualization: using a simple GUI or loading the data with pc_read and work with it interactively. Please note that
the GUI was written and tested only with IDL, see §5.19.4.
Here, the pc_read family of IDL routines to read the data is described. Try
GDL>

pc_read

to get an overview.
To plot a time series, use
GDL>
GDL>
GDL>
GDL>

pc_read_ts, OBJECT=ts
help, ts, /STRUCT ;; (to see which slots are available)
plot, ts.t, ts.umax
oplot, ts.t, ts.urms

Alternatively, you could simply use the ‘ts.pro’ script:
GDL>

.run ts

To work with data from ‘var.dat’ and similar snapshot files, you can e.g. use the following routines:

5.19 Visualization

GDL>
GDL>
GDL>
GDL>
GDL>

43

pc_read_dim, OBJECT=dim
$$PENCIL_HOME/bin/nl2idl -d ./data/param.nml> ./param.pro
pc_read_param, OBJECT=par
pc_read_grid, OBJECT=grid
pc_read_var, OBJECT=var

Having thus read the data structures, we can have a look at them to see what information is available:
GDL>
GDL>
GDL>
GDL>

help,
help,
help,
help,

dim, /STRUCT
par, /STRUCT
grid, /STRUCT
var, /STRUCT

To visualize data, we can e.g. do9
GDL>
GDL>

plot, grid.x, var.ss[*, dim.ny/2, dim.nz/2]
contourfill, var.ss[*, *, dim.nz/2], grid.x, grid.y

GDL>
GDL>
GDL>

ux_slice = var.uu[*, *, dim.nz/2, 0]
uy_slice = var.uu[*, *, dim.nz/2, 1]
wdvelovect, ux_slice, uy_slice, grid.x, grid.y

GDL>

surface, var.lnrho[*, *, dim.nz/2, 0]

See also Sect. 5.19.4.
5.19.4 IDL

IDL is a commercial visualization program for two-dimensional and simple threedimensional graphics. It allows to access and manipulate numerical data in a fashion
quite similar to how Fortran handles them.
In ‘$PENCIL_HOME/idl’, we provide a number of general-purpose IDL scripts that we are
using all the time in connection with the P ENCIL C ODE. While IDL is quite an expensive
software package, it is quite useful for visualizing results from numerical simulations.
In fact, for many applications, the 7-minute demo version of IDL is sufficient.
Visualization in IDL The Pencil Code GUI is a data post-processing tool for the usage
on a day-to-day basis. It allows fast inspection of many physical quantities, as well as advanced features like horizontal averages, streamline tracing, freely orientable 2D-slices,
and extraction of cut images and movies. To use the Pencil Code GUI, it is sufficient to
run:
unix>
IDL>

idl
.r pc_gui

If you like to load only some subvolume of the data, like any 2D-slices from the given
data snapshots, or 3D-subvolumes, it is possible to choose the corresponding options in
the file selector dialog. The Pencil Code GUI offers also some options to be set on the
command-line, please refer to their description in the source code.
9

If contourfill produces just contour lines instead of a color-coded plot, your version of GDL is too
old. E.g. the version shipped with Ubuntu 9.10 is based on GDL 0.9rc1 and has this problem.

44

T HE P ENCIL C ODE

There are also other small GUIs available, e.g. the file ‘time-series.dat’ can easily be
analyzed with the command:
unix>
IDL>

idl
pc_show_ts

The easiest way to derive physical quantities at the command-line is to use one of the
many pc_read_var-variants (pc_read_var_raw is recommended for large datasets because of its high efficiency regarding computation and memory usage) for reading the
data. With that, one can make use of pc_get_quantity to derive any implemented physical quantity. Packed in a script, this is the recommended way to get reproducible results,
without any chance for accidental errors on the interactive IDL command-line.
Alternatively, by using the command-line to see the time evolution of e.g. velocity and
magnetic field (if they are present in you run), start IDL 10 and run ‘ts.pro’:
unix>
IDL>

idl
.run ts

The IDL script ‘ts.pro’ reads the time series data from ‘data/time_series.dat’ and sorts
the column into the structure ts , with the slot names corresponding to the name of the
variables (taken from the header line of ‘data/time_series.dat’). Thus, you can refer to
time as ts.t, to the rms velocity as ts.urms, and in order to plot the mean density as a
function of time, you would simply type
IDL>

plot, ts.t, ts.rhom

The basic command sequence for working with a snapshot is:
unix>
IDL>
IDL>
IDL>

idl
.run start
.run r
[specific commands]

You run ‘start.pro’ once to initialize (or reinitialize, if the mesh size has changed, for
example) the fields and read in the startup parameters from the code. To read in a new
snapshot, run ‘r.pro’ (or ‘rall.pro’, see below).
If you are running in parallel on several processors, the data are scattered over different
directories. To reassemble everything in IDL , use
10

If you run IDL from the command line, you will highly appreciate the following tip: IDL’s command
line editing is broken beyond hope. But you can fix it, by running IDL under rlwrap, a wrapper for the
excellent GNU readline library.
Download and install rlwrap from http://utopia.knoware.nl/~hlub/uck/rlwrap/ (on some systems
you just need to run ‘emerge rlwrap’, or ‘apt-get install rlwrap’), and alias your idl command:
csh>
alias idl ’rlwrap -a -c idl’
bash> alias idl=’rlwrap -a -c idl’
From now on, you can
• use long command lines that correctly wrap around;
✄
• type the first letters of a command and then ✂PageUp ✁to recall commands starting with these letters;
✄
✄
✄
• capitalize, uppercase or lowercase a word with ✂Esc ✁-C, ✂Esc ✁-U, ✂Esc ✁-L;
• use command line history across IDL sessions (you might need to create ‘~/.idl_history’ for this);
✄
• complete file names with ✂Tab ✁(works to some extent);
• . . . use all the other readline features that you are using in bash, octave, bc, gnuplot, ftp, etc.

5.19 Visualization

IDL>

45

.r rall

instead of .r r (here, .r is a shorthand for .run). The procedure ‘rall.pro’ reads (and
assembles) the data from all processors and correspondingly requires large amounts of
memory for very large runs. If you want to look at just the data from one processor, use
‘r.pro’ instead.
If you need the magnetic field or the current density, you can calculate them in IDL by
11

IDL>
IDL>

bb=curl(aa)
jj=curl2(aa)

By default one is reading always the current snapshot ‘data/procN /var.dat’; if you want
to read one of the permanent snapshots, use (for example)
IDL>
IDL>

varfile=’VAR2’
.r r (or .r rall)

See Sect. 5.6.1 for details on permanent snapshots.
With ‘r.pro’, you can switch the part of the domain by changing the variable datadir :
IDL>
IDL>

datadir=’data/proc3’
.r r

will read the data written by processor 3.
Reading data into IDL arrays or structures As an alternative to the method described
above, there is also the possibility to read the data into structures. This makes some
more operations possible, e.g. reading data from an IDL program where the command
.r is not allowed.
An efficient and still scriptable way would look like the following:
IDL>
IDL>
IDL>

pc_read_var_raw, obj=var, tags=tags
bb = pc_get_quantity (’B’, var, tags)
jj = pc_get_quantity (’j’, var, tags)

This reads the data into an array ‘var’, as well as the array indices of the contained physical variables into a separate structure ‘tags’. To use a caching mechanism within pc_get_quantity, please refer to the documentation and the examples contained in ‘pencilcode/idl/pc get quantity.pro’, where you can also start adding not yet implemented physical quantities.
To read a snapshot ’VAR10’ into the IDL structure ff, type the following command
IDL>

pc_read_var, obj=ff, varfile=’VAR10’, /trimall

The option /trimall removes ghost zones from the data. A number of other options are
documented in the source code of pc_read_var. You can see what data the structure
contains by using the command tag_names
IDL> print, tag_names(ff)
T X Y Z DX DY DZ UU LNRHO AA
11

Keep in mind that jj=curl(bb) would use iterated first derivatives instead of the second derivatives
and thus be numerically less accurate than jj=curl2(aa), particularly at small scales.

46

T HE P ENCIL C ODE

One can access the individual variables by typing ff.varname, e.g.
IDL> help, ff.aa

FLOAT

= Array[32, 32, 32, 3]

There are a number of files that read different data into structures. They are placed in
the directory $PENCIL_HOME/idl/files. Here is a list of files (including suggested options
to call them with)
• pc_read_var_raw, obj=var, tags=tags
Efficiently read a snapshot into an array.
• pc_read_var, obj=ff, /trimall
Read a snapshot into a structure.
• pc_read_ts, obj=ts
Read the time series into a structure.
• pc_read_xyaver, obj=xya
pc_read_xzaver, obj=xza
pc_read_yzaver, obj=yza
Read 1-D time series into a structure.
• pc_read_const, obj=cst
Read code constants into a structure.
• pc_read_pvar, obj=fp
Read particle data into a structure.
• pc_read_param, obj=par
Read startup parameters into a structure.
• pc_read_param, obj=par2, /param2
Read runtime parameters into a structure.
Other options are documented in the source code of the files.
For some examples on how to use these routines, see Sect. 5.19.3.
5.19.5 Python
Pencil supports reading, processing and the visualization of data using python . A number of scripts are placed in the subfolder ‘$PENCIL_HOME/python’. A README file placed
under that subfolder contains the information needed to read Pencil output data into
python.
Installation For modern operating systems, Python is generally installed together with
the system. If not, it can be installed via your preferred package manager or downloaded
from the website https://www.python.org/. For convenience, it is strongly recommend
to also install IPython, which is a more convenient console for Python. You will also need
the NumPy, matplotlib and Tk library.
Perhaps the easiest way to obtain all the required software mentioned above is to install
either Continuum’s Anaconda or Enthought’s Canopy. These Python distributions also
provide (or indeed are) integrated graphical development environments.

5.19 Visualization

47

In order for Python to find the Pencil Code commands you will have to add to your
.bashrc:
export PYTHONPATH=$PENCIL_HOME/python
ipythonrc If you use IPython, for convenience, you
.ipython/ipythonrc (create it if it doesn’t exist) and add:

should

modify

your

import_all pencil
Additional, add to your .ipython/profile_default/startup/init.py the following lines:
import numpy as np
import pylab as plt
import pencil as pc
import matplotlib
from matplotlib import rc
plt.ion()
matplotlib.rcParams[’savefig.directory’] = ’’
.pythonrc In case you are on a cluster and don’t have access to IPython you can edit
your .pythonrc:
#!/usr/bin/python
import numpy as np
import pylab as plt
import pencil as pc
import atexit
#import readline
import rlcompleter
# enables search with CTR+r in the history
try:
import readline
except ImportError:
print "Module readline not available."
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
# enables command history
historyPath = os.path.expanduser("~/.pyhistory")
def save_history(historyPath=historyPath):
import readline
readline.write_history_file(historyPath)
if os.path.exists(historyPath):
readline.read_history_file(historyPath)
atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath
plt.ion()

48

T HE P ENCIL C ODE

create the file .pythonhistory and add to your .bashrc:
export PYTHONSTARTUP=~/.pythonrc
Pencil Code Commands in General For a list of all Pencil Code commands start
IPython and type pc.  (as with auto completion). To access the help of any command just type the command followed by a ’?’ (no spaces), e.g.:
In [1]: pc.dot?
Type:
function
String Form:
File:
~/pencil-code/python/pencil/math/vector_multiplication.py
Definition: pc.dot(a, b)
Docstring:
take dot product of two pencil-code vectors a & b with shape
a.shape = (3,mz,my,mx)
You can also use help(pc.dot) for a more complete documentation of the command.
There are various reading routines for the Pencil Code data. All of them return an object
with the data. To store the data into a user defined variable type e.g.
In [1]: ts = pc.read_ts()
Most commands take some arguments. For most of them there is a default value, e.g.
In [1]: pc.read_ts(filename=’time_series.dat’, datadir=’data’, plot_data=True)
You can change the values by simply typing e.g.
In [1]: pc.read_ts(plot_data = False)
Reading and Plotting Time Series Reading the time series file is very easy. Simply
type
In [1]: ts = pc.read_ts()
and Python stores the data in the variable ts. The physical quantities are members of
the object ts and can be accessed accordingly, e.g. ts.t, ts.emag. To check which other
variables are stored simply do the tab auto completion ts. .
Plot the data with the matplotlib commands:
In [1]: plt.plot(ts.t, ts.emag)
The standard plots are not perfect and need a little polishing. See
the online wiki for a few examples on how to make pretty plots
(https://github.com/pencil-code/pencil-code/wiki/PythonForPencil).
You
can
save the plot into a file using the GUI or with
In [1]: plt.savefig(’plot.eps’)
Reading and Plotting VAR files and slice files Read var files:
In [1]: var = pc.read_var()

5.20 Running on multi-processor computers

49

Read slice files:
In [1]: slices, t = pc.read_slices(field=’bb1’, extension=’xy’)
This returns the array slices with indices slices[nTimes, my, mx] and the time array
t.
If you want to plot e.g. the x-component of the magnetic field at the central plane simply
type:
In [1]: plt.imshow(zip(*var.bb[0, 128, :, :]), origin=’lower’, extent=[-4, 4, -4, 4])
For a complete list of arguments of plt.imshow refer to its documentation.
For a more interactive plot use:
In [1]: pc.animate_interactive(slices, t)
Be aware: arrays from the reading routines are ordered f[nvar, mz, my, mx], i.e. reversed to IDL. This affects reading var files and slice files.
5.20 Running on multi-processor computers
The code is parallelized using MPI (message passing interface ) for a simple domain
decomposition (data-parallelism), which is a straight-forward and very efficient way of
parallelizing finite-difference codes. The current version has a few restrictions, which
need to be kept in mind when using the MPI features.
The global number of grid points (but excluding the ghost zones) in a given direction
must be an exact multiple of the number of processors you use in that direction. For
example if you have nprocy=8 processors for the y direction, you can run a job with
nygrid=64 points in that direction, but if you try to run a problem with nygrid=65 or
nygrid=94, the code will complain about an inconsistency and stop. (So far, this has not
been a serious restriction for us.)
5.20.1 How to run a sample problem in parallel
To run the sample problem in the directory ‘samples/conv-slab’ on 16 CPUs, you need
to do the following (in that directory):
1. Edit ‘src/Makefile.local’ and replace
MPICOMM

= nompicomm

by
MPICOMM

=

mpicomm

2. Edit ‘src/cparam.local’ and replace
integer, parameter :: ncpus=1, nprocy=1, nprocz=ncpus/nprocy, nprocx=1
integer, parameter :: nxgrid=32, nygrid=nxgrid, nzgrid=nxgrid
by
integer, parameter :: ncpus=16, nprocy=4, nprocz=ncpus/nprocy, nprocx=1
integer, parameter :: nxgrid=128, nygrid=nxgrid, nzgrid=nxgrid

50

T HE P ENCIL C ODE
The first line specifies a 4×4 layout of the data in the y and z direction. The second
line increases the resolution of the run because running a problem as small as 323
on 16 CPUs would be wasteful. Even 1283 may still be quite small in that respect.
For performance timings, one should try and keep the size of the problem per CPU
the same, so for example 2563 on 16 CPUs should be compared with 1283 on 2 CPUs.
3. Recompile the code
unix>

(cd src; make)

4. Run it
unix>
unix>

start.csh
run.csh

Make sure that all CPUs see the same ‘data/’ directory; otherwise things will go wrong.
Remember that in order to visualize the full domain with IDL (rather than just the
domain processed and written by one processor), you need to use ‘rall.pro’ instead of
‘r.pro’.
5.20.2 Hierarchical networks (e.g. on Beowulf clusters)
On big Beowulf clusters, a group of nodes is often connected with a switch of modest
speed, and all these groups are connected via a n times faster uplink switch. When
bandwidth-limited, it is important to make sure that consecutive processors are mapped
onto the mesh such that the load on the uplink is . n times larger than the load on the
slower switch within each group. On a 512 node cluster, where groups of 24 processors
are linked via fast ethernet switches, which in turn are connected via a Gigabit uplink
(∼ 10 times faster), we found that nprocy =4 is optimal. For 128 processors, for example
we find that nprocy × nprocz = 4 × 32 is the optimal layout, while. For comparison, 8 × 16
is 3 times slower, and 16 × 8 is 17 (!) times slower. These results can be understood from
the structure of the network, but the basic message is to watch out for such effects and
to try varying nprocy and nprocz .
In cases where nygrid>nzgrid it may be advantageous to swap the ordering of processor
numbers. This can be done by setting lprocz slowest =F.
5.20.3 Extra workload caused by the ghost zones
Normally, the workload caused by the ghost zones is negligible. However, if one increases
the number of processors, a significant fraction of work is done in the ghost zones. In
other words, the effective mesh size becomes much larger than the actual mesh size.
Consider a mesh of size Nw = Nx × Ny × Nz , and distribute the task over Pw = Px × Py ×
Pz processors. If no communication were required, the number of points per processor
would be
Nx × Ny × Nz
Nw
=
.
(26)
Pw
Px × Py × Pz

However, for finite difference codes some communication is required, and the amount of
communication depends on spatial order of the scheme, Q. The P ENCIL C ODE works by

5.21

Running in double-precision

51

default with sixth order finite differences, so Q = 6, i.e. one needs 6 ghost zones, 3 on
each end of the mesh. With Q 6= 0 the number of points per processor is
(eff)

Nw
=
Pw



 
 

Nx
Ny
Nz
+Q ×
+Q ×
+Q .
Px
Py
Pz

(27)

There is efficient scaling only when
min



Nx Ny Nz
,
,
Px Py Pz



(28)

≫ Q.

1/3

1/2

In the special case were Nx = Ny = Nz ≡ N = Nw , with Px = 1 and Py = Pz ≡ P = Pw ,
we have
2

(eff)
N
Nw
+Q .
= (N + Q) ×
(29)
Pw
P

For N = 128 and Q = 6 the effective mesh size exceeds the actual mesh size by a factor
(eff)

Nw
= (N + Q) ×
Nw



N
+Q
P

2

×

Pw
.
Nw

(30)

These factors are listed in Table 3.
(eff)

Table 3: Nw

P \N
1
2
4
8
16
32
64
128
256
512

128
1.15
1.19
1.25
1.34
1.48
1.68
1.98
2.45
3.21
4.45

/Nw versus N and P .

256
1.07
1.09
1.12
1.16
1.22
1.31
1.44
1.64
1.93
2.40

512 1024 2048
1.04 1.02 1.01
1.05 1.02 1.01
1.06 1.03 1.01
1.08 1.04 1.02
1.11 1.05 1.03
1.15 1.07 1.04
1.21 1.10 1.05
1.30 1.14 1.07
1.43 1.20 1.10
1.62 1.29 1.14

Ideally, one wants to keep the work in the ghost zones at a minimum. If one accepts
that 20–25% of work are done in the ghost zones, one should use 4 processors for 1283
mesh points, 16 processors for 2563 mesh points, 64 processors for 5123 mesh points, 256
processors for 10243 mesh points, and 512 processors for 15363 mesh points.
5.21 Running in double-precision
With many compilers, you can easily switch to double precision (8-byte floating point
numbers) as follows.
Add the lines
# Use double precision
REAL_PRECISION = double

52

T HE P ENCIL C ODE

to ‘src/Makefile.local’ and (re-)run pc_setupsrc.
If REAL PRECISION is set to ‘double’, the flag FFLAGS DOUBLE is appended to
the Fortran compile flags. The default for FFLAGS DOUBLE is -r8, which works for
g95 or ifort; for gfortran , you need to make sure that FFLAGS DOUBLE is set to
-fdefault-real-8.
You can see the flags in ‘src/Makefile.inc’, which is the first place to check if you have
problems compiling for double precision.
Using double precision might be important in turbulence runs where the resolution
is 2563 meshpoints and above (although such runs often seem to work fine at single
precision). To continue working in double precision, you just say lread_from_other_prec=T in run_pars; see Section F.1.
5.22

Power spectrum

Given a real variable u, its Fourier transform ũ is given by
y −1 Nz −1
N
x −1 N
X
X
X
1
ũ(kx , ky , kz ) = F(u(x, y, z)) =
u(xp , yq , zr )
Nx Ny Nz p=0 q=0 r=0

× exp(−ikx xp ) exp(−iky yq ) exp(−ikz zr ),

where
|kx | <

πNx
,
Lx

|ky | <

πNy
,
Ly

|kz | <

(31)

πNz
,
Lz

when L is the size of the simulation box. The three-dimensional power spectrum P (k) is
defined as
1
P (k) = ũũ∗ ,
(32)
2
where
q
(33)
k = kx2 + ky2 + kz2 .
Note that we can only reasonably calculate P (k) for k < πNx /Lx .

To get power spectra from the code, edit ‘run.in’ and add for example the following lines
dspec=5., ou_spec=T, ab_spec=T !(for energy spectra)
oned=T
under run_pars. The kinetic (vel_spec) and magnetic (mag_spec) power spectra will
now be calculated every 5.0 (dspec) time units. The Fourier spectra is calculated using
fftpack . In addition to calculating the three-dimensional power spectra also the onedimensional power spectra will be calculated (oned).
In addition one must edit ‘src/Makefile.local’ and add the lines
FOURIER = fourier_fftpack
POWER = power_spectrum
Running the code will now create the files ‘powerhel_mag.dat’ and ‘power_kin.dat’
containing the three-dimensional magnetic and kinetic power spectra respectively. In
addition to these three-dimensional files we will also find the one-dimensional files
‘powerbx_x.dat’, ‘powerby_x.dat’, ‘powerbz_x.dat’, ‘powerux_x.dat’, ‘poweruy_x.dat’ and

5.22 Power spectrum

53

‘poweruz_x.dat’. In these files the data are stored such that the first line contains the
time of the snapshot, the following nxgrid /2 numbers represent the power at each
wavenumber, from the smallest to the largest. If several snapshots have been saved,
they are being stored immediately following the preceding snapshot.
You can read the results with the idl procure ‘power’, like this:
power,’_kin’,’_mag’,k=k,spec1=spec1,spec2=spec2,i=n,tt=t,/noplot
power,’hel_kin’,’hel_mag’,k=k,spec1=spec1h,spec2=spec2h,i=n,tt=t,/noplot
If powerhel is invoked, krms is written during the first computation. The relevant output file is ‘power_krms.dat’. This is needed for a correct calculation of k used in the
realizability conditions.
A caveat of the implementation of Fourier transforms in the P ENCIL C ODE is that, due
to the parallelization, the permitted resolution is limited to the case when one direction
is an integer multiple of the other. So, it can be done for
Nx = n*Ny
Unfortunately, for some applications one wants Nx < Ny. Wlad experimented with arbitrary resolution by interpolating x to the same resolution of y prior to transposing,
then transform the interpolated array and then interpolating it back (check ‘fourier_transform_y’ in ‘fourier_fftpack.f90’).
A feature of our current implementation with x parallelization is that fft_xyz_parallel_3D requires nygrid to be an integer multiple of nprocy*nprocz. Examples of
good mesh layouts are listed in Table 4.
Table 4: Examples of mesh layouts for which Fourier transforms with x parallelization is possible.

ny nprocyx
256
1
2
256
256
4
256
8
288
2
512
2
4
512
512
4
576
4
576
8
576
16
4
1024
1024
4
1024
8
1152
4
1152
4
2
2304
2304
4
2304
4

nprocy nprocz
16
16
16
16
16
16
16
16
16
18
16
32
16
16
16
32
18
32
18
32
18
32
32
32
16
64
16
32
36
32
32
36
32
72
36
64
32
72

ncpus
256
512
1024
2048
576
1024
1024
2048
2304
4608
9216
4096
4096
4096
4608
4608
4608
9216
9216

To visualize with IDL just type power and you get the last snapshot of the three-

54

T HE P ENCIL C ODE

dimensional power spectrum. See head of ‘$PENCIL_HOME/idl/power.pro’ for options to
power.
5.23

Structure functions

We define the p-th order longitudinal structure function of u as
p
Slong
(l) = h|ux (x+l, y, z) − ux (x, y, z)|p i ,

(34)

p
Strans
(l) = h|uy (x+l, y, z) − uy (x, y, z)|p i + h|uz (x+l, y, z) − uz (x, y, z)|p i .

(35)

while the transverse is

Edit ‘run.in’ and add for example the following lines
dspec=2.3,
lsfu=T, lsfb=T, lsfz1=T, lsfz2=T
under run_pars. The velocity (lsfu), magnetic (lsfb) and Elsasser (lsfz1 and lsfz2)
structure functions will now be calculated every 2.3 (dspec) time unit.
In addition one must edit ‘src/Makefile.local’ and add the line
STRUCT_FUNC = struct_func
In ‘src/cparam.local’, define lb nxgrid and make sure that
nxgrid = nygrid = nzgrid = 2**lb_nxgrid
E.g.
integer, parameter :: lb_nxgrid=5
integer, parameter :: nxgrid=2**lb_nxgrid,nygrid=nxgrid,nzgrid=nxgrid
Running the code will now create the files:
‘sfu-1.dat’, ‘sfu-2.dat’, ‘sfu-3.dat’ (velocity),
‘sfb-1.dat’, ‘sfb-2.dat’, ‘sfb-3.dat’ (magnetic field),
‘sfz1-1.dat’, ‘sfz1-2.dat’, ‘sfz1-3.dat’ (first Elsasser variable),
‘sfz2-1.dat’, ‘sfz2-2.dat’, ‘sfz2-3.dat’ (second Elsasser variable),
which contains the data of interest. The first line in each file contains the time t and
the number qmax , such that the largest moment calculated is qmax − 1. The next imax
numbers represent the first moment structure function for the first snapshot, here

imax = 2

ln(nxgrid )
− 2.
ln 2

(36)

The next imax numbers contain the second moment structure function, and so on until
qmax − 1. The following imax numbers then contain the data of the signed third order
3
structure function i.e. Slong
(l) = h[ux (x+l, y, z) − ux (x, y, z)]3 i.
The following imax × qmax ×2 numbers are zero if nr directions = 1 (default), otherwise
they are the same data as above but for the structure functions calculated in the y and
z directions.

5.24 Particles

55

If the code has been run long enough as to calculate several snapshots, these snapshots
will now follow, being stored in the same way as the first snapshot.
To visualize with IDL just type structure and you get the time-average of the first order
longitudinal structure function (be sure that ‘pencil-runs/forced/idl/’ is in IDL_PATH).
See head of ‘pencil-runs/forced/idl/structure.pro’ for options to structure.
5.24 Particles
The P ENCIL C ODE has modules for tracer particles and for dust particles (see Sect. 6.14).
The particle modules are chosen by setting the value of the variable PARTICLES in
Makefile.local to either particles_dust or particles_tracers. For the former case
each particle has six degrees of freedom, three positions and three velocities. For the
latter it suffices to have only three position variables as the velocity of the particles are
equal to the instantaneous fluid velocity at that point. In addition one can choose to have
several additional internal degrees of freedoms for the particles. For example one can
temporally evolve the particles radius by setting PARTICLES_RADIUS to particles_radius
in Makefile.local.
All particle infrastructure is controlled and organized by the Particles_main module.
This module is automatically selected by Makefile.src if PARTICLES is different from
noparticles. Particle modules are compiled as a separate library. This way the main
part of the Pencil Code only needs to know about the particles_main.a library, but not
of the individual particle modules.
For a simulation with particles one must in addition define a few parameters in
cparam.local. Here is a sample of cparam.local for a parallel run with 2,000,000 particles:
integer,
integer,
integer,
integer,

parameter
parameter
parameter
parameter

::
::
::
::

ncpus=16, nprocy=4, nprocz=4, nprocx=1
nxgrid=128, nygrid=256, nzgrid=128
npar=2000000, mpar_loc=400000, npar_mig=1000
npar_species=2

The parameter npar is the number of particles in the simulation, mpar_loc is the number
of particles that is allowed on each processor and npar_mig is the number of particles
that are allowed to migrate from one processor to another in any time-step. For a nonparallel run it is enough to specify npar. The number of particle species is set through
npar_species (assumed to be one if not set). The particle input parameters are given in
start.in and run.in. Here is a sample of the particle part of start.in for dust particles:
/
&particles_init_pars
initxxp=’gaussian-z’, initvvp=’random’
zp0=0.02, delta_vp0=0.01, eps_dtog=0.01, tausp=0.1
lparticlemesh_tsc=T
/
The initial positions and velocities of the dust particles are set in initxxp and initvvp.
The next four input parameters are further specifications of the initial condition. Interaction between the particles and the mesh, e.g. through drag force or self-gravity, require
a mapping of the particles on the mesh. The P ENCIL C ODE currently supports NGP
(Nearest Grid Point, default), CIC (Cloud in Cell, set lparticlemesh_cic=T) and TSC

56

T HE P ENCIL C ODE

(Triangular Shaped Cloud, set lparticlemesh_tsc=T). See Youdin & Johansen (2007) for
details.
Here is a sample of the particle part of run.in (also for dust particles):
/
&particles_run_pars
ldragforce_gas_par=T
cdtp=0.2
/
The logical ldragforce_gas_par determines whether the dust particles influence the gas
with a drag force. cdtp tells the code how many friction times should be resolved in a
minimum time-step.
The sample run ‘samples/sedimentation/’ contains the latest setup for dust particles.
5.24.1 Particles in parallel
The particle variables (e.g. xi and v i ) are kept in the arrays fp and dfp. For parallel
runs, particles must be able to move from processor to processor as they pass out of the
(x, y, z)-interval of the local processor. Since not all particles are present at the same
processor at the same time (hopefully), there is some memory optimization in making
fp not big enough to contain all the particles at once. This is achieved by setting the code
variable mpar_loc less than npar in cparam.local for parallel runs. When running with
millions of particles, this trick is necessary to keep the memory need of the code down.
The communication of migrating particles between the processors happens as follows
(see the subroutine redist_particles_procs in particles_sub.f90):
1. In the beginning of each time-step all processors check if any of their particles have
crossed the local (x, y, z)-interval. These particles are called migrating particles. A
run can have a maximum of npar_mig migrating particles in each time-step. The
value of npar_mig must be set in cparam.local. The number should (of course) be
slightly larger than the maximum number of migrating particles at any time-step
during the run. The diagnostic variable nmigmax can be used to output the maximum number of migrating particles at a given time-step. One can set lmigration_redo=T in &particles_run_pars to force the code to redo the migration step if more
than npar_mig want to migrate. This does slow the code down somewhat, but has
the benefit that the code does not stop when more than npar_mig particles want to
migrate.
2. The index number of the receiving processor is then calculated. This requires some
assumption about the grid on other processors and will currently not work for
nonequidistant grids. Particles do not always pass to neighboring processors as the
global boundary conditions may send them to the other side of the global domains
(periodic or shear periodic boundary conditions).
3. The migrating particle information is copied to the end of fp, and the empty spot
left behind is filled up with the particle of the highest index number currently
present at the processor.
4. Once the number of migrating particles is known, this information is shared with
neighboring processors (including neighbors over periodic boundaries) so that they

5.24 Particles

57

all know how many particles they have to receive and from which processors.
5. The communication happens as directed MPI communication. That means that
processors 0 and 1 can share migrating particles at the same time as processors 2
and 3 do it. The communication happens from a chunk at the end of fp (migrating
particles) to a chunk that is present just after the particle of the highest index
number that is currently at the receiving processor. Thus the particles are put
directly at their final destination, and the migrating particle information at the
source processor is simply overwritten by other migrating particles at the next
time-step.
6. Each processor keeps track of the number of particles that it is responsible for. This
number is stored in the variable npar_loc. It must never be larger than mpar_loc
(see above). When a particle leaves a processor, npar_loc is reduced by one, and
then increased by one at the processor that receives that particle. The maximum
number of particles at any processor is stored in the diagnostic variable nparmax. If
this value is not close to npar/ncpus, the particles have piled up in such a way that
computations are not evenly shared between the processors. One can then try to
change the parallelization architecture (nprocy and nprocz) to avoid this problem.
In simulations with many particles (comparable to or more than the number of grid
cells), it is crucial that particles are shared relatively evenly among the processors. One
can as a first approach attempt to not parallelize directions with strong particle density
variations. However, this is often not enough, especially if particles clump locally.
Alternatively one can use Particle Block Domain Decomposition (PBDD, see Johansen
et al. 2011). The steps in Particle Block Domain Decomposition scheme are as follows:
1. The fixed mesh points are domain-decomposed in the usual way (with
ncpus=nprocx×nprocy×nprocz).
2. Particles on each processor are counted in bricks of size nbx×nby×nbz (typically
nbx= nby=nbz=4).
3. Bricks are distributed among the processors so that each processor has approximately the same number of particles
4. Adopted bricks are referred to as blocks.
5. The Pencil Code uses a third order Runge-Kutta time-stepping scheme. In the beginning of each sub-time-step particles are counted in blocks and the block counts
communicated to the bricks on the parent processors. The particle density assigned
to ghost cells is folded across the grid, and the final particle density (defined on the
bricks) is communicated back to the adopted blocks. This step is necessary because
the drag force time-step depends on the particle density, and each particle assigns
density not just to the nearest grid point, but also to the neighboring grid points.
6. In the beginning of each sub-time-step the gas density and gas velocity field is
communicated from the main grid to the adopted particle blocks.
7. Drag forces are added to particles and back to the gas grid points in the adopted
blocks. This partition aims at load balancing the calculation of drag forces.
8. At the end of each sub-time-step the drag force contribution to the gas velocity field
is communicated from the adopted blocks back to the main grid.

58

T HE P ENCIL C ODE

Particle Block Domain Decomposition is activated by setting PARTICLES = particles_dust_blocks and PARTICLES_MAP = particles_map_blocks in Makefile.local. A sample of cparam.local for Particle Block Domain Decomposition can be found in
samples/sedimentation/blocks:
integer,
integer,
integer,
integer,
integer,

parameter
parameter
parameter
parameter
parameter

::
::
::
::
::

ncpus=4, nprocx=2, nprocy=2, nprocz=1
nxgrid=32, nygrid=32, nzgrid=32
npar=10000, mpar_loc=5000, npar_mig=100
npar_species=4
nbrickx=4, nbricky=4, nbrickz=4, nblockmax=32

The last line defines the number of bricks in the total domain – here we divide the grid
into 4 × 4 × 4 bricks each of size 8 × 8 × 8 grid points. The parameter nblockmax tells the
code the maximum number of blocks any processor may adopt. This should not be so low
that there is not room for all the bricks with particles, nor so high that the code runs out
of memory.

5.24.2 Large number of particles
When dealing with large number of particles, one needs to make sure that the number
of particles npar is less than the maximum integer that the compiler can handle with.
The maximum integer can be checked by the Fortran intrinsic function huge,
program huge_integers
print *, huge(0_4) ! for default Fortran integer (32 Bit)
print *, huge(0_8) ! for 64 Bit integer in Fortran
end program huge_integers
If the number of particles npar is larger than default maximum integer, one can promote
the maximum integer to 64 Bit by setting
integer(kind=8), parameter :: npar=4294967296
in the cparam.local file. This works because the data type of npar is only set here. It is
worth noting that one should not use the flag
FFLAGS += -integer-size 64
to promote all the integers to 64 Bit. This will break the Fortran-C interface. One
should also make sure that npar_mig<=npar/ncpus. It is also beneficial to set mpar_loc=2*npar/ncpus.

5.24.3 Random number generator
There are several methods to generate random number in the code. It is worth noting
that when simulating coagulation with the super-particle approach, one should use the
intrinsic random number generator of FORTRAN instead of the one implemented in the
code. When invoking random_number_wrapper, there will be back-reaction to the gas flow.
This unexpected back-reaction can be tracked by inspecting the power spectra, which
exhibits the oscillation at the tail. To avoid this, one should set luser_random_number_wrapper=F under the module particles_coag_run_pars in run.in.

5.25 Non-cartesian coordinate systems

59

5.25 Non-cartesian coordinate systems
Since the spring of 2007 spherical and cylindrical polar coordinates have been implemented, although this development is not yet completed. Spherical coordinates are invoked by adding the following line in the file ‘start.in’
&init_pars
coord_system=’spherical_coords’
Another possibility is to put cylindrical_coords instead. In practice, the names (x, y, z)
are still used, but they refer then to (r, θ, φ) or (r, φ, z) instead.
Bug reports, corrections, and improvements on these are appreciated.

60

6

T HE P ENCIL C ODE

The equations

The equations solved by the P ENCIL C ODE are basically the standard compressible
MHD equations. However, the modular structure allows some variations of the MHD
equations, as well as switching off some of the equations or individual terms of the
equation (nomagnetic, noentropy, etc.).
In this section the equations are presented in their most complete form. It may be expected that the code can evolve most subsets or simplifications of these equations.
6.1

Continuity equation

In the code the continuity equation, ∂ρ/∂t + ∇ · ρu = 0, is written in terms of ln ρ,
D ln ρ
= −∇ · u .
Dt

(37)

Here ρ denotes density, u the fluid velocity, t is time and D/Dt ≡ ∂/∂t + u · ∇ is the
convective derivative.
6.2

Equation of motion

In the equation of motion, using a perfect gas, the pressure term, can be expressed as
−ρ−1 ∇p = −c2s (∇s/cp + ∇ ln ρ), where the squared sound speed is given by


ρ
p
2
2
,
(38)
cs = γ = cs0 exp γs/cp + (γ−1) ln
ρ
ρ0
and γ = cp /cv is the ratio of specific heats, or adiabatic index. Note that c2s is proportional
to the temperature with c2s = (γ − 1)cp T .
The equation of motion is accordingly


Du
j×B
s
2
=
−cs ∇
+ ln ρ − ∇Φgrav +
Dt
cp
ρ


1
2
+ν ∇ u + ∇∇ · u + 2S · ∇ ln ρ + ζ (∇∇ · u) ;
3

(39)

Here Φgrav is the gravity potential, j the electric current density, B the magnetic flux
density, ν is kinematic viscosity, ζ describes a bulk viscosity, and, in Cartesian coordinates


2
1 ∂ui ∂uj
+
− δij ∇ · u
(40)
Sij =
2 ∂xj
∂xi
3

is the rate-of-shear tensor that is traceless, because it can be written as the generic rateof-strain tensor minus its trace. In curvilinear coordinates, we have to replace partial
differentiation by covariant differentiation (indicated by semicolons), so we write Sij =
1
(ui;j + uj;i ) − 13 δij ∇ · u.
2
The interpretation of the two viscosity terms varies greatly depending upon the Viscosity module used, and indeed on the parameters given to the module. See §6.6.
For isothermal hydrodynamics, see §6.4 below.

6.3

6.3

Induction equation

61

Induction equation
∂A
= u × B − ηµ0 j .
∂t

(41)

Here A is the magnetic vector potential, B = ∇ × A the magnetic flux density, η =
1/(µ0 σ) is the magnetic diffusivity (σ denoting the electrical conductivity), and µ0 the
magnetic vacuum permeability. This form of the induction equation corresponds to the
Weyl gauge Φ = 0, where Φ denotes the scalar potential.
6.4

Entropy equation

The current thermodynamics module entropy formulates the thermal part of the physics
in terms of entropy s, rather than thermal energy e, which you may be more familiar with. Thus the two fundamental thermodynamical variables are ln ρ and s. The
reason for this choice of variables is that entropy is the natural physical variable for
(at least) convection processes: the sign of the entropy gradient determines convective
(in)stability, the Rayleigh number is proportional to the entropy gradient of the associated hydrostatic reference solution, etc. The equation solved is
ρT

Ds
= H − C + ∇ · (K∇T ) + ηµ0 j 2 + 2ρνS ⊗ S + ζρ (∇ · u)2 .
Dt

(42)

Here, T is temperature, cp the specific heat at constant pressure, H and C are explicit
heating and cooling terms, K is the radiative (thermal) conductivity, ζ describes a bulk
viscosity, and S is the rate-of-shear tensor that is traceless.
In the entropy module we solve for the specific entropy, s. The heat conduction term on
the right hand side can be written in the form
∇ · (K∇T )
ρT
h
i
2
= cp χ ∇ ln T + ∇ ln T · ∇(ln T + ln χ+ ln ρ)


= cp χ γ∇2 s/cp + (γ−1)∇2 ln ρ
+cp χ [γ∇s/cp + (γ−1)∇ ln ρ] · [γ (∇s/cp + ∇ ln ρ) + ∇ ln χ] ,

(43)
(44)
(45)

where χ = K/(ρcp ) is the thermal diffusivity. The latter equation shows that the diffusivity for s is γχ, which is what we have used in Eq. (24).
In an alternative formulation for a constant K, the heat conduction term on the right
hand side can also be written in the form
i
∇ · (K∇T )
Kh 2
=
(46)
∇ ln T + (∇ ln T )2
ρT
ρ
which is the form used when constant K is chosen.

Note that by setting γ = 1 and initially s = 0, one obtains an isothermal equation of
state (albeit at some unnecessary expense of memory). Similarly, by switching off the
evolution terms of entropy, one immediately gets polytropic behavior (if s was initially
constant) or generalized polytropic behavior (where s is not uniform, but ∂s/∂t = 0).
A better way to achieve isothermality is to use the noentropy module.

62

T HE P ENCIL C ODE

6.4.1 Viscous heating
We can write the viscosity as the divergence of a tensor τij,j ,
ρ

∂ui
= ... + τij,j ,
∂t

(47)

where τij = 2νρSij is the stress tensor. The viscous power density P is
P = ui τij,j
∂
=
(ui τij ) − ui,j τij
∂xj

(48)
(49)

The term under the divergence is the viscous energy flux and the other term is the
kinetic energy loss due to heating. The heating term +ui,j τij is positive definite, because
τij is a symmetric tensor and the term only gives a contribution from the symmetric part
of ui,j , which is 12 (ui,j + uj,i ), so
1
ui,j τij = νρ(ui,j + uj,i )(2Sij ) .
2

(50)

But, because Sij is traceless, we can add anything proportional to δij and, in particular,
we can write
1
(ui,j + uj,i )(2νρSij )
2
1
1
=
(ui,j + uj,i − δij ∇ · u)(2νρSij )
2
3
2
= 2νρS ,

ui,j τij =

(51)
(52)
(53)

which is positive definite.

6.4.2 Alternative description
By setting pretend_lnTT=T in init_pars or run_pars (i.e. the general part of the name
list) the logarithmic temperature is used instead of the entropy. This has computational
advantages when heat conduction (proportional to K∇T ) is important. Another alternative is to use another module, i.e. set ENTROPY=temperature_idealgas in ‘Makefile.local’.
When pretend_lnTT=T is set, the entropy equation
1
∂s
= −u · ∇s +
RHS
∂t
ρT

(54)

1
∂ ln T
= −u · ∇ ln T +
RHS − (γ − 1) ∇ · u,
∂t
ρcv T

(55)

is replaced by

where RHS is the right hand side of equation (42).

6.5

6.5

Transport equation for a passive scalar

63

Transport equation for a passive scalar

In conservative form, the equation for a passive scalar is
∂
(ρc) + ∇ · [ρcu − ρD∇c] = 0.
(56)
∂t
Here c denotes the concentration (per unit mass) of the passive scalar and D its diffusion
constant (assumed constant). In the code this equation is solved in terms of ln c,


D ln c
= D ∇2 ln c + (∇ ln ρ + ∇ ln c) · ∇ ln c .
(57)
Dt
Using ln c instead of c has the advantage that it enforces c > 0 for all times. However,
the disadvantage is that one cannot have c = 0. For this reason we ended up using the
non-logarithmic version by invoking PSCALAR=pscalar_nolog.
6.6

Bulk viscosity

For a monatomic gas it can be shown that the bulk viscosity vanishes. We therefore don’t
use it in most of our runs. However, for supersonic flows, or even otherwise, one might
want to add a shock viscosity which, in its simplest formulation, take the form of a bulk
viscosity.
6.6.1 Shock viscosity
Shock viscosity, as it is used here and also in the Stagger Code of Åke Nordlund, is
proportional to positive flow convergence, maximum over five zones, and smoothed to
second order,
D
E
ζshock = cshock max[(−∇ · u)+ ] (min(δx, δy, δz))2 ,
(58)
5

where cshock is a constant defining the strength of the shock viscosity. In the code this
dimensionless coefficient is called nu_shock, and it is usually chosen to be around unity.
Assume that the shock viscosity only enters as a bulk viscosity, so the whole stress
tensor is then
τ ij = 2ρνSij + ρζshock δij ∇ · u.
(59)
Assume ν = const, but ζ 6= const, so


1
−1
2
ρ F visc = ν ∇ u + ∇∇ · u + 2S · ∇ ln ρ + ζshock [∇∇ · u + (∇ ln ρ + ∇ ln ζshock ) ∇ · u] .
3
(60)
and
ρ−1 Γvisc = 2νS2 + ζshock (∇ · u)2 .
(61)

In the special case with periodic boundary conditions, we have 2hS2 i = hω 2 i + 43 h(∇ · u)2 i.
6.7

Equation of state

In its present configuration only hydrogen ionization is explicitly included. Other constituents (currently He and H2 ) can have fixed values. The pressure is proportional to
the total number of particles, i.e.
p = (nHI + nHII + nH2 + ne + nHe + ...)kB T.

(62)

64

T HE P ENCIL C ODE

It is convenient to normalize to the total number of H both in atomic and in molecular
hydrogen, nHtot ≡ nH + 2nH2 , where nHI + nHII = nH , and define xe ≡ ne /nHtot , xHe ≡
nHe /nHtot , and xH2 ≡ nH2 /nHtot . Substituting nH = nHtot − 2nH2 , we have
p = (1 − xH2 + xe + xHe + ...)nHtot kB T.

(63)

This can be written in the more familiar form
p=

R
ρT,
µ

(64)

where R = kB /mu and mu is the atomic mass unit (which is for all practical purposes the
same as mHtot ) and
µ=

1 + 4xHe
nH + 2nH2 + ne + 4nHe
=
nH + nH2 + ne + nHe
1 − xH2 + xe + xHe

(65)

is the mean molecular weight (which is here dimensionless; see Kippenhahn & Weigert
1990, p. 102). The factor 4 is really to be substituted for 3.97153. Some of the familiar
relations take still the usual form, in particular e = cv T and h = cp T with cv = 32 R/µ and
cp = 25 R/µ.
The number ratio, xHe , is more commonly expressed as the mass ratio, Y =
mHe nHe /(mH nHtot + mHe nHe ), or Y = 4xHe /(1 + 4xHe ), or 4xHe = (1/Y − 1)−1 . For example, Y = 0.27 corresponds to xHe = 0.092 and Y = 0.25 to xHe = 0.083. Note also that for
100% H2 abundance, xH2 = 1/2.
In the following, the ionization fraction is given as y = ne /nH , which can be different
from xe if there is H2 . Substituting for nH in terms of nHtot yields y = xe /(1 − 2xH2 ).
6.8

Ionization

This part of the code can be invoked by setting EOS=eos_ionization (or EOS=eos_temperature_ionization) in the ‘Makefile.local’ file. The equation of state described
below works for variable ionization, and the entropy offset is different from that used in
Eq. (38), which is now no longer valid. As a replacement, one can use EOS=eos_fixed_ionization, where the degree of ionization can be given by hand. Here the normalization
of the entropy is the same as for EOS=eos_ionization. This case is described in more detail below.12
We treat the gas as being composed of partially ionized hydrogen and neutral helium.
These are four different particle species, each of which regarded as a perfect gas.
The ionization fraction y, which gives the ratio of ionized hydrogen to the total amount
of hydrogen nH , is obtained from the Saha equation which, in this case, may be written
as

3/2


1 me kB T
χH
y2
=
exp −
.
(66)
1−y
nH
2π~2
kB T
The temperature T cannot be obtained directly from the P ENCIL C ODE’s independent
variables ln ρ and s, but is itself dependent on y. Hence, the calculation of y essentially
becomes a root finding problem.
12

We omit here the contribution of H2 .

6.8 Ionization

65

The entropy of a perfect gas consisting of particles of type
Tetrode equation
"

3/2 #
mi kB T
1
+
Si = kB Ni ln
ntot
2π~2

i is known from the Sackur5
2

!

.

(67)

Here Ni is the number of particles of a single species and ntot is the total number density
of all particle species.
In addition to P
the individual entropies we also have to take the entropy of mixing,
Smix = −Ntot kB i pi ln pi , into account. Summing up everything, we can get a closed expression for the specific entropy s in terms of y, ln ρ and T , which may be solved for
T.

Figure 5: Dependence of temperature on entropy for different values of the density.

For given ln ρ and s we are then able to calculate the ionization fraction y by finding the
root of
"

1−y 1
f (y) = ln
y 2 nH



me kB T (y)
2π~2

3/2 #

−

χH
.
kB T (y)

(68)

In the ionized case, several thermodynamic quantities of the gas become dependent on
the ionization fraction y such as its pressure, P = (1 + y + xHe )nH kB T , and its internal
energy, E = 32 (1 + y + xHe )nH kB T + yχH , where xHe gives the ratio of neutral helium to the
total amount of hydrogen. The dependence of temperature on entropy is shown in Fig. 5
for different values of the density.
For further details regarding the procedure of solving for the entropy see Sect. H.6 in
the appendix.
6.8.1 Ambipolar diffusion
Another way of dealing with ionization in the P ENCIL CODE is through use of the neutrals module. That module solves the coupled equations of neutral and ionized gas, in a
two-fluid model

66

T HE P ENCIL C ODE

∂ρi
∂t
∂ρn
∂t
∂(ρi ui )
∂t
∂(ρn un )
∂t
∂A
∂t

(69)

= −∇ · (ρi ui ) + G

(70)

= −∇ · (ρn un ) − G


= −∇ · (ρi ui : ui ) − ∇ pi + pe +


2

B
2µ0

+F

(71)

= −∇ · (ρn un : un ) − ∇pn − F

(72)

= ui × B

(73)

where the subscripts n and i are for neutral and ionized, respectively. The terms G and
F, through which the two fluids exchange mass and momentum, are given by
G = ζρn − αρ2i
F = ζρn un − αρ2i ui + γρi ρn (un − ui ) .

(74)
(75)

In the above equations, ζ is the ionization coefficient, α is the recombination coefficient,
and γ the collisional drag strength. By the time of writing (spring 2009), these three
quantities are supposed constant. The electron pressure pe is also assumed equal to the
ion pressure. Only isothermal neutrals are supported so far.
In the code, Eq. (69) and Eq. (71) are solved in ‘density.f90’ and ‘hydro.f90’ respectively.
Equation 70 is solved in ‘neutraldensity.f90’ and Eq. (72) in ‘neutralvelocity.f90’. The
sample ‘1d-test/ambipolar-diffusion’ has the current setup for a two-fluid simulation
with ions and neutrals.

6.9

Radiative transfer

Here we only state the basic equations. A full description about the implementation is
given in Sect. H.7 and in the original paper by Heinemann et al. (2006).
The basic equation for radiative transfer is
dI
= −I + S ,
dτ
where
τ≡

Zs

χ(s′ ) ds′

(76)

(77)

0

is the optical depth (s is the geometrical coordinate along the ray).
Note that radiative transfer is called also in ‘start.csh’, and again each time a snapshot
is being written, provided the output of auxiliary variables is being requested lwrite_aux=T. (Also, of course, the pencil check runs radiative transfer 7 times, unless you put
pencil_check_small=F.)

6.10 Self-gravity

67

6.10 Self-gravity
The P ENCIL C ODE can consider the self-gravity of the fluid in the simulation box by
adding the term
∂u
= . . . − ∇φself
(78)
∂t
to the equation of motion. The self-potential φself (or just φ for simplicity) satisfies Poisson’s equation
∇2 φ = 4πGρ .
(79)
The solution for a single Fourier component at scale k is
φk = −

4πGρk
.
k2

(80)

Here we have assumed periodic boundary conditions. The potential is obtained by
Fourier-transforming the density, then finding the corresponding potential at that scale,
and finally Fourier-transforming back to real space.
The x-direction in the shearing sheet is not strictly periodic, but is rather shear periodic
with two connected points at the inner and outer boundary separated by the distance
∆y(t) = mod[(3/2)Ω0 Lx t, Ly ] in the y-direction. We follow here the method from [16]
to allow for shear-periodic boundaries in the Fourier method for self-gravity. First we
take the Fourier transform along the periodic y-direction. We then shift the entire ydirection by the amount δy(x) = ∆y(t)x/Lx to make the x-direction periodic. Then we
proceed with Fourier transforms along x and then z. After solving the Poisson equation
in Fourier space, we transform back to real space in the opposite order. We differ here
from the method by [16] in that we shift in Fourier space rather than in real space13 .
The Fourier interpolation formula has the advantage over polynomial interpolation in
that it is continuous and smooth in all its derivatives.
6.11 Incompressible and anelastic equations
This part has not yet been documented and is still under development.
6.12 Dust equations
The code treats gas and dust as two separate fluids14 . The dust and the gas interact
through a drag force. This force can most generally be written as an additional term to
the equation of motion as
Dud
1
= . . . − (ud − u) .
(81)
Dt
τs
Here τs is the so-called stopping time of the considered dust species. This measures the
coupling strength between dust and gas. In the Epstein drag regime
τs =
13

ad ρs
,
cs ρ

(82)

We were kindly made aware of the possibility of interpolating in Fourier space by C. McNally on his
website.
14
See master’s thesis of A. Johansen (can be downloaded from
http://www.mpia.de/homes/johansen/research_en.php)

68

T HE P ENCIL C ODE

where ad is the radius of the dust grain and ρs is the solid density of the dust grain.
Two other important effects work on the dust. The first is coagulation controlled
by the discrete coagulation equation
∞
X
dnk
1 X
=
Aij ni nj − nk
Aik ni .
dt
2 i+j=k
i=1

(83)

In the code N discrete dust species are considered. Also the bins are logarithmically
spaced in order to give better mass resolution. It is also possible to keep track of both
number density and mass density of each bin, corresponding to having a variable grain
mass in each bin.
Dust condensation is controlled by the equation
d−1
1
dN
=
N d .
dt
τcond

(84)

Here N is the number of monomers in the dust grain (such as water molecules) and d is
the physical dimension of the dust grain. The condensation time τcond is calculated from


1
1
,
(85)
= A1 vth αnmon 1 −
τcond
Smon
where A1 is the surface area of a monomer, α is the condensation efficiency, nmon is the
number density of monomers in the gas and Smon is the saturation level of the monomer
given by
Pmon
.
(86)
Smon =
Psat
Here Psat is the saturated vapor pressure of the monomer. Currently only water ice has
been implemented in the code.
All dust species fulfill the continuity equation
∂ρd
+ ∇ · (ρd ud ) = 0.
∂t
6.13

(87)

Cosmic ray pressure in diffusion approximation

Cosmic rays are treated in the diffusion approximation. The equation of state is pc =
(γc )ec where the value of γc is usually somewhere between 14/9 and 4/3. In the momentum equation (39) the cosmic ray pressure force, −ρ−1 ∇pc is added on the right hand
side, and ec satisfies the evolution equation
∂ec
+ ∇ · (ec u) + pc ∇ · u = ∂i (Kij ∂j ec ) + Qc ,
∂t

(88)

where Qc is a source term and
Kij = K⊥ δij + (Kk − K⊥ )B̂i B̂j
is an anisotropic diffusivity tensor.

(89)

6.14 Particles

69

In the non-conservative formulation of this code it is advantageous to expand the diffusion term using the product rule, i.e.
∂i (Kij ∂j ec ) = −U c · ∇ec + Kij ∂i ∂j ec .

(90)

where Uc i = −∂Kij /∂xj acts like an extra velocity trying to straighten magnetic field
lines. We can write this term also as U c = −(Kk − K⊥ )∇ · (B̂ B̂), where the last term
is a divergence of the dyadic product of unit vectors.15 However, near magnetic nulls,
this term can becomes infinite. In order to avoid this problem we are forced to limit
∇ · (B̂ B̂), and hence |U c |, to the maximum possible value that can be resolved at a given
resolution.
A physically appealing way of limiting the maximum propagation speed is to restore
an explicit time dependence in the equation for the cosmic ray flux, and to replace the
diffusion term in Eq. (88) by a divergence of a flux that in turn obeys the equation
∂Fci
Fci
= −K̃ij ∇j ec −
∂t
τ

(non-Fickian diffusion),

(91)

where Kij = τ K̃ij would be the original diffusion tensor of Eq. (89), if the time derivative
were negligible. Further details are described in Snodin et al. (2006).
6.14 Particles
Particles are entities that each have a space coordinate and a velocity vector, where
a fluid only has a velocity vector field (the continuity equation of a fluid in some way
corresponds to the space coordinate of particles). In the code particles are present either
as tracer particles or as dust particles
6.14.1 Tracer particles
Tracer particles always have the local velocity of the gas. The dynamical equations are
thus
∂xi
= u,
(92)
∂t
where the index i runs over all particles. Here u is the gas velocity at the position of
the particle. One can choose between a first order (default) and a second order spline
interpolation scheme (set lquadratic_interpolation=T in &particles_init_pars) to calculate the gas velocity at the position of a tracer particle.
The sample run ‘samples/dust-vortex’ contains the latest setup for tracer particles.
6.14.2 Dust particles
Dust particles are allowed to have a velocity that is not similar to the gas,
dxi
= vi .
dt
15

(93)

In practice, we calculate ∂j (B̂i B̂j ) = (δij − 2B̂i B̂k )B̂j Bk,j /|B|, where derivatives of B are calculated as
Bi,j = ǫikl Al,jk .

70

T HE P ENCIL C ODE

The particle velocity follows an equation of motion similar to a fluid, only there is no
advection term. Dust particles also experience a drag force from the gas (proportional to
the velocity difference between a particle and the gas).
dv i
1
= . . . − (v i − u) .
dt
τs

(94)

Here τs is the stopping time of the dust particle. The interpolation of the gas velocity to
the position of a particle is done using one of three possible particle-mesh schemes,
• NGP (Nearest Grid Point, default)
The gas velocity at the nearest grid point is used.
• CIC (Cloud in Cell, set lparticlemesh_cic=T)
A first order interpolation is used to obtain the gas velocity field at the position of
a particle. Affects 8 grid points.
• TSC (Triangular Shaped Cloud, set lparticlemesh_tsc=T)
A second order spline interpolation is used to obtain the gas velocity field at the
position of a particle. Affects 27 grid points.
The particle description is the proper description of dust grains, since they do not feel
any pressure forces (too low number density). Thus there is no guarantee that the grains
present within a given volume will be equilibrated with each other, although drag force
may work for small grains to achieve that. Larger grains (meter-sized in protoplanetary
discs) must be treated as individual particles.
To conserve momentum the dust particles must affect the gas with a friction force as
well. The strength of this force depends on the dust-to-gas ratio ǫd , and it can be safely
ignored when there is much more gas than there is dust, e.g. when ǫd = 0.01. The friction
force on the gas appears in the equation of motion as
(i)

ρp
∂u
= ... −
∂t
ρ



∂v (i)
∂t



(95)
drag

(i)

Here ρp is the dust density that particle i represents. This can be set through the parameter eps_todt in &particle_init_pars. The drag force is assigned from the particles
onto the mesh using either NGP, CIC or TSC assignment. The same scheme is used both
for interpolation and for assignment to avoid any risk of a particle accelerating itself (see
Hockney & Eastwood 1981).
6.15

N -body solver

The N -body code takes advantage of the existing Particles module, which was coded with
the initial intent of treating solid particles whose radius a• is comparable to the mean
free path λ of the gas, for which a fluid description is not valid. A N -body implementation
based on that module only needed to include mass as extra state for the particles, solve
for the N 2 gravitational pair interactions and distinguish between the N -body and the
small bodies that are mapped into the grid as a ρp density field.
The particles of the N -body ensemble evolve due to their mutual gravity and by interacting with the gas and the swarm of small bodies. The equation of motion for particle i
is

6.16 Test-field equations

71

N

X GMj
dv pi
= F gi −
R̂ij
2
dt
R
ij
j6=i

(96)

where Rij = |r pi −r pj | is the distance between particles i and j, and R̂ij is the unit vector
pointing from particle j to particle i. The first term of the R.H.S. is the combined gravity
of the gas and of the dust particles onto the particle i, solved via
[ρg (r) + ρp (r)]Ri
dV,
(97)
(R2i + b2i )3/2
V
where the integration is carried out over the whole disk. The smoothing distance bi is
taken to be as small as possible (a few grid cells). For few particles (<10), calculating the
integral for every particle is practical. For larger ensembles one would prefer to solve
the Poisson equation to calculate their combined gravitational potential.
F gi = −G

Z

The evolution of the particles is done with the same third-order Runge-Kutta timestepping routine used for the gas. The particles define the timestep also by the Courant
condition that they should not move more than one cell at a time. For pure particle runs,
−1
where the grid is absent, one can adopt a fixed time-step tp ≪ 2πΩfp
where Ωfp is the
angular frequency of the fastest particle.
By now (spring 2009), no inertial accelerations are included in the N -body module, so
only the inertial frame - with origin at the barycenter of the N -body ensemble - is available. For a simulation of the circular restricted three-body problem with mass ratio
q=10−3 , the Jacobi constant of a test particle initially placed at position (x, y)=(2,0) was
found to be conserved up to one part in 105 within the time span of 100 orbits.
We stress that the level of conservation is poor when compared to integrators designed
to specifically deal with long-term N -body problems. These integrators are usually symplectic, unlike the Runge-Kutta scheme of the P ENCIL C ODE. As such, P ENCIL should
not be used to deal with evolution over millions of years. But for the time-span typical
of astrophysical hydrodynamical simulations, this degree of conservation of the Jacobi
constant can be deemed acceptable.
As an extension of the particle’s module, the N -body is fully compatible with the parallel optimization of P ENCIL, which further speeds up the calculations. Parallelization,
however, is not yet possible for pure particle runs, since it relies on splitting the grid
between the processors. At the time of writing (spring 2009), the N -body code does not
allow the particles to have a time-evolving mass.
6.16 Test-field equations
The test-field method is used to calculate turbulent transport coefficients for magnetohydrodynamics. This is a rapidly evolving field and we refer the interested reader to
recent papers in this field, e.g. by Sur et al. (2008) or Brandenburg et al. (2008). For
technical details, see also Sect. F.3.
6.17 Gravitational wave equations
The expansion of the universe with time is describedR by the scale factor a(τ ), where
τ
τ is the physical time. Using conformal time, t(τ ) = 0 dτ ′ /a(τ ′ ), and dependent vari-

72

T HE P ENCIL C ODE

Table 5: Scale factor and conformal Hubble parameter for different values of n.

n
0
1/2
2/3

a
1
η/2
η 2 /3

H
0
1/η
2/η

H
0
1/η
6/η 2

ables that are appropriately scaled with powers of a, the hydromagnetic equations can
be expressed completely without scale factor [8, 15]. This is not true, however, for the
gravitational wave (GW) equations, where a dependence on a remains [15]. The time
dependence of a can be modeled as a power law, a ∝ τ n , where n = 1/2 applies to the
radiation-dominated era; see Table 5 the general relationship. To compare with cases
where the expansion is ignored, we put n = 0.
In the transverse traceless (TT) gauge, the six components of the spatial part of the
symmetric tensor characterizing the linearized evolution of the metric perturbations
hij , reduce to two components which, in the linear polarization basis, are the + and
× polarizations. However, the projection onto that basis is computationally intensive,
because it requires nonlocal operations involving Fourier transformations. It is therefore
advantageous to evolve instead the perturbation of the metric tensor, hij , in an arbitrary
gauge, compute then hTT
ij in the TT gauge, and perform then the decomposition into the
linear polarization basis whenever we compute diagnostic quantities such as averages
or spectra. Thus, we solve the linearized GW equation in the form [15]
∂ 2 hij
∂hij
16πG
= −2H
+ c2 ∇2 hij + 2 2 Tij
2
∂t
∂t
ac

(98)

for the six components 1 ≤ i ≤ j ≤ 3, where t is comoving time, a is the scale factor,
H = ȧ/a is the comoving Hubble parameter, Tij is the source term, c is the speed of light,
and G is Newton’s constant. For n = 0, when the cosmic expansion is ignored, we have
a = 1 and H = 0. We use the P ENCIL C ODE; for the numerical treatment of Eq. (98) and
equations (100)–(102). For most of the simulations, we use 11523 meshpoints on 1152
cores of a Cray XC40 system with 2.3 GHz processors.
The source term is chosen to be the traceless part of the stress tensor,
Tij (x, t) = ρui uj − Bi Bj − 31 δij (ρu2 − B 2 ).

(99)

The removal of the trace is in principle not necessary, but it helps preventing a continuous build-up of a large trace, which would be numerically disadvantageous. We have
ignored here the viscous stress, which is usually small.
We compute Tij by solving the energy, momentum, and induction equations for an ultrarelativistic gas in the form [8, 10]

4
1
∂ ln ρ
= − (∇ · u + u · ∇ ln ρ) +
u · (J × B) + ηJ 2 ,
∂t
3
ρ

Du
u
u
u · (J × B) + ηJ 2
=
(∇ · u + u · ∇ ln ρ) −
Dt
3
ρ
1
3
2
− ∇ ln ρ + J × B + ∇ · (ρνS) + f ,
4
4ρ
ρ

(100)

(101)

6.17 Gravitational wave equations

73

∂B
= ∇ × (u × B − ηJ),
(102)
∂t
where B = ∇ × A is the magnetic field expressed in terms of the magnetic vector potential to ensure that ∇ · B = 0, J = ∇ × B is the current density, D/Dt = ∂/∂t + u · ∇ is
the advective derivative, Sij = 12 (ui,j + uj,i ) − 31 δij uk,k is the trace-free rate of strain tensor,
√
and p = ρc2s is the pressure, where cs = c/ 3 is the sound speed for an ultra-relativistic
gas. Lorentz-Heaviside units for the magnetic field are used.
We are interested in the rms value of the metric tensor perturbations and the GW energy
density in the linear polarization basis. To compute hTT
ij from hij , we Fourier transform
the six components of hij and ḣij ,
Z
h̃ij (k, t) = hij (x, t) e−ik·v d3 x for 1 ≤ i ≤ j ≤ 3
(103)
and compute the components in the TT gauge as
1
h̃TT
ij (k, t) = (Pil Pjm − 2 Pij Plm ) h̃lm (k, t),

(104)

where Pij = δij − k̂i k̂j is the projection operator, and k̂ = k/k is the unit vector of k, with
k = |k| being the modulus. Next, we compute the linear polarization bases
1 1
2 2
e+
ij = ei ej − ei ej ,

1 2
2 1
e×
ij = ei ej + ei ej ,

(105)

where e1 and e2 are unit vectors perpendicular to k. Thus
h̃+ (k, t) =
h˜× (k, t) =

1 +
e (k) h̃ij (k, t),
2 ij
1 ×
e (k) h̃ij (k, t).
2 ij

We then return into real space and compute
Z
h+/× (x, t) = h̃+/× (k, t) eik·x d3 k/(2π)3 .

(106)
(107)

(108)

Analogous calculations are performed for ḣ+/× (x, t), which are used to compute the GW
energy via

c2  2
hḣ+ i + hḣ2× i ,
(109)
EGW (t) =
32πG
where angle brackets denote volume averages.
Analogously to kinetic and magnetic energy and helicity spectra, it is convenient to
computeR the GW energy and polarization spectra integrated over concentric shells of
surface 4π k 2 dΩk in k space, defined by
Z 

(110)
Sḣ (k) =
|h̃˙ + |2 + |h̃˙ × |2 k 2 dΩk ,
4π
Z

∗
Aḣ (k) =
2 Im h̃˙ + h̃˙ × k 2 dΩk ,
(111)
4π
R∞
and normalized
such that 0 Sḣ (k) dk = hḣ2+ i + hḣ2× i is proportional to the energy density
R∞
and 0 Aḣ (k) dk is proportional to the polarized energy density. The Aḣ (k) spectra are
not to be confused with the magnetic vector potential A(x, t). The corresponding GW
energy spectra are noted by
EGW (k) = (c2 /32πG) Sḣ (k),
HGW (k) = (c2 /32πG) Aḣ (k).

(112)
(113)

74

T HE P ENCIL C ODE

We also define spectra for the metric tensor perturbation,
Z 

2
2
˜
˜
Sh (k) =
|h+ | + |h× | k 2 dΩk ,
4π
Z


∗
˜
˜
Ah (k) =
2 Im h+ h× k 2 dΩk ,

(114)
(115)

4π

which are normalized such that
perturbation.

R∞
0

Sh (k) dk = h2rms is the mean squared metric tensor

7. Troubleshooting / Frequently Asked Questions

75

7 Troubleshooting / Frequently Asked Questions
7.1

Download and setup

7.1.1 Download forbidden
A: Both GitHub and SourceForge are banned from countries on the United
States Office of Foreign Assets Control sanction list, including Cuba, Iran, Libya,
North Korea, Sudan and Syria; see http://de.wikipedia.org/wiki/GitHub and
http://en.wikipedia.org/wiki/SourceForge. As a remedy, you might download a tarball from http://pencil-code.nordita.org/; see also Section 2.

7.1.2 When sourcing the ‘sourceme.sh’/‘sourceme.csh’ file or running pc_setupsrc, I get
error messages from the shell, like ‘if: Expression Syntax.’ or ‘set: Variable name
must begin with a letter.’

A: This sounds like a buggy shell setup, either by yourself or your system administrator
— or a shell that is even more idiosyncratic than the ones we have been working with.
To better diagnose the problem, collect the following information before filing a bug
report to us:
1. uname -a
2. /bin/csh -v
3. echo $version
4. echo $SHELL
5. ps -p $$
6. If you have problems while sourcing the ‘sourceme’ script,
(a) unset the PENCIL_HOME variable:
for csh and similar: unsetenv PENCIL_HOME
for bash and similar: unexport PENCIL_HOME; unset PENCIL_HOME
(b) switch your shell in verbose mode,
for csh and similar: set verbose; set echo
for bash and similar: set -v; set -x
then source again.
7. If you have problems with pc_setupsrc, run it with csh in verbose mode:
/bin/csh -v -x $PENCIL_HOME/bin/pc_setupsrc

76

7.2

T HE P ENCIL C ODE

Compilation

7.2.1 Linker can’t find the syscalls functions:
ld:
ld:
ld:
ld:
ld:
ld:

0711-317
0711-317
0711-317
0711-317
0711-317
0711-317

ERROR:
ERROR:
ERROR:
ERROR:
ERROR:
ERROR:

Undefined
Undefined
Undefined
Undefined
Undefined
Undefined

symbol:
symbol:
symbol:
symbol:
symbol:
symbol:

.is_nan_c
.sizeof_real_c
.system_c
.get_env_var_c
.get_pid_c
.file_size_c

A: The Pencil Code needs a working combination of a Fortran- and a C-compiler. If this
is not correctly set up, usually the linker won’t find the functions inside the syscalls
module. If that happens, either the combination of C- and Fortran-compiler is inappropriate (e.g. ifort needs icc ), or the compiler needs additional flags, like g95 might need
the option ‘-fno-second-underscore’ and xlf might need the option ‘-qextname’. Please
refer to Sect. 5.2, Table 1.
7.2.2 Make gives the following error now:
PGF90-S-0017-Unable to open include file: chemistry.h (nochemistry.f90: 43)
0 inform,
0 warnings,
1 severes, 0 fatal for chemistry
Line 43 of the nochemistry routine, only has ’contains’.
A: This is because somebody added a new module (together with a corresponding nomodule.f90 and a module.h file (chemistry in this case). These files didn’t exist before, so you
need to say:
pc_setupsrc
If this does not help, say first make clean and then pc_setupsrc.
7.2.3 How do I compile the P ENCIL C ODE with the Intel (ifc ) compiler under Linux ?
A: The P ENCIL C ODE should compile successfully with ifc 6.x, ifc 7.0, sufficiently recent
versions of ifc 7.1 (you should get the latest version; if yours is too old, you will typically
get an ‘internal compiler error’ during compilation of ‘src/hydro.f90’), as well as with
recent versions of ifort 8.1 (8.0 may also work).

You can find the ifort compiler at ftp://download.intel.com/software/products/compilers/downloa
On many current (as of November 2003) Linux systems, there is a mismatch between
the glibc versions used by the compiler and the linker. To work around this, use the
following flag for compiling
FC=ifc -i_dynamic
and set the environment variable
LD_ASSUME_KERNEL=2.4.1; export LD_ASSUME_KERNEL
or

7.2

Compilation

77

setenv LD_ASSUME_KERNEL 2.4.1
This has solved the problems e.g. on a system with glibc-2.3.2 and kernel 2.4.22.
Thanks to Leonardo J. Milano (http://udel.edu/~lmilano/) for part of this info.
7.2.4 I keep getting segmentation faults with ‘start.x’ when compiling with ifort 8.0
A: There was/is a number of issues with ifort 8.0. Make sure you have the latest patches
applied to the compiler. A number of things to consider or try are:
1. Compile with the the ‘-static -nothreads’ flags.
2. Set your stacksize to a large value (but a far too large value may be problematic,
too), e. g.
limit stacksize 256m
ulimit -s 256000
3. Set the environment variable KMP STACKSIZE to a large value (like 100M)
See also http://softwareforums.intel.com/ids/board/message?board.id=11&message.id=1375
7.2.5 When compiling with MPI on a Linux system, the linker complains:
mpicomm.o: In function
mpicomm.o(.text+0x36):
mpicomm.o(.text+0x55):
mpicomm.o(.text+0x6f):
[...]

‘mpicomm_mpicomm_init_’:
undefined reference to ‘mpi_init_’
undefined reference to ‘mpi_comm_size_’
undefined reference to ‘mpi_comm_rank_’

A: This is the infamous underscore problem . Your MPI libraries have been compiled
with G77 without the option ‘-fno-second-underscore’, which makes the MPI symbol
names incompatible with other Fortran compilers.
As a workaround, use
MPICOMM = mpicomm_
in ‘Makefile.local’. Or, even better, you can set this globally (for the given computer)
by inserting that line into the file ‘~/.adapt-mkfile.inc’ (see perldoc adapt-mkfile for
more details).
7.2.6 Compilation stops with the cryptic error message:
f95 -O3 -u -c .f90.f90
Error : Could not open sourcefile .f90.f90
compilation aborted for .f90.f90 (code 1)
make[1]: *** [.f90.o] Error 1
What is the problem?
A: There are two possibilities:

78

T HE P ENCIL C ODE
1. One of the variables for make has not been set, so make expands it to the empty
string. Most probably you forgot to specify a module in ‘src/Makefile.local’. One
possibility is that you have upgraded from an older version of the code that did not
have some of the modules the new version has.
Compare your ‘src/Makefile.local’ to one of the examples that work.
2. One of the variables for make has a space appended to it, e.g. if you use the line
MPICOMM = mpicomm_
(see § 7.2.5) with a trailing blank, you will encounter this error message. Remove
the blank. This problem can also occur if you added a new module (and have an
empty space after the module name in ‘src/Makefile.src’, i.e. CHIRAL=nochiral ),
in which case the compiler will talk about “circular dependence” for the file
‘nochiral’.

7.2.7 The code doesn’t compile,
. . . there is a problem with mvar :
make start.x run.x
f95 -O3 -u
-c cdata.f90
Error: cdata.f90, line 71: Implicit type for MVAR
detected at MVAR@)
[f95 terminated - errors found by pass 1]
make[1]: *** [cdata.o] Error 2
A: Check and make sure that ‘mkcparam’ (directory ‘$PENCIL_HOME/bin’) is in your path. If
this doesn’t help, there may be an empty ‘cparam.inc’ file in your ‘src’ directory. Remove
‘cparam.inc’ and try again (Note that ‘cparam.inc’ is automatically generated from the
‘Makefile’).

7.2.8 Some samples don’t even compile,
as you can see on the web, http://www.nordita.org/software/pencil-code/tests.html.
samples/helical-MHDturb:
Compiling..
not ok:
make start.x run.x read_videofiles.x
make[1]: Entering directory ‘/home/dobler/f90/pencil-code/samples/helical-MHDturb/src’
/usr/lib/lam/bin/mpif95 -O3
-c initcond.f90
/usr/lib/lam/bin/mpif95 -O3
-c density.f90
use Gravity, only: gravz, nu_epicycle
^
Error 208 at (467:density.f90) : No such entity in the module
Error 355 : In procedure INIT_LNRHO variable NU_EPICYCLE has not been given a type
Error 355 : In procedure POLYTROPIC_LNRHO_DISC variable NU_EPICYCLE has not been given a
3 Errors
compilation aborted for density.f90 (code 1)
make[1]: *** [density.o] Error 1

7.2

Compilation

79

make[1]: Leaving directory ‘/home/dobler/f90/pencil-code/samples/helical-MHDturb/src’
make: *** [code] Error 2
A: Somebody may have checked in something without having run auto-test beforehand.
The problem here is that something has been added in one module, but not in the corresponding no-module. You can of course check with svn who it was. . .

7.2.9 Internal compiler error with Compaq/Dec F90
The Dec Fortran optimizer has occasional problems with ‘nompicomm.f90’:
make start.x run.x read_videofiles.x
f90 -fast -O3 -tune ev6 -arch ev6 -c cparam.f90
[...]
f90 -fast -O3 -tune ev6 -arch ev6 -c nompicomm.f90
otal vm 2755568
otal vm 2765296
otal vm 2775024
otal vm 2784752
otal...
Assertion failure: Compiler internal error - please submit problem r...
GEM ASSERTION, Compiler internal error - please submit problem report
Fatal error in: /usr/lib/cmplrs/fort90_540/decfort90 Terminated
*** Exit 3
Stop.
*** Exit 1
Stop.
A: The occurrence of this problem depends upon the grid size; and the problem never
seems to occur with ‘mpicomm.f90’, except when ncpus=1. The problem can be avoided by
switching off the loop transformation optimization (part of the ‘-O3’ optimization), via:
#OPTFLAGS=-fast -O3 -notransform_loops
This is currently the default compiler setting in ‘Makefile’, although it has a measurable
performance impact (some 8% slowdown).

7.2.10 Assertion failure under SunOS
Under SunOS, I get an error message like
user@sun> f90 -c param_io.f90
Assertion failed: at_handle_table[at_idx].tag == VAR_TAG,
file ../srcfw/FWcvrt.c, line 4018
f90: Fatal error in f90comp: Abort
A: This is a compiler bug that we find at least with Sun’s WorkShop Compiler version ‘5.0
00/05/17 FORTRAN 90 2.0 Patch 107356-05’. Upgrade the compiler version (and possibly also the operating system): we find that the code compiles and works with version
‘Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-05 2002/01/17’ under SunOS
version ‘5.8 Generic 108528-11’.

80

T HE P ENCIL C ODE

7.2.11 After some dirty tricks I got pencil code to compile with MPI, ...
> Before that i installed lam-7.1.4 from source.
Goodness gracious me, you shouldn’t have to compile your own MPI library.
A: Then don’t use the old LAM-MPI. It is long superseded by open-mpi now. Open-mpi
doesn’t need a daemon to be running. I am using the version that ships with Ubuntu
(e.g. 9.04):
frenesi:~> aptitude -w 210 search openmpi | grep ’^i’
i
libopenmpi-dev
i A libopenmpi1
i
openmpi-bin
i A openmpi-common
i
openmpi-doc

-

high
high
high
high
high

performance
performance
performance
performance
performance

message
message
message
message
message

passing
passing
passing
passing
passing

library
library
library
library
library

------

header files
shared library
binaries
common files
man pages

Install that and keep your configuration (Makefile.src and getconf.csh) close to that for
‘frenesi’ or ‘norlx50’. That should work.

7.2.12 Error: Symbol ’mpi comm world’ at (1) has no IMPLICIT type
I installed the pencil code on Ubuntu system and tested "run.csh"
in ...\samples\conv-slab. Here the code worked pretty well.
Nevertheless, running (auto-test), I found there are some errors.
The messages are,
Error: Symbol ’mpi_comm_world’ at (1) has no IMPLICIT type
Fatal Error: Error count reached limit of 25.
make[2]: *** [mpicomm_double.o] Error 1
make[2]: Leaving directory
‘/home/pkiwan/Desktop/pencil-code/samples/2d-tests/selfgravitating-shearwave/src’
make[1]: *** [code] Error 2
make[1]: Leaving directory
‘/home/pkiwan/Desktop/pencil-code/samples/2d-tests/selfgravitating-shearwave/src’
make: *** [default] Error 2
Finally, ### auto-test failed ###
Will it be OK? Or, how can I fix this?
A: Thanks for letting me know about the status, and congratulations on your progress!
Those tests that fail are those that use MPI. If your machine is a dual or multi core
machine, you could run faster by running under MPI. But this is probably not crucial
for you at this point. (I just noticed that there is a ToDo listed in the auto-test command
to implement the option not to run the MPI tests, but this hasn’t been done yet. So I
guess you can start with the science next.

7.3

Pencil check

81

7.2.13 Error: Can’t open included file ’mpif.h’
It always worked, but now, after some systems upgrade, I get
gfortran -O3 -o mpicomm.o -c mpicomm.f90
Error: Can’t open included file ’mpif.h’
When I say locate mpif.h I only get things like
/scratch/ntest/1.2.7p1-intel/include/mpif.h
But since I use FC=mpif90 I thought I don’t need to worry.
A: Since you use FC=mpif90 there must definitely be something wrong with their setup.
Try mpif90 -showme or mpif90 -show; the ‘-I’ option should say where it looks for ’mpif.h’.
If those directories don’t exist, it’s no wonder that it doesn’t work, and it is time to
complain.

7.3

Pencil check

7.3.1 The pencil check complains for no reason.
A: The pencil check only complains for a reason.

7.3.2 The pencil check reports MISSING PENCILS and quits
A: This could point to a serious problem in the code. Check where the missing pencil
is used in the code. Request the right pencils, likely based on input parameters, by
adapting one or more of the pencil_criteria_MODULE subroutines.

7.3.3 The pencil check reports unnecessary pencils
The pencil check reports possible overcalculation... pencil rho ( 43) is
requested, but does not appear to be required!
A: Such warnings show that your simulation is possibly running too slowly because it is
calculating pencils that are not actually needed. Check in the code where the unnecessary pencils are used and adapt one or more of the pencil_criteria_MODULE subroutines
to request pencils only when they are actually needed.

7.3.4 The pencil check reports that most or all pencils are missing
A: This is typically a thing that can happen when testing new code development for the
first time. It is usually an indication that the reference df changes every time you call
pde. Check whether any newly implemented subroutines or functionality has a “memory”, i.e. if calling the subroutine twice with the same f gives different output df.

82

T HE P ENCIL C ODE

7.3.5 Running the pencil check triggers mathematical errors in the code
A: The pencil check puts random numbers in f before checking the dependence of df on
the chosen set of pencils. Sometimes these random numbers are inconsistent with the
physics and cause errors. In that case you can set lrandom_f_pencil_check=F in &run_pars in ‘run.in’. The initial condition may contain many idealized states (zeros or ones)
which then do not trigger pencil check errors when lrandom_f_pencil_check=F, even if
pencils are missing. But it does prevent mathematical inconsistencies.
7.3.6 The pencil check still complains
A: Then you need to look into the how the code and the pencil check operate. Reduce the
problem in size and dimensions to find the smallest problem that makes the pencil check
fail (e.g. 1x1x8 grid points). At the line of ‘pencil_check.f90’ when a difference is found
between df_ref and df, add some debug lines telling you which variable is inconsistent
and in what place. Often you will be surprised that the pencil check has correctly found
a problem in the simulation.
7.3.7 The pencil check is annoying so I turned it off
A: Then you are taking a major risk. If one or more pencils are not calculated properly,
then the results will be wrong.
7.4

Running

7.4.1 Why does ‘start.x’ / ‘start.csh’ write data with periodic boundary conditions?
A: Because you are setting the boundary conditions in ‘run.in’, not in ‘start.in’; see
Sect. 5.16.1. There is nothing wrong with the initial data — the ghost-zone values will
be re-calculated during the very first time step.
7.4.2 csh problem?
Q: On some rare occasions we have problems with csh not being supported on other
machines. (We hope to fix this by contacting the responsible person, but may not be that
trivial today!) Oliver says this is a well known bug of some years ago, etc. But maybe in
the long run it would be good to avoid csh.
A: These occasions will become increasingly frequent, and eventually for some architectures, there may not even be a csh variant that can be installed.
We never pushed people to use pc_run and friends (and to report corresponding bugs
and get them fixed), but if we don’t spend a bit of effort (or annoy users) now, we create
a future emergency, where someone needs to run on some machine, but there is no csh
and he or she just gets stuck.
We don’t have that many csh files, and for years now it should be possible to compile
run without csh (using bin/pc_run) — except that people still fall back on the old way of

7.4 Running

83

doing things. This is both cause and consequence of the ‘new’ way not being tested that
much, at least for the corner cases like ‘RERUN’, ‘NEWDIR’, ‘SCRATCH_DIR’.

7.4.3 ‘run.csh’ doesn’t work:
Invalid character ’’’ in NAMELIST input
Program terminated by fatal I/O error
Abort
A: The string array for the boundary condition, e.g. bcx or bcz is too long. Make sure it
has exactly as many elements as nvar is big.

7.4.4 Code crashes after restarting
> > removing mu_r from the namelist just ‘like that’ makes the code
> > backwards incompatible.
>
> That means that we can never get rid of a parameter in start.in once we
> have introduced it, right?
A: In the current implementation, without a corresponding cleaning procedure, unfortunately yes.
Of course, this does not affect users’ private changes outside the central svn tree.

7.4.5 auto-test gone mad...?
Q: Have you ever seen this before:
giga01:/home/pg/n7026413/cvs-src/pencil-code/samples/conv-slab> auto-test
.
/home/pg/n7026413/cvs-src/pencil-code/samples/conv-slab:
Compiling..
ok
No data directory; generating data -> /var/tmp/pencil-tmp-25318
Starting..
ok
Running..
ok
Validating results..Malformed UTF-8 character (unexpected continuation
byte 0x80, with no preceding start byte) in split at
/home/pg/n7026413/cvs-src/pencil-code/bin/auto-test line 263.
Malformed UTF-8 character (unexpected continuation byte 0x80, with no
preceding start byte) in split at
/home/pg/n7026413/cvs-src/pencil-code/bin/auto-test line 263.
A: You are running on a RedHat 8 or 9 system, right?
Set LANG=POSIX in your shell’s startup script and life will be much better.

84

T HE P ENCIL C ODE

7.4.6 Can I restart with a different number of cpus?
Q: I am running a simulation of nonhelical turbulence on the cluster using MPI. Suppose
if I am running a 1283 simulation on 32 cpus/cores i.e.
integer, parameter :: ncpus=32,nprocy=2,nprocz=ncpus/nprocy,nprocx=1
integer, parameter :: nxgrid=128,nygrid=nxgrid,nzgrid=nxgrid
And I stop the run after a bit. Is there a way to resume this run with different number
of cpus like this :
integer, parameter :: ncpus=16,nprocy=2,nprocz=ncpus/nprocy,nprocx=1
integer, parameter :: nxgrid=128,nygrid=nxgrid,nzgrid=nxgrid
I understand it has to be so in a new directory but making sure that the run starts from
where I left it off in the previous directory.
A: The answer is no, if you use the standard distributed io. There is also parallel io, but
I never used it. That would write the data in a single file, and then you could use the
data for restart in another processor layout.

7.4.7 Can I restart with a different number of cpus?
Q: Is it right that once the simulation is resumed, pencil-code takes the last data from
var.dat (which is the current snapshot of the fields)? If that is true, then, is it not possible
to give that as the initial condition for the run in the second directory (with changed
”ncpus”)? Is there a mechanism already in place for that?
A: Yes, the code restarts from the last var.dat. It is written after a successful completion
of the run, but it crashes or you hit a time-out, there will be a var.dat that is overwritten
every isave timesteps. If the system stops during writing, some var.dat files may be
corrupt or have the wrong time. In that case you could restart from a good VAR file, if
you have one, using, e.g.,
restart-new-dir-VAR . 46
where 46 is the number of your VAR file, i.e., VAR46 im this case. To restart in another
directory, you say, from the old run directory,
restart-new-dir ../another_directory
Hope this helps. Look into pencil-code/bin/restart-new-dir to see what it is doing.

7.4.8 fft xyz parallel 3D: nygrid needs to be an integer multiple...
Q: I just got an:
fft_xyz_parallel_3D: nygrid needs to be an integer multiple of nprocy*nprocz
In my case, nygrid=2048, nprocy=32, and nprocz=128, so nprocy*nprocz=4096. In other
words, 2048 needs to be a multiple of 4096. But isn’t this the case then?
A: No, because 2048 = 0.5 * 4096 and 0.5 is not an integer. Maybe try either setting
nprocz=64 or nprocy=64. You could compensate the change of ncpus with the x-direction.

7.4 Running

85

For 20483 simulations, nprocy=32 and nprocz=64 would be good. A list of good meshes is
given in Table 4.
7.4.9 Unit-agnostic calculations?
Q: The manual speaks about unit-agnostic calculations, stating that one may choose to
interpret the results in any (consistent) units, depending on the problem that is solved
at hand. So, for example, if I chose to run the ‘2d-tests/battery_term’ simulation for an
arbitrary number of time-steps and then choose to examine the diagnostics, am I correct
in assuming the following:
1)
2)
3)
4)

[Brms] = Gauss (as output by unit_magnetic, before the run begins)
[t] = s (since the default unit system is left as CGS)
[urms] = cm/s (again, as output by unit_velocity, before the run begins)
and etc. for the units of the other diagnostics

A:
Detailed
correspondence
on
this
item
can
be
found
on:
https://groups.google.com/forum/?fromgroups#!topic/pencil-code-discuss/zek-uYNbgXI
There
is
also
working
material
on
unit
systems
under
http://www.nordita.org/~brandenb/teach/PencilCode/MixedTopics.html with a link to
http://www.nordita.org/~brandenb/teach/PencilCode/material/AlfvenWave_SIunits/
Below is a pedagogical response from Wlad Lyra:
In the sample battery-term, the sound speed cs0=1 sets the unit of
velocity. Together with the unit of length, that sets your unit of
time. The unit of magnetic field follows from the unit of velocity,
density, and your choice of magnetic permittivity, according to the
definition of the Alfven velocity.
If you are assuming cgs, you are saying that your sound speed cs0=1
actually means [U]=1 cm/s. Your unit of length is equivalently 1 cm,
and therefore the unit of time is [t] = [L]/[U]=1 s. The unit of
density is [rho] = 1 g/cm^3. Since in cgs vA=B/sqrt(4*pi * rho), your
unit of magnetic field is [B] = [U] * sqrt([rho] * 4*pi) ~= 3.5
sqrt(g/cm) / s = 3.5 Gauss.
If instead you are assuming SI, you have cs0=1 assuming that means
[U]=1 m/s and rho0=1 assuming that to mean [rho]=1 kg/m^3. Using [L]=1
m, you have still [t]=1 s, but now what appears as B=1 in your output
is actually [B] = [U] * sqrt (mu * [rho]) = 1 m/s * sqrt(4*pi * 1e-7
N*A-2 1 kg/m^3) ~= 0.0011210 kg/(s^2*A) ~ 11 Gauss.
You can make it more interesting and use units relevant to the
problem. Say you are at the photosphere of the Sun. You may want to
use dimensionless cs0=1 meaning a sound speed of 10 km/s. Your
appropriate length can be a megameter. Now your time unit is
[t]=[L]/[U] = 1e3 km/ 10 km/s = 10^2 s, i.e., roughly 1.5 minute. For
density, assume rho=2x10-4 kg/m^3, typical of the solar photosphere.
Your unit of magnetic field is therefore [B] = [U] * sqrt([rho] *
4*pi) = 1e6 cm/s * sqrt(4*pi * 2e-7 g/cm^3) ~ 1585.33 Gauss.
Notice that for mu0=1 and rho0=1 you simply have vA=B. Then you can
conveniently set the field strength by your choice of plasma beta (=

86

T HE P ENCIL C ODE

2*cs^2/vA^2). There’s a reason why we like dimensionless quantities!

7.5

Visualization

7.5.1 ‘start.pro’ doesn’t work:
Reading grid.dat..
Reading param.nml..
\% Expression must be a structure in this context: PAR.
\% Execution halted at: \$MAIN\$
104
/home/brandenb/pencil-code/runs/forced/hel1/../../../idl/start.pro
A: You don’t have the subdirectory ‘data’ in your IDL variable !path . Make sure you
source ‘sourceme.csh’/‘sourceme.sh’ or set a sufficient IDL path otherwise.

7.5.2 ‘start.pro’ doesn’t work:
Isn’t there some clever (or even trivial) way that one can avoid the annoying error messages that one gets, when running e.g. ”.r rall” after a new variable has been introduced
in ”idl/varcontent.pro”? Ever so often there’s a new variable that can’t be found in my
param2.nml – this time it was IECR, IGG, and ILNTT that I had to circumvent. . .
A: The simplest solution is to invoke ‘NOERASE’, i.e. say
touch NOERASE
start.csh
or, alternatively, start_run.csh. What it does is that it reruns src/start.x with a new
version of the code; this then produces all the necessary auxiliary files, but it doesn’t
overwrite or erase the ‘var.dat’ and other ‘VAR’ and ‘slice’ files.

7.5.3 Something about tag name undefined:
Q: In one of my older run directories I can’t read the data with idl anymore. What should
I do? Is says something like
Reading param.nml..
% Tag name LEQUIDIST is undefined for structure .
% Execution halted at: $MAIN$
182
/people/disk2/brandenb/pencil-code/idl/start.pro
A: Go into ‘data/param.nml’ and add , LEQUIDIST=T anywhere in the file (but before the
last slash).

7.5.4 Something INC in start.pro
Q: start doesn’t even work:

7.5

Visualization

87

% Compiled module: $MAIN$.
nname=
11
Reading grid.dat..
Reading param.nml..
Can’t locate Namelist.pm in INC (INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr
BEGIN failed--compilation aborted at /home/brandenb/pencil-code/bin/nl2idl line 49.
A: Go into ‘$PENCIL_HOME’ and say svn up sourceme.csh and/or svn up sourceme.sh.
(They were just out of date.)
7.5.5 nl2idl problem when reading param2.nml
Q: Does anybody encounter a backward problem with nl2idl? The file param*.nml files
are checked in under ‘pencil-code/axel/couette/SStrat128a_mu0.20_g2’ and the problem is below.
at /people/disk2/brandenb/pencil-code/bin/nl2idl line 120
HCOND0= 0.0,HCOND1= 1.000000,HCOND2= 1.000000,WIDTHSS= 1.192093E-06,MPOLY0=
^------ HERE
at /people/disk2/brandenb/pencil-code/bin/nl2idl line 120
A: The problem is the stupid ifc compiler writing the following into the namelist file:
COOLING_PROFILE=’gaussian
’,COOLTYPE=’Temp
’COOL= 0.0,CS2COOL= 0.0,RCOOL= 1.000000,WCOOL= 0.1000000,FBOT= 0.0,CHI_T= 0.0
If you add a comma after the closing quote:
COOLING_PROFILE=’gaussian
’,COOLTYPE=’Temp
’,COOL= 0.0,CS2COOL= 0.0,RCOOL= 1.000000,WCOOL= 0.1000000,FBOT= 0.0,CHI_T= 0.0
things will work.
Note that ifc cannot even itself read what it is writing here, so if this happened to occur
in param.nml, the code would require manual intervention after each start.csh.
7.5.6 Spurious dots in the time series file
Q: Wolfgang, you explained it to me once, but I forget. How can one remove spurious
dots after the timestep number if the time format overflows?
A: I don’t know whether it exists anywhere, but it’s easy. In Perl you’d say
perl -pe ’s/^(\s*[-0-9]+)\.([-0-9eEdD])/$1 $2/g’
and in sed (but that’s harder to read)
sed ’s/^\( *[-0-9]\+\)\.\([-0-9eEdD]\)/\1 \2/g’
7.5.7 Problems with pc_varcontent.pro
Q:

88

T HE P ENCIL C ODE

% Subscript range values of the form low:high must be >= 0, < size, with low
<= high: VARCONTENT.
% Error occurred at: PC_VARCONTENT
391
/home/brandenb/pencil-code/idl/read/pc_varcontent.pro
%
PC_READ_VAR
318
/home/brandenb/pencil-code/idl/read/pc_read_var.pro
%
$MAIN$
A: Make sure you don’t have any unused items in your src/cparam.local such as
! MAUX CONTRIBUTION 3
! COMMUNICATED AUXILIARIES 3
They would leave gaps in the counting of entries in your data/index.pro file.
7.6

General questions

7.6.1 “Installation” procedure
Why don’t you use GNU autoconf/automake for installation of the P ENCIL C ODE?
A: What do you mean by “installation”? Unlike the applications that normally use autoconf , the Pencil Code is neither a binary executable, nor a library that you compile once
and then dump somewhere in the system tree. Autoconf is the right tool for these applications, but not for numerical codes, where the typical compilation and usage pattern is
very different:
You have different directories with different ‘Makefile.local’ settings, recompile after
introducing that shiny new term in your equations, etc. Moreover, you want to sometimes switch to a different compiler (but just for that run directory) or another MPI
implementation. Our adapt-mkfile approach gives you this flexibility in a reasonably
convenient way, while doing the same thing with autoconf would be using that system
against most of its design principles.
Besides, it would really get on my (WD’s) nerves if I had to wait two minutes for autoconf
to finish before I can start compiling (or maybe 5–10 minutes if I worked on a NEC
machine. . . ).
Finally, if you have ever tried to figure out what a ‘configure’ script does, you will appreciate a comprehensible configuration system.
7.6.2 Small numbers in the code
What is actually the difference between epsi, tini and tiny?
A:
F90 has two functions epsilon() and tiny(), with
epsilon(x) = 1.1920929e-07
tiny(x)
= 1.1754944e-38
(and then there is huge(x) = 3.4028235e+38)

7.6

General questions

89

for a single-precision number x.
epsilon(x) is the smallest number that satisfies
1+epsilon(1.) /= 1 ,
while tiny(x) is the smallest number that can be represented without
precision loss.
In the code we have variants hereof,
epsi=5*epsilon(1.0)
tini=5*tiny(1.0)
huge1=0.2*huge(1.0)
that have added safety margins, so we don’t have to think about doing
things like 1/tini.
So in sub.f90,
evr = evr / spread(r_mn+epsi,2,3)
did (minimally) affect the result for r_mn=O(1), while the correct version
+
evr = evr / spread(r_mn+tini,2,3)
only avoids overflow.

7.6.3 Why do we need a /lphysics/ namelist in the first place?
Wolfgang answered on 29 July 2010: “‘cdata.f90’ has the explanation”
!
!
!
!

Constant ’parameters’ cannot occur in namelists, so in order to get the
now constant module logicals into the lphysics name list...
We have some proxies that are used to initialize private local variables
called lhydro etc, in the lphysics namelist!

So the situation is this: we want to write parameters like ldensity to param.nml so
IDL (and potentially octave, python, etc.) can know whether density was on or not. To
avoid confusion, we want them to have exactly their original names. But we cannot
assemble the original ldensity etc. constants in a namelist, so we have to define a local
ldensity variable. And to provide it with the value of the original cdata.ldensity, we need
to transfer the value via ldensity var . That’s pretty scary, although it seems to work
fine. I can track the code back to the big eos merger commit, so it may originate from
that branch. One obvious problem is that you have to add code in a number of places
(the ldensity → ldensity var assignment and the local definition of ldensity) to really
get what you need. And when adding a new boolean of that sort to ‘cdata.f90’, you may
not even have a clue that you need all the other voodoo.
There may be a cleaner solution involving generated code. Maybe something like
logical :: ldensity ! INCLUDE_IN_LPHYSICS
could later generate code (in some param io extra.inc file) that looks like this:
write(unit, *) ’ldensity = ’, ldensity
i.e. we can manually write in namelist format. But maybe there are even simpler solutions?

90

T HE P ENCIL C ODE

7.6.4 Can I run the code on a Mac?
A: Macs work well for Linux stuff, except that the file structure is slightly different.
Problems when following Linux installs can usually be traced to the PATH. For general
reference, if you need to set an environment variable for an entire OS-X login session,
google environment.plist. That won’t be needed here.
For a Mac install, the following should work:
a) Install Dev Tools (an optional install on the MacOS install disks). Unfortunately,
last time I checked the svn version that comes with DevTools is obsolete. So:
b) Install MacPorts (download from web). Note that MacPorts installs to a nonstandard location, and will need to be sourced. The installation normally drops
an appropriate line in .profile. If it does so, make sure that that line gets sourced.
Otherwise
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
export MANPATH=/opt/local/share/man:$MANPATH
c) Install g95 (download from web). Make sure it is linked in /bin.
d) execute macports svn install
e) download the pencil-code and enjoy.
Note: the above way to get svn works. It takes a while however, so there are certainly
faster ways out there. If you already have a non-obsolete svn version, use that instead.
7.6.5 Pencil Code discussion forum
Do I just need to send an email somewhere to subscribe or what?
A” The answer is yes; just go to:
http://groups.google.com/group/pencil-code-discuss
7.6.6 The manual
It would be a good idea to add this useful information in the manual, no?
A: When you have added new stuff to the code, don’t forget to mention this in the
‘pencil-code/doc/manual.tex’ file.
Again, the answer is yes; just go to:
cd pencil-code/doc/
vi manual.tex
svn ci -m "explanations about a new module in the code"

91

Part II

Programming the P ENCIL C ODE
All developers are supposed to have an up-to-date entry in the file
‘pencil-code/license/developers.txt’ so that they can be contacted in case a code
change breaks an auto-test or other code functionality.
Several P ENCIL C ODE committers have done several hundred check-ins, but many of
the currently 68 registered people on the repository have hardly done anything. To put
a number to this, one can define an h index, which gives the number of users, who have
done at least as many as that number of check-ins. This h index is currently 32, i.e., 32
users have done at least 32 check-ins; see Figure 6.

Figure 6: The h index of P ENCIL C ODE check-ins.

The P ENCIL C ODE has expanded approximately linearly in the number of lines of code
and the number of subroutines (Fig. 7). The increase in the functionality of the code is
documented by the rise in the number of sample problems (Fig. 8). It is important to
monitor the performance of the code as well. Figure 9 shows that for most of the runs
the run time has not changed much.
Before making changes to the code, it is important that you verify that you can run
the pc_auto-test successfully. Don’t do this when you have already modified the code,
because then you cannot be sure that any problems are caused by your changes, or
because it wouldn’t have worked anyway. Also, keep in mind that the code is public,
so your changes should make sense from a broader perspective and should not only
be intended for yourself. Regarding more general aspects about coding standards see
Sect. B.2.
In order to keep the development of the code going, it is important that the users are
able to understand and modify (program!) the code. In this section we explain first how

92

T HE P ENCIL C ODE

Figure 7: Number of lines of code and the number of subroutines since the end of 2001. The jump in the
Summer of 2005 was the moment when the developments on the side branch (eos branch) were merged
with the main trunk of the code. Note the approximately linear scaling with time.

to orient yourself in the code and to understand what is in it, and then to modify it
according to your needs.
The Pencil Code check-ins occur regularly all the time. By the Pencil Code User Meeting
2010 we have arrived at a revision number of 15,000. In February 2017, the number of
check-ins has risen to 26,804; see https://github.com/pencil-code/pencil-code. Major code changes are nowadays being discussed by the Pencil Code Steering Committee
(https://www.nordita.org/~brandenb/pencil-code/PCSC/). The increase of the revision
number with time is depicted in Figure 10. The number of Pencil Code developers increases too (Figure 11), but the really active ones are getting rare. This may indicate
that new users can produce new science with the code as it is, but it may also indicate
that it is getting harder to understand the code. How to understand the code will be
discussed in the next section.

Figure 8: Number of tests in the sample directory that are used in the nightly auto tests. Note again the
approximately linear scaling with time.

93

Figure 9: Run time of the daily auto-tests since August 17, 2008. For most of the runs the run time has
not changed much. The occasional spikes are the results of additional load on the machine.

94

T HE P ENCIL C ODE

Figure 10: Number of check-ins since 2002. Note again the linear increase with time, although in the last
part of the time series there is a notable speed-up.

Figure 11: Check-ins since 2002 per user. Users with more than 100 check-ins are color coded.

8. Understanding the code

95

8 Understanding the code
Understanding the code means looking through the code. This is not normally done by
just printing out the entire code, but by searching your way through the code in order to
address your questions. The general concept will be illustrated here with an example.

8.1

Example: how is the continuity equation being solved?

All the physics modules are solved in the routine pde, which is located in the file and
module ‘Equ’. Somewhere in the pde subroutine you find the line
call dlnrho_dt(f,df,p)
This means that here the part belonging to ∂ ln ρ/∂t is being assembled. Using the grep
command you will find that this routine is located in the module density, so look in
there and try to understand the pieces in this routine. We quickly arrive at the following
crucial part of code,
!
!
!

Continuity equation.
if (lcontinuity_gas) then
if (ldensity_nolog) then
df(l1:l2,m,n,irho)
= df(l1:l2,m,n,irho)
- p%ugrho - p%rho*p%divu
else
df(l1:l2,m,n,ilnrho) = df(l1:l2,m,n,ilnrho) - p%uglnrho - p%divu
endif
endif

where, depending on some logicals that tell you whether the continuity equation should
indeed be solved and whether we do want to solve for the logarithmic density and not
the actual density, the correct right hand side is being assembled. Note that all these
routines always only add to the existing df(l1:l2,m,n,ilnrho) array and never reset
it. Resetting df is only done by the timestepping routine. Next, the pieces p%uglnrho
and p%divu are being subtracted. These are pencils that are organized in the structure
with the name p. The meaning of their names is obvious: uglnrho refers to u · ∇ ln ρ
and divu refers to ∇ · u. In the subroutine pencil_criteria_density you find under
which conditions these pencils are requested. Using grep, you also find where they are
calculated. For example p%uglnrho is calculated in ‘density.f90’; see
call u_dot_grad(f,ilnrho,p%glnrho,p%uu,p%uglnrho,UPWIND=lupw_lnrho)
So this is a call to a subroutine that calculates the u·∇ operator, where there is the possibility of upwinding , but this is not the default. The piece divu is calculated in ‘hydro.f90’
in the line
!
!
!

Calculate uij and divu, if requested.
if (lpencil(i_uij)) call gij(f,iuu,p%uij,1)
if (lpencil(i_divu)) call div_mn(p%uij,p%divu,p%uu)

96

T HE P ENCIL C ODE

Note that the divergence calculation uses the velocity gradient matrix as input, so no
new derivatives are recalculated. Again, using grep, you will find that this calculation
and many other ones happen in the module and file ‘sub.f90’. The various derivatives
that enter here have been calculated using the gij routine, which calls the der routine,
e.g., like so
k1=k-1
do i=1,3
do j=1,3
if (nder==1) then
call der(f,k1+i,tmp,j)
For all further details you just have to follow the trail. So if you want to know how the
derivatives are calculated, you have to look in deriv.f90, and only here is it where the
indices of the f array are being addressed.
If you are interested in magnetic fields, you have to look in the file ‘magnetic.f90’. The
right hand side of the equation is assembled in the routine
!***********************************************************************
subroutine daa_dt(f,df,p)
!
! Magnetic field evolution.
!
! Calculate dA/dt=uxB+3/2 Omega_0 A_y x_dir -eta mu_0 J.
! For mean field calculations one can also add dA/dt=...+alpha*bb+delta*WXJ.
! Add jxb/rho to momentum equation.
! Add eta mu_0 j2/rho to entropy equation.
!
where the header tells you already a little bit of what comes below. It is also here where
ohmic heating effects and other possible effects on other equations are included, e.g.
!
!
!

Add Ohmic heat to entropy or temperature equation.
if (lentropy .and. lohmic_heat) then
df(l1:l2,m,n,iss) = df(l1:l2,m,n,iss) &
+ etatotal*mu0*p%j2*p%rho1*p%TT1
endif

We leave it at this and encourage the user to do similar inspection work on
a number of other examples. If you think you find an error, file a ticket at
http://code.google.com/p/pencil-code/issues/list. You can of course also repair it!

9. Adapting the code

97

9 Adapting the code
9.1

The P ENCIL C ODE coding standard

As with any code longer than a few lines the appearance and layout of the source code
is of the utmost importance. Well laid out code is more easy to read and understand and
as such is less prone to errors.
A consistent coding style has evolved in the P ENCIL C ODE and we ask that those contributing try to be consistent for everybody’s benefit. In particular, it would be appreciated if those committing changes of existing code via svn follow the given coding style.
There are not terribly many rules and using existing code as a template is usually the
easiest way to proceed. In short the most important rules are:
• tab characters do not occur anywhere in the code (in fact the use of tab character
is an extension to the Fortran standard).
• Code in any delimited block, e.g. if statements, do loops, subroutines etc., is indented be precisely 2 spaces. E.g.
if (lcylindrical) then
call fatal_error(’del2fjv’,’del2fjv not implemented’)
endif
• continuation lines (i.e. the continuation part of a logical line that is split using
the & sign) are indented by 4 spaces. E.g. (note the difference from the previous
example)
if (lcylindrical) &
call fatal_error(’del2fjv’,’del2fjv not implemented’)
[...]
• There is always one space separation between ’if’ and the criterion following in
parenthesis:
if (ldensity_nolog) then
rho=f(l1:l2,m,n,irho)
endif
This is wrong:
if(ldensity_nolog) then
rho=f(l1:l2,m,n,irho)
endif

! WRONG

• In general, try to follow common practice used elsewhere in the code. For example,
in the code fragment above there are no empty spaces within the mathematical
expressions programmed in the code. A unique convention helps in finding certain
expressions and patterns in the code. However, empty spaces are often used after
commas and semicolons, for examples in name lists.
• Relational operators are written with symbols (==, / =, <, <=, >, >=), not with
characters (.eq., .ne., .lt., .le., .gt., .ge.).
• In general all comments are placed on their own lines with the ’!’ appearing in the
first column.

98

T HE P ENCIL C ODE
• All subroutine/functions begin with a standard comment block describing what
they do, when and by whom they were created and when and by whom any nontrivial modifications were made.
• Lines longer that 78 characters should be explicitly wrapped using the & character,
unless there is a block of longer lines that can only be read easily when they are
not wrapped. Always add one whitespace before the & character.

These and other issues are discussed in more depth and with examples in Appendix B,
and in particular in Sect. B.2.

9.2

Adding new output diagnostics

With the implementation of new physics and the development of new procedures it will
become necessary to monitor new diagnostic quantities that have not yet been implemented in the code. In the following, we describe the steps necessary to set up a new
diagnostic variable.
This is nontrivial as, in order to keep latency effects low on multi-processor machines,
the code minimizes the number of global reduction operations by assembling all quantities that need the maximum taken in fmax , and those that need to be summed up
over all processors (mostly for calculating mean quantities) in fsum (see subroutine
diagnostic in file ‘src/equ.f90’).
As a sample variable, let us consider jbm (the volume average j · B ). Only the module magnetic will be affected, as you can see (the diagnostic quantity jbm is already
implemented) with
unix>

grep -i jbm src/*.f90

If we pretend for the sake of the exercise that no trace of jbm was in the code, and we
were only now adding it, we would need to do the following
1. add the variable idiag jbm to the module variables of Magnetic in both
‘magnetic.f90’ and ‘nomagnetic.f90’:
integer :: idiag_jbm=0
The variable idiag jbm is needed for matching the position of jbm with the list of
diagnostic variables specified in ‘print.in’.
2. in the subroutine daa_dt in ‘magnetic.f90’, declare and calculate the quantity jb
(the average of which will be jbm ), and call sum_mn_name
real, dimension (nx) :: jb ! jj · BB
[. . . ]
if (ldiagnos) then
! only calculate if diagnostics is required
if (idiag_jbm/=0) then
! anybody asked for jbm?
call dot_mn(jj,bb,jb)
! assuming jj and bb are known
call sum_mn_name(jb,i_jbm)
endif
endif

9.2 Adding new output diagnostics

99

3. in the subroutine rprint_magnetic in both ‘magnetic.f90’, add the following:
!
! reset everything in case of RELOAD
! (this needs to be consistent with what is defined above!)
!
if (lreset) then ! need to reset list of diagnostic variables?
[. . . ]
idiag_jbm=0
[. . . ]
endif
!
! check for those quantities that we want to evaluate online
!
do iname=1,nname
[. . . ]
call parse_name(iname,cname(iname),cform(iname),’jbm’,idiag_jbm)
[. . . ]
enddo
[. . . ]
!
! write column, i_XYZ, where our variable XYZ is stored
!
[. . . ]
write(3,*) ’i_jbm=’,idiag_jbm
[. . . ]

4. in the subroutine rprint_magnetic in ‘nomagnetic.f90’, add the following (newer
versions of the code may not require this any more):
!
! write column, i_jbm, where our variable jbm is stored
! idl needs this even if everything is zero
!
[. . . ]
write(3,*) ’i_jbm=’,idiag_jbm
[. . . ]

5. and don’t forget to add your new variable to ‘print.in’:
jbm(f10.5)

If, instead of a mean value, you want a new maximum quantity, you need to replace
sum_mn_name() by max_mn_name().
Sect. 5.8.1 describes how to output horizontal averages of the magnetic and velocity
fields. New such averages can be added to the code by using the existing averaging
procedures calc_bmz() or calc_jmz() as examples.

100

9.3

T HE P ENCIL C ODE

The f-array

The ‘f’ array is the largest array in the P ENCIL C ODE and its primary role is to store
the current state of the timestepped PDE variables. The f-array and its slightly smaller
counter part (the df-array; see below) are the only full size 3D arrays in the code. The
f-array is of type real but PDEs for a complex variable may be solved by using two
slots in the f-array. The actual size of the f-array is mx × my × mz × mfarray. Here,
mfarray = mvar + maux + mglobal + mscratch where mvar refers to the number of real PDE
variables.
As an example, we describe here how to put the time-integrated velocity, uut , into the
f-array (see ‘hydro.f90’). If this is to be invoked, there must be the following call somewhere in the code:
call farray_register_auxiliary(’uut’,iuut,vector=3)
Here, iuut is the index of the variable uut in the f-array. Of course, this requires that
maux is increased by 3, but in order to do this for a particular run only one must write
a corresponding entry in the ‘cparam.local’ file,
!
-*-f90-*(for Emacs)
! cparam.local
!
!** AUTOMATIC CPARAM.INC GENERATION ****************************************
! Declare (for generation of cparam.inc) the number of f array
! variables and auxiliary variables added by this module
!
! MAUX CONTRIBUTION 3
!
!***************************************************************************
! Local settings concerning grid size and number of CPUs.
! This file is included by cparam.f90
!
integer, parameter :: ncpus=1,nprocy=1,nprocz=ncpus/nprocy,nprocx=1
integer, parameter :: nxgrid=16,nygrid=nxgrid,nzgrid=nxgrid
This way such a change does not affect the memory usage for other applications where
this addition to ‘cparam.local’ is not made. In order to output this part of the f-array,
one must write lwrite_aux=T in the init_pars of ‘start.in’. (Technically, lwrite_aux=T
can also be invoked in run_pars of ‘run.in’, but this does not work at the moment.)

9.4

The df-array

The ‘df’ array is the second largest chunk of data in the P ENCIL C ODE. By using a 2N
storage scheme (see H.4) after Williamson [29] the code only needs one more storage
area for each timestepped variable on top of the current state stored in the f-array. As
such, and in contrast to the f-array, the df-array is of size mx × my × mz × mvar. Like the
df-array it is of type real. In fact the ghost zones of df are not required or calculated but
having f- and df-arrays of the same size make the coding more transparent. For mx, my
and mz large the wasted storage becomes negligible.

9.5

9.5

The fp-array

101

The fp-array

Similar to the ‘f’ array the code also has a ’fp’ array which contains current states of
all the particles. Like the f-array the fp-array also has a time derivative part, the dfparray. The dimension of the fp-array is mparl ocal × mpvar where mparl ocal is the number
of particles in the local processor (for serial runs this is the total number of particles)
and mpvar depends on the problem at hand. For example if we are solving for only
tracer particles then mpvar = 3, for dust particles mpvar = 6 The sequence in which the
slots in the fp-array are filled up depends on the sequence in which different particle
modules are called from the particles_main.f90. The following are the relevant lines
from particles_main.f90.
!***********************************************************************
subroutine particles_register_modules()
!
! Register particle modules.
!
! 07-jan-05/anders: coded
!
call register_particles
()
call register_particles_radius
()
call register_particles_spin
()
call register_particles_number
()
call register_particles_mass
()
call register_particles_selfgrav
()
call register_particles_nbody
()
call register_particles_viscosity
()
call register_pars_diagnos_state
()
!
endsubroutine particles_register_modules
!***********************************************************************
The subroutine resister_particles can mean either the tracer particles or dust particles. For the former the first three slots of the fp-array are the three spatial coordinates.
For the latter the first six slots of the fp-array are the three spatial coordinates followed
by the three velocity components. The seventh slot (or the fourth if we are use tracer
particles) is the radius of the particle which can also change as a function of time as
particles collide and fuse together to form bigger particles.
9.6

The pencil case

Variables that are derived from the basic physical variables of the code are stored in
one-dimensional pencils of length nx . All the pencils that are defined for a given set
of physics modules are in turn bundled up in a Fortran structure called p (or, more
illustrative, the pencil case ). Access to individual pencils happens through the variable
p%name, where name is the name of a pencil, e.g. rho that is a derived variable of the
logarithmic density lnrho.
The pencils provided by a given physics module are declared in the header of the file,
e.g. in the Density module:

102

T HE P ENCIL C ODE

! PENCILS PROVIDED lnrho; rho; rho1; glnrho(3); grho(3); uglnrho; ugrho
Notice that the pencil names are separated with a semi-colon and that vector pencils
are declared with “(3)” after the name. Before compiling the code, the script ‘mkcparam’
collects the names of all pencils that are provided by the chosen physics modules. It
then defines the structure p with slots for every single of these pencils. The definition of the pencil case p is written in the include file ‘cparam_pencils.inc’. When the
code is run, the actual pencils that are needed for the run are chosen based on the input parameters. This is done in the subroutines pencil_criteria_modulename that are
present in each physics module. They are all called once before entering the time loop.
In the pencil_criteria subroutines the logical arrays lpenc_requested, lpenc_diagnos,
lpenc_diagnos2d, and lpenc_video are set according to the pencils that are needed for
the given run. Some pencils depend on each other, e.g. uglnrho depends on uu and glnrho.
Such interdependencies are sorted out in the subroutines pencil_interdep_modulename
that are called after pencil_criteria_modulename.
In each time-step the values of the pencil logicals lpenc_requested, lpenc_diagnos,
lpenc_diagnos2d, and lpenc_video are combined to one single pencil array lpencil
which is different from time-step to time-step depending on e.g. whether diagnostics
or video output are done in that time-step. The pencils are then calculated in the subroutines calc_pencils_modulename. This is done before calculating the time evolution of
the physical variables, as this depends very often on derived variables in pencils.
The centralized pencil calculation scheme is a guarantee that
• All pencils are only calculated once
• Pencils are always calculated by the proper physics module
Since the P ENCIL C ODE is a multipurpose code that has many different physics modules, it can lead to big problems if a module tries to calculate a derived variable that
actually belongs to another module, because different input parameters can influence
how the derived variables are calculated. One example is that the Density module can
consider both logarithmic and non-logarithmic density, so if the Magnetic module calculates
rho = exp(f(l1:l2,m,n,ilnrho)
it is wrong if the Density module works with non-logarithmic density! The proper way
for the Magnetic module to get to know the density is to request the pencil rho in
pencil_criteria_magnetic.
9.6.1 Pencil check
To check that the correct pencils have been requested for a given run, one can run a
pencil consistency check in the beginning of a run by setting the logical lpencil_check
in &run_pars. The check is meant to see if
• All needed pencils have been requested
• All requested pencils are needed
The consistency check first calculates the value of df with all the requested pencils. Then
the pencil requests are flipped one at a time – requested to not requested, not requested
to requested. The following combination of events can occur:

9.7

Adding new physics: the Special module

103

• not requested → requested, df not changed
The pencil is not requested and is not needed.
• not requested → requested, df changed
The pencil is not requested, but is needed. The code stops.
• requested → not requested, df not changed
The pencil is requested, but is not needed. The code gives a warning.
• requested → not requested, df changed
The pencil is requested and is needed.

9.6.2 Adding new pencils
Adding a new pencil to the pencil case is trivial but requires a few steps.
• Declare the name of the pencil in the header of the proper physics module. Pencils
names must appear come in a ”;” separated list, with dimensions in parenthesis
after the name [(3) for vector, (3,3) for matrix, etc.].
• Set interdependency of the new pencil (i.e. what other pencils does it depend on) in
the subroutine pencil_interdep_modulename
• Make rule for calculating the pencil in calc_pencils_modulename
• Request the new pencil based on the input parameters in any relevant physics
module
Remember that the centralized pencilation scheme is partially there to force the users
of the code to think in general terms when implementing new physics. Any derived
variable can be useful for a number of different physics problems, and it is important
that a pencil is accessible in a transparent way to all modules.

9.7

Adding new physics: the Special module

If you want to add new physics to the code, you will in many cases want to add a new
Special module. Doing so is relatively straight forward and there is even a special directory for such additions.
To create your own special module, copy ‘nospecial.f90’ from the src/ directory to a
new name in the src/special/ directory. It is currently only possible to have one special
modules at a time and so several new bits of physics are often put in to one special
module. For this reasons a name should be chosen that relates to the problem to be
solved rather than the specific physics being implemented.
The first thing to do in your new module is to change the lspecial=.false. header to say
lspecial=.true.
The file is heavily commented though all such comments can be removed as you go. You
may implement any of the subroutines/function that exist in nospecial.f90 and those
routines must have the names and parameters as in nospecial.f90 . You do not however need to implement all routines, and you may either leave the dummy routines

104

T HE P ENCIL C ODE

copied from nospecial.f90 or delete them all together (provided the ”include ’special dummy.inc’” is kept intact at the end of the file. Beyond that, and data and subroutines
can be added to a special module as required, though only for use within that module.
There are routines in the special interface to allow you to add new equations, modify the
existing equation, add diagnostics, add slices, and many more things. If you feel there is
something missing extra hooks can easily be added - please contact the P ENCIL C ODE
team for assistance.
You are encouraged to submit/commit your special modules to the Pencil Code source.
When you have added new stuff to the code, don’t forget to mention this in the
‘pencil-code/doc/manual.tex’ file.

9.8

Adding switchable modules

In some cases where a piece of physics is thought to be more fundamental, useful in
many situations or simply more flexibility is required it may be necessary to add a new
module newphysics together with the corresponding nonewphysics module. The special
modules follow the same structure as the rest of the switchable modules and so using a
special module to prototype new ideas can make writing a new switchable module much
easier.
For an example of module involving a new variable (and PDE), the pscalar module is a
good prototype. The grep command
unix>

grep -i pscalar src/*

gives you a good overview of which files you need to edit or add.

9.9

Adding your initial conditions: the InitialCondition module

Although the code has many initial conditions implemented, we now discourage such
practice. We aim to eventually removed most of them. The recommended course of action
is to make use of the InitialCondition module.
InitialCondition works pretty much like the Special module. To implement your own
custom initial conditions, copy the file ‘noinitial_condition.f90’ from the src/ to src/
initial_condition, with a new, descriptive, name.
The first thing to do in your new module is to change the linitialcondition=.false. header
to say linitialcondition=.true. Also, don’t forget to add ../ in front of the file names in
include statements.
This file has hooks to implement a custom initial condition to most variables. After implementing your initial condition, add the line INITIAL_CONDITION=initial_condition/myinitialcondition to your ‘src/Makefile.local’
file. Here, myinitialcondition is the name you gave to your initial condition file. Add
also initial_condition_pars to the ‘start.in’ file, just below init_pars. This is a
namelist, which you can use to add whichever quantity your initial condition needs
defined, or passed. You must also un-comment the relevant lines in the subroutines
for reading and writing the namelists. For compiling reasons, these subroutines in
‘noinitial_condition.f90’ are dummies. The lines are easily identifiable in the code.

9.9 Adding your initial conditions: the InitialCondition module

105

Check e.g. the samples ‘2d-tests/baroclinic’, ‘2d-tests/spherical_viscous_ring’, or
‘interlocked-fluxrings’, for examples of how the module is used.

106

10

T HE P ENCIL C ODE

Testing the code

To maintain reproducibility despite sometimes quite rapid development, the P ENCIL
C ODE is tested nightly on various architectures. The front end for testing are the scripts
pc_auto-test and (possibly) pencil-test.
To see which samples would be tested, run
unix>

pc_auto-test -l

, to actually run the tests, use
unix>

pc_auto-test

unix>

pc_auto-test --clean

or

. The latter compiles every test sample from scratch and currently (September 2009)
takes about 2 hours on a mid-end Linux PC.
The pencil-test script is useful for cron jobs and allows the actual test to run on a
remote computer. See Sect. 10.1 below.
For a complete list of options, run pc_auto-test --help and/or pencil-test --help.

10.1

How to set up periodic tests

To set up a nightly test of the P ENCIL C ODE, carry out the following steps.
1. Identify a host for running the actual tests (the work host) and one to initiate
the tests and collect the results (the scheduling host). On the scheduling host, you
should be able to
(a) run cron jobs,
(b) ssh to the work host without password,
(c) publish HTML files (optional, but recommended),
(d) send e-mail (optional, but recommended).
Work host and scheduling host can be the same (in this case, use pencil-test’s ‘-l’
option, see below), but often they will be two different computers.
2. [Recommended, but optional:] On the work host, check out a separate copy of the
P ENCIL C ODE to reduce the risk that you start coding in the auto-test tree. In the
following, we will assume that you checked out the code as ‘~/pencil-auto-test’.
3. On the work host, make sure that the code finds the correct configuration file for
the tests you want to carry out. [Elaborate on that: PENCIL_HOME/local_config and
‘-f’ option; give explicit example]
Remember that you can set up a custom host ID file for your auto-test tree under
‘${PENCIL_HOME}/config-local/hosts/’.
4. On the scheduling host, use crontab -e to set up a cron job similar to the following:

10.1 How to set up periodic tests

107

30 02 * * * $HOME/pencil-auto-test/bin/pencil-test \
-D $HOME/pencil-auto-test \
--use-pc_auto-test \
-N15 -Uc -rs \
-T $HOME/public_html/pencil-code/tests/timings.txt \
-t 15m
-m  \
 \
-H > $HOME/public_html/pencil-code/tests/nightly-tests.html
Note 1: This has to be one long line. The backslash characters are written only
for formatting purposes for this manual you cannot use them in a crontab file.
Note 2: You will have to adapt some parameters listed here and may want to
modify a few more:
‘-D ’: Sets the directory (on the work host) to run in.
‘-T ’: If this option is given, append a timing statistics line for each test to
the given file.
‘--use-pc’: You want this option (and at some point, it will be the default).
‘-t 15m’: Limit the time for ‘start.x’ and ‘run.x’ to 15 minutes.
‘-N 15’: Run the tests at nice level 15 (may not have an effect for MPI tests).
‘-Uc’: Do svn update and pc_build --cleanall before compiling.
‘-m ’: If this option is given, send e-mails to everybody in the
(comma-separated) list of e-mail addresses if any test fails. As soon as this
option is set, the maintainers (as specified in the ‘README’ file) of failed tests
will also receive an e-mail.
‘work-host.inter.net|-l’: Replace this with the remote host that is to run the
tests. If you want to run locally, write -l instead.
‘-H’: Output HTML.
‘> $HOME/public_html/pencil-code/tests/nightly-tests.html’: Write output to
the given file.
If you want to run fewer or more tests, you can use the ‘-Wa,--max-level’ option:
-Wa,--max-level=3
will run all tests up to (and including) level 3. The default corresponds to
‘-Wa,--max-level=2’.
For a complete listing of pencil-test options, run
unix>

pencil-test --help

108

11
11.1

T HE P ENCIL C ODE

Useful internals
Global variables

The following variables are defined in ‘cdata.f90’ and are available in any routine that
uses the module Cdata .
Variable

Meaning
real

t

simulated time t.
integer

n[xyz]grid
nx , ny , nz
mx , my , mz

l1 , l2
m1 , m2
n1 , n2

m, n

global number of grid points (excluding ghost cells)
in x, y and z direction.
number of grid points (excluding ghost cells) as seen
by the current processor, i. e. ny=nygrid/nprocy, etc.
number of grid points seen by the current processor,
but including ghost cells. Thus, the total box for the
ivar th variable (on the given processor) is given by
f(1:mx,1:my,1:mz,ivar).
smallest and largest x-index for the physical domain
(i. e. excluding ghost cells) on the given processor.
smallest and largest y-index for physical domain.
smallest and largest z-index for physical domain,
i. e. the physical part of the ivar th variable is given
by f(l1:l2,m1:m2,n1:n2,ivar)
pencil indexing variables: During each time-substep
the box is traversed in x-pencils of length mx such
that the current pencil of the ivar th variable is
f(l1:l2,m,n,ivar).
logical

lroot
lfirst
headt

headtt
lfirstpoint

lout

11.2

true only for MPI root processor.
true only during first time-substep of each time step.
true only for very first full time step (comprising 3
substeps for the 3rd-order Runge–Kutta scheme) on
root processor.
= (lfirst .and. lroot): true only during very first
time-substep on root processor.
true only when the very first pencil for a given timesubstep is processed, i. e. for the first set of (m , n ),
which is probably (3, 3) .
true when diagnostic output is about to be written.

Subroutines and functions

output(file,a,nv) (module IO ): Write (in each ‘procN ’ directory) the content of the
global array a to a file called file, where a has dimensions mx ×my ×mz ×nv , or

11.2 Subroutines and functions

109

mx ×my ×mz if nv=1.
output_pencil(file,a,nv) (module IO ): Same as output(), but for a pencil variable,
i. e. an auxiliary variable that only ever exists on a pencil (e. g. the magnetic field
strength bb in ‘magnetic.f90’, or the squared sound speed cs2 in ‘entropy.f90’).
The file has the same structure as those written by output(), because the values
of a on the different pencils are accumulated in the file. This involves a quite nontrivial access pattern to the file and has thus been coded in C (‘src/debug_c.c’).
cross(a,b,c) (module Sub ): Calculate the cross product of two vectors a and b and
store in c. The vectors must either all be of size mx ×my ×mz ×3 (global arrays), or
of size nx ×3 (pencil arrays).
dot(a,b,c) (module Sub ): Calculate the dot product of two vectors a and b and store
in c. The vectors must either be of size mx ×my ×mz ×3 (a and b) and mx ×my ×mz
(c), or of size nx ×3 (a and b) and nx (c).
dot2(a,c) (module Sub ): Same as dot(a,a,c).

110

T HE P ENCIL C ODE

111

Part III

Appendix
APPENDIX
A

Date, Revision

Timings

In the following table we list the results of timings of the code on different machines.
Shown is (among other quantities) the wall clock time per mesh point (excluding the
ghost zones) and per full 3-stage time step, a quantity that is printed by the code at the
end of a run.16
As these results were assembled during the development phase of the code (that hasn’t
really finished yet,. . . ), you may not get the same numbers, but they should give some
orientation of what to expect for your specific application on your specific hardware.
The code will output the timing (in microseconds per grid point per time-step) at the
end of a run. You can also specify walltime in print.in to have the code continuously
output the physical time it took to reach the time-steps where diagnostics is done. The
time-dependent code speed can then be calculated by differentiating, e.g. in IDL with
IDL> pc_read_ts, obj=ts
IDL> plot, ts.it, 1/nw*deriv(ts.it,ts.walltime/1.0e-6), psym=2
where nw=nx*ny*nz.
proc
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
16

machine
Nl3
Nl3
Nq1
Ukaff
Nl6
Nl6
Nl6
Nl6
Nl6
Nl6
Nl6n
Mhd
Nq4
Nq5
fe1
Kabul
Hwwsx5
Mac/g95
Mac/ifc
Kabul
Nq3+4
Nq4+4
Nq4+5
Nq5+5

µs
pt step
19
30
10
9.2
6.8
36.3
42.7
37.6
19.6
8.7
9.8
7.8
14.4
6.7
5.1
4.4
3.4
7.7
4.5
2.5
7.4
8.9
7.3
3.7

resol.
643
643
643
643
643
64×128×64
162×256
162×256
162×256
162×256
323
643
1283
1283
1283
1283
2563
323
323
1283
1283
1283
1283
1283

what
kinematic
magn/noentro
magn/noentro
magn/noentro
magn/noentro
nomag/entro/dust
nomag/entro/rad6/ion
nomag/entro/rad2/ion
nomag/entro/ion
nomag/entro
magn/noentro/pscalar
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
convstar
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro

mem/proc
10 MB
20 MB

130 MB
7.8 GB

80 MB

when

who

20-may-02
20-may-02
30-may-02
20-may-02
10-mar-03
19-sep-03
22-oct-03
22-oct-03
22-oct-03
22-oct-03
17-mar-06
20-may-02
8-oct-02
8-oct-02
9-oct-02
20-jun-02
29-jan-03
14-jan-07
14-jan-07
20-jun-02
8-oct-02
8-oct-02
8-oct-02
8-oct-02

Note that when using ‘nompicomm.f90’, the timer currently used will overflow on some machines, so
you should not blindly trust the timings given by the code.

AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
WD
WD
BD
BD
WD
AB
AB
AB
AB

112

T HE P ENCIL C ODE

2
2
2
2
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
12
12
16
16
16
16
16
16
16
16
16
16
16

fe1
Nq2
Nq1+2
Hwwsx5
Nq1+2
Nq1235
Nq0-3
Mhd
fe1
Rasm.
Mhd
fe1
fe1
fe1
fe1
fe1
Luci
Lenn
Lenn
Kabul
Hwwsx5
Nqall
fe1
fe1
Ukaff
Kabul
fe1
fe1
fe1
Kabul
Gridur
Kabul
fe1
fe1
cetus
cetus
Neolith
Mhd
Hwwsx5
Neolith
Ferlin
Ferlin
Ferlin
nor52
hydra(2)
charybdis
scylla
scylla
janus
fe1
copson
fe1
fe1
workq
giga
giga2
giga
giga
Mhd
Mhd

3.45
9.3
8.3
1.8
5.4
4.1
6.8
2.76
3.39
2.02
8.2
6.35
2.09
1.45
7.55
5.48
1.77
0.65
1.21
1.5
1.8
3.0
3.15
2.36
1.24
1.25
1.68
1.50
1.44
0.83
1.46
0.87
0.99
0.98
0.58
0.73
0.82
1.46
0.50
0.444
0.450
0.269
0.245
2.00
0.317
0.169
0.150
0.151
6.02
1.77
0.596
0.94
0.75
0.88
0.76
0.39
0.47
0.43
2.03
0.64

1283
643
643
2563
643
1283
2563
643
323
643
2
64 ×16
64×128×64
1283
1283
2
16 ×512
162×512
643
643
643
1283
2563
1283
643
643
643
2
64 ×128
1283
1283
1283
1283
1283
2563
2563
2563
643
2563
643
1602×40
2563
1283
643
643
1283
323
723
723
723
723
2
72 × 22
643
1283
1283
1283
1283
1283
1283
1283
1283
1283
2563

magn/noentro
magn/noentro
magn/noentro
convstar
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
nomag/entro
nomag/entro/dust
magn/noentro
magn/noentro
nomag/entro/rad2/ion
nomag/entro/rad2/ion
magn/noentro
nomag/noentro
magn/noentro
magn/noentro
convstar
magn/noentro
magn/noentro
magn/noentro
magn/noentro
nomag/entro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
nomag/entro
convstar
magn/noentro
1test/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
coag/noentro
convstar
geodynamo/ks95
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
chiral
nomag/noentro
magn/noentro
magn/noentro

7.9 GB

294 MB

2x2

giga
4x1
1x4

47 MB
8.2 GB
1x8
2x4

1x8
2x4
4x2
28 MB
160 MB
2x4
4x2
4x2
4x2,156M
4x2
46 MB
8.6 GB

1x3x3
1x3x3
1x3x3
1x4x3

4x4
4x4/ifc6
4x4/ifc6
4x4/ifc6
4x4/ifc6
4x4/ifc6
4x4/ifc6
60 MB

9-oct-02
11-sep-02
11-sep-02
29-jan-03
11-sep-02
11-sep-02
10-jun-02
30-may-02
16-aug-02
8-sep-02
23-jul-02
19-sep-03
9-oct-02
9-oct-02
1-nov-03
1-nov-03
27-feb-07
13-jan-07
7-nov-06
20-jun-02
29-jan-03
8-oct-02
8-sep-02
8-sep-02
20-may-02
11-jul-02
8-sep-02
8-sep-02
8-sep-02
20-jun-02
19-aug-02
20-jun-02
8-sep-02
8-sep-02
19-aug-07
19-aug-07
5-dec-07
7-oct-02
29-jan-03
6-dec-07
21-jun-09
2-apr-10
2-feb-11
2-dec-09
8-may-16
8-may-16
8-may-16
8-may-16
17-dec-15
9-feb-03
21-nov-03
8-sep-02
9-may-03
21-aug-04
21-aug-04
20-aug-04
29-may-04
28-apr-03
26-nov-02
22-may-02

AB
AB
AB
WD
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
WD
WD
AB
AB
AB
AB
WD
AB
AB
AB
WD
NE
WD
AB
AB
SS
SS
AB
AB
WD
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
DM
AB
AB
AB
AB
AB
AB
AB
AB
AB

A. Timings

16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
16
24
24
32
32
32
32
32
32
32
32
32
32
32
32
32
32
32
36
36
48
64
64
64
64
64
64
64
64
64
64
64
64
64
64
64
64
64
64
64
72
72
72
72

fe1
fe1
fe1
Ukaff
Ukaff
Kabul
Kabul
Gridur
Gridur
Sander
Luci
Lenn
Neolith
Triolith
Triolith
Triolith
Coma
Gardar
Summit
giga?
Ukaff
Ukaff
Hermit
fe1
fe1
Luci
Lenn
Steno
Steno
Steno
Steno
Sanss
Neolith
Ferlin
Kraken
scylla
janus
Coma
fe1
giga
giga
fe1
giga
giga
giga
Gridur
Ukaff
Steno
Neolith
Ferlin
Ferlin
Akka
Triolith
Triolith
Hermit
Sisu
Kraken
Kraken
Kraken
Kraken

0.56
6.30
1.31
0.61
0.64
0.80
0.51
0.81
0.66
0.53
0.375
0.284
0.180
0.075
0.065
0.054
0.603
0.44
0.041
0.32
0.34
0.32
0.200
0.168
1.26
0.182
0.147
0.076
0.081
0.085
0.235
0.273
0.275
0.556
0.177
0.096
0.028
0.573
0.24
0.11
0.23
0.164
0.091
0.150
0.166
0.25
0.17
0.075
0.0695
8.51
0.156
0.038
0.0146
0.0164
0.101
0.00205
0.093
0.151
0.091
0.071

2563
128×256×128
1282×512
1283
2563
1283
2563
1283
2563
2563
1283
1283
2563
1283
1283
2563
1283
2
128 × 48
1443
2563
2563
5123
256×512×256
5123
2
64 ×256
2563
2563
2563
2563
2563
2
512 ×256
128×2562
1283
1283
192×384×64
723
722 ∗ 216
1283
2563
2563
2563
5123
5123
2563
5123
2563
5123
5123
2563
150×1282
2563
2
256 ×512
2563
2563
256×512×256
256×512×256
192×384×64
96×192×16
192×384×32
384×768×64

magn/noentro
nomag/entro/dust
nomag/entro/rad2/ion
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
GW/magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
nomag/noentro
nomag/entro/rad/ion
magn/noentro
nomag/entro/cool/fo
nomag/entro/cool/fo
nomag/entro/cool
nomag/entro/cool/sh
mag/entro
nomag
testfield4
testscalar
magn/noentro
magn/noentro
magn/noentro
GW/magn/noentro
magn/noentro
nomag/noentro
nomag/noentro/hyp
nomag/noentro/hyp
nomag/noentro/hyp
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
Li mechanism
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro

113

4x4
4x4

16 MB
9 MB

2x2x4
1x4x4
1x4x4
1x4x4

1x8x4

4x8
4x8
4x8
4x8
4x8
4x8

3x6x2
1x6x6
4x12
1x8x8
8x8
4x16
4x16
4x16
4x16
4x16
64*173MB

8x16
8x8
8x8
8x8
1x8x8
2x4x8
1x8x8
1x8x8
3x12x2
6x12
6x12
6x12

16-aug-02
19-sep-03
1-nov-03
22-may-02
20-may-02
20-jun-02
20-jun-02
19-aug-02
19-aug-02
8-sep-02
28-oct-06
8-nov-06
6-dec-07
1-mar-14
1-mar-14
1-mar-14
27-jul-17
6-nov-13
28-jul-17
13-sep-03
20-may-02
20-may-02
22-aug-13
9-oct-02
7-sep-03
26-feb-07
8-nov-06
20-jun-06
20-jun-06
20-jun-06
9-jul-06
3-jul-07
24-oct-08
7-jan-09
12-jan-12
8-may-16
28-mar-16
7-aug-17
2-sep-02
29-apr-03
8-dec-03
17-dec-03
17-dec-03
1-jul-03
10-jul-03
19-aug-02
21-may-02
19-oct-06
6-dec-07
21-jun-09
14-jun-09
27-dec-12
1-mar-14
1-mar-14
22-aug-13
22-aug-13
12-jan-12
17-jan-12
17-jan-12
17-jan-12

AB
AB
AB
AB
AB
WD
WD
NE
NE
AB
AB
AB
AB
AB
AB
AB
SM
AB
AB
AB
AB
AB
PJK
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
AB
WL
AB
AB
SM
AB
AB
AB
AB
AB
AB
AB
NE
AB
AB
AB
AB
AB
AB
AB
AB
PJK
PJK
WL
WL
WL
WL

114

72
128
128
128
128
128
64+64
128l
128
128
128
128
128
144
144
144
144
144
192
256
256
256
256
256
256
256
256
256
256
256
256
288
288
288
288
288
288
512
512
512
512
512
512
512
512
512
512
512
512
512
576
576
576
576
576
768
1024
1024
1024
1024

T HE P ENCIL C ODE

Summit
fe1
fe1
fe1
fe1
fe1
giga2
giga2
fe1
fe1
Hermit
Hermit
Sisu
Kraken
Kraken
Kraken
Gardar
Summit
Janus
Hermit
Hermit
giga2
Hermit
Hermit
Hermit
Akka
Sisu
Sisu
Triolith
Triolith
Beskow
Gardar
Kraken
Kraken
Kraken
Janus
Summit
Hermit
Hermit
Hermit
Hermit
Hermit
Hermit
Akka
Neolith
Gardar
Lindgren
Sisu
Sisu
Sisu
Kraken
Kraken
Kraken
Summit
Beskow
Lindgren
Hermit
Hermit
Hermit
Hermit

0.0128
0.44
2.8
0.51
0.27
0.108
0.0600
0.0605
0.35
0.094
0.0532
0.0493
0.00108
0.080
0.058
0.044
2.19
0.0064
0.0123
0.0328
0.0285
0.028
0.0262
0.0254
0.0226
0.0113
0.00618
0.00500
0.030
0.0049
3.36
0.042
0.0432
0.0447
0.0201
0.0360
0.0033
0.01717
0.0166
0.0142
0.01340
0.01189
0.01165
0.0081
0.0073
0.0035
0.0040
0.00446
0.00435
0.00268
0.0257
0.0317
0.0116
0.00183
0.00174
0.0049
0.00943
0.00707
0.00698
0.00630

5763
2563
5123
5123
5123
5123
5123
5123
5123
7863
256×512×256
256×512×256
256×512×256
96×192×32
192×384×64
384×768×128
288×1×288
5763
144×288×72
512×1024×512
256×512×256
10243
256×512×256
512×1024×512
512×1024×512
5123
256×512×256
512×1024×512
2562 × 512
2563
1×1×1024
5762×288
192×384×64
96×192×64
384×768×256
2883
5763
512×1024×512
256×512×256
256×512×256
512×1024×512
512×1024×512
512×1024×512
5123
2563
5123
2
512 ×1024
256×512×256
1024×2048×1024
512×1024×512
192×384×64
1922×64
7682×256
5763
5763
256×11522
512×1024×512
512×1024×512
1024×2048×1024
512×1024×512

magn/noentro
nomag/entro/rad8/ion
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
coag43
magn/noentro
magn/noentro/sph
spherical conv/magn
spherical conv/magn
magn/noentro
spherical conv/magn
spherical conv/magn
spherical conv/magn
magn/noentro
spherical conv/magn
spherical conv/magn
magn/rad
magn/noentro
coag43
magn/rad
magn/noentro
magn/noentro
magn/noentro
magn/entro/rad
magn/noentro
spherical conv/magn
spherical conv/magn
spherical conv/magn
spherical conv/magn
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro/sph
spherical conv/magn
spherical conv/magn
spherical conv/magn
spherical conv/magn

4x32
16x8
8x16
4x32
4x32/ifc6
4x32/ifc6
4x32/ifc6
2x64
4x32/ifc6
1x16x8
2x8x8
1x16x8
6x12x2
6x12x2
6x12x2
8x1x18
1x24x32
1x16x16
1x16x16
4x64/ifc6
2x16x8
2x16x8
4x8x8
16x16
1x16x16
1x16x16
1x16x16
1x16x16
1x1x256
1x18x16
6x12x4
6x12x4
6x12x4
1x16x18
1x16x18
1x32x16
1x32x16
2x16x16
2x16x16
8x8x8
4x16x8
16x32

16x32
4x16x8
1x32x16
6x24x4
122 x4
122 x4
1x24x48
1x24x48
1x24x32
1x32x32
2x32x16
4x16x16
4x16x16

7-aug-17
10-mar-04
5-sep-02
5-sep-02
5-sep-02
5-jan-02
21-aug-04
21-aug-04
9-sep-02
9-sep-02
22-aug-13
22-aug-13
22-aug-13
13-jan-12
17-jan-12
18-jan-12
13-sep-15
7-aug-17
24-jul-16
22-aug-13
22-aug-13
20-aug-04
22-aug-13
22-aug-13
22-aug-13
12-jun-11
22-aug-13
22-aug-13
17-mar-14
1-mar-14
3-mar-15
17-mar-14
12-jan-12
13-jan-12
18-jan-12
22-feb-16
7-aug-17
22-aug-13
22-aug-13
22-aug-13
22-aug-13
22-aug-13
22-aug-13
10-sep-11
20-nov-09
14-jan-13
8-jul-12
22-aug-13
22-aug-13
22-aug-13
12-jan-12
13-jan-12
18-jan-12
29-jul-17
23-may-16
17-oct-14
22-aug-13
22-aug-13
22-aug-13
22-aug-13

AB
TH
AB
AB
AB
AB
AB
AB
AB
AB
PJK
PJK
PJK
WL
WL
WL
AB
AB
AB
PJK
PJK
AB
PJK
PJK
PJK
AB
PJK
PJK
AB
AB
AB
AB
WL
WL
WL
AB
AB
PJK
PJK
PJK
PJK
PJK
PJK
AB
AB
AB
AB
PJK
PJK
PJK
WL
WL
WL
AB
AB
SJ
PJK
PJK
PJK
PJK

A. Timings

1024
1024
1024
1024
1024
1152
1152
1152
1152
1152
1152
1152
1152
1152
1152
1152
1536
2048
2048
2048
2048
2048
2048
2048
2048
2048
2304
2304
2304
2304
4096
4096
4096
4096
4096
4096
4608
4608
4608
4608
4608
4608
4608
6144
6144
8192
8192
8192
9216
9216
9216
9216
9216
9216
9216
16384
18432
18432
18432
36864

Triolith
Triolith
Triolith
Sisu
Sisu
Kraken
Kraken
Kraken
Lindgren
Lindgren
Beskow
Beskow
Beskow
Beskow
Beskow
Beskow
Lindgren
Hermit
Hermit
Hermit
Hermit
Lindgren
Lindgren
Triolith
Sisu
Sisu
Triolith
Kraken
Kraken
Kraken
Hermit
Triolith
Triolith
Triolith
Lindgren
Sisu
Triolith
Triolith
Triolith
Triolith
Triolith
Kraken
Kraken
Lindgren
Lindgren
Hermit
Sisu
Triolith
Kraken
Kraken
Kraken
Lindgren
Triolith
Triolith
Triolith
Hermit
Kraken
Kraken
Kraken
Kraken

0.00236
0.00126
0.00129
0.00225
0.00148
0.0212
0.00856
0.00549
0.016
0.0066
0.0055
0.0024
0.00098
0.00090
0.0060
0.0063
0.00171
0.00451
0.00380
0.00355
0.00350
0.00129
0.00129
9.3×10−4
0.00120
9.2×10−4
1.07×10−3
0.02267
0.01233
0.00300
0.00193
3.6×10−4
3.8×10−4
4.2×10−4
4.6×10−4
6.7×10−4
7.4×10−4
2.7×10−4
3.0×10−4
3.7×10−4
2.36×10−4
0.00764
0.00144
4.2×10−4
8.9×10−4
0.00101
4.1×10−4
1.48×10−4
0.00485
0.00158
8.0×10−4
2.36×10−4
1.04×10−3
1.28×10−4
1.30×10−4
6.4×10−4
0.00316
8.8×10−4
4.0×10−4
0.0020

2563
5123
5123
1024×2048×1024
512×1024×512
192×384×64
384×768×128
768×1536×256
5122×512
11523
11523
11523
11523
11523
11523
5763
2
512 ×384
1024×2048×1024
512×1024×512
512×1024×512
1024×2048×1024
5122×1024
10242×2048
5123
1024×2048×1024
512×1024×512
5763
192×384×64
192×768×64
768×3072×256
1024×2048×1024
10243
10243
10243
20483
1024×2048×1024
5763
11523
11523
11523
23043
192×768×128
768×3072×512
10243 × 1536
2562
1024×2048×1024
1024×2048×1024
20483
192×768×256
768×1536×256
1536×3072×512
23043
5763
23043
23043
1024×2048×1024
384×768×256
768×1536×512
1536×3072×1024
384×768×512

magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/rad
magn/noentro
magn/noentro/GW
magn/noentro
magn/noentro
magn/noentro
magn/noentro/GW
magn/entro/rad
magn/noentro
spherical conv/magn
spherical conv/magn
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro/sph
spherical conv/magn
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro
spherical conv/magn
magn/noentro
magn/noentro
magn/noentro
magn/noentro

115

4x16x16
2x16x32
4x16x16
2x32x16
12x24x4
12x24x4
12x24x4
1x36x32
1x32x36
1x32x36
1x32x36
1x32x36
1x32x36
1x32x36
1x32x36
2x32x24
2x32x32
8x16x16
4x32x16
4x32x16
32x64
32x64
4x16x32
4x32x16
4x18x32
12x24x8
12x48x4
12x48x4
4x32x32
4x32x32
8x16x32
4x16x64
4x16x64
8x18x32
4x32x36
4x36x32
4x18x64
2x32x72
12x48x8
12x48x8
4x16x64
2x48x64
8x32x32
4x32x64
24x48x8
24x48x8
24x48x8
4x48x48
16x18x32
4x36x64
4x32x72
16x32x32
24x48x16
24x48x16
24x48x16
482 x16

1-mar-14
1-mar-14
1-mar-14
22-aug-13
22-aug-13
13-jan-12
17-jan-12
17-jan-12
17-mar-14
25-nov-14
27-aug-17
20-jan-15
18-jan-16
30-mar-17
31-mar-18
17-feb-18
15-jul-13
22-aug-13
22-aug-13
22-aug-13
22-aug-13
20-apr-13
31-jul-12
1-mar-14
22-aug-13
22-aug-13
1-mar-14
13-jan-12
13-jan-12
18-jan-12
22-aug-13
1-mar-14
1-mar-14
1-mar-14
26-mar-13
22-aug-13
1-mar-14
1-mar-14
1-mar-14
1-mar-14
1-mar-14
13-jan-12
18-jan-12
21-oct-13
6-jan-15
22-aug-13
22-aug-13
1-mar-14
13-jan-12
17-jan-12
18-jan-12
15-feb-14
1-mar-14
1-mar-14
1-mar-14
22-aug-13
13-jan-12
17-jan-12
18-jan-12
14-jan-12

AB
AB
AB
PJK
PJK
WL
WL
WL
AB
AB
AB
AB
AB-gnu
AB
AB
AB
AB
PJK
PJK
PJK
PJK
AB
AB
AB
PJK
PJK
AB
WL
WL
WL
PJK
AB
AB
AB
AB
PJK
AB
AB
AB
AB
AB
WL
WL
AB
SJ
PJK
PJK
AB
WL
WL
WL
AB
AB
AB
AB
PJK
WL
WL
WL
WL

116

36864
36864
73728
73728
73728

T HE P ENCIL C ODE

Kraken
Kraken
Kraken
Kraken
Kraken

4.9×10−4
2.2×10−4
0.00121
2.9×10−4
1.2×10−4

15362×512
1536×3072×2048
7682×512
15362×1024
30722×2048

magn/noentro
magn/noentro
magn/noentro
magn/noentro
magn/noentro

482 x16
24x48x32
482 x32
482 x32
482 x32

17-jan-12
18-jan-12
19-jan-12
26-jan-12
26-jan-12

The machines we have used can be characterized as follows:
Nl3: 500 MHz Pentium III single CPU; RedHat Linux 6.2; 256 MB memory
Nq0: 931 MHz Pentium III single CPU; RedHat Linux 7.3; 0.5 GB memory
Nq[1-4]: 869 MHz Pentium III dual-CPU cluster; RedHat Linux 7.3; 0.77 GB memory
per (dual) node
Nq[5-6]: 1.2 GHz Athlon dual-CPU cluster; RedHat Linux 7.3; 1 GB memory per (dual)
node
Kabul: 1.9 GHz Athlon dual-CPU cluster; 1 GB memory per (dual) node; 256 kB cache
per CPU; Gigabit ethernet; SuSE Linux 8.0; LAM-MPI
Cincinnatus: 1.7 GHz Pentium 4 single CPU; 1 GB memory; 256 kB cache per CPU;
SuSE Linux 7.3
Horseshoe (fe1, giga, and giga2): consists of different subclusters. The old one
(queue name: workq, referred to as fe1) 2.0 GHz Pentium 512 single CPU; 25x 24port fast ethernet switches with gigabit ethernet uplink; 1 30-port gigabit ethernet
switch; 1 GB memory. The next generation has gigabit switches directly between
nodes, and 2.6 GHz processors. The third generation (giga2) has 3.2 GHz processors (most of which have 1 GB, some 2 GB), is organized in 2 blocks interconnected
with 2 Gb links, with 10 Gb uplinks within each block.
Ukaff: SGI Origin 3000; 400 MHz IP35 CPUs; IRIX 6.5; native MPI
Mhd: EV6 Compaq cluster with 4 CPUs per node; 4 GB memory per node (i. e. 1 GB per
CPU) OSF1 4.0; native MPI
Sander and Rasmussen: Origin 3000
Steno 118 node IBM cluster with dual node AMD Opteron processors with 10 Gb infiniband network, compiled with pgf90 -fastsse -tp k8-64e (Copenhagen).
Gridur: Origin 3000
Luci: (full name Lucidor) is an HP Itanium cluster, each of the 90 nodes has two 900
MHz Itanium 2 ”McKinley” processors and 6 GB of main memory. The interconnect
is myrinet.
Lenn: (full name Lenngren) is a Dell Xeon cluster with 442 nodes. Each node has two
3.4GHz “Nocona” Xeon processors and 8GB of main memory. A high performance
Infiniband network from Mellanox is used for MPI traffic.
Kraken: Cray Linux Environment (CLE) 3.1, with a peak performance of 1.17
PetaFLOP; the cluster has 112,896 cores, 147 TB of memory, in 9,408 nodes. Each
node has two 2.6 GHz six-core AMD Opteron processors (Istanbul), 12 cores, and
16 GB of memory. Connection via Cray SeaStar2+ router.

WL
WL
WL
WL
WL

A. Timings

117

Hermit: Cray XE6 with 7104 2.3 GHz AMD Interlagos 16 core processors (113,664
cores in total), nodes with either 1 or 2 GB of memory per core.
Sisu: Cray XC30 with 1472 2.6 GHz Intel (Xeon) Sandy Bridge 8 core (E5-2670) processors (11,776 cores in total), 2 GB of memory per core.
Beskow: Cray XC40 with 2.3 GHz Intel (Xeon) Haswell 16 core (E5-2698v3) processors
(67,456 cores in total), 2 GB of memory per core. Theoretical peak performance 2.43
pflops.
Table 8 shows a similar list, but for a few well-defined sample problems. The svn checkin patterns are displayed graphically in Fig 1.

Figure 12: Scaling results on three different machines. The thin straight line denotes perfectly linear
scaling.

Time per step per point [µs]

5123 gas + 64×106 particles
0.100

0.010
NGP
TSC
NGP+FFT
0.001
100

1000
No. of cores

10000

Figure 13: Scaling results of particle-mesh problem on Blue Gene/P on up to 4096 cores. The different
lines denote different particle-mesh schemes (NGP=Nearest Grid Point, TSC=Triangular Shaped Cloud)
and whether self-gravity is included (FFT).

118

T HE P ENCIL C ODE

µs/timestep/point

Strong scaling − Problem size 192 x 384 x 64

0.10

323

163
0.01
100

1000
proc #

Figure 14: Scaling results on Kraken at fixed problem size, for a magnetized disk model in cylindrical
coordinates. The black line shows ideal scaling from 32 cores. The blue line is the best second-order fit to
the data points. A load of 163 mesh points per processor marks the best strong scaling.

A.1

Test case

In the following test samples, we run isothermal magnetohydrodynamics in a periodic
domain17 . Power spectra are computed during the run, but our current parallelization
of the Fourier transform requires that the meshpoint number is an integer multiple of
the product of processor numbers in the y and z directions and the product of processor
numbers in the x and y directions. In addition, the number of processors in one direction
17

Run directories are available on http://norlx51.nordita.org/~brandenb/pencil-code/timings/bforced/

Weak Scaling
163 p=0.70
323 p=0.85
643 p=0.93

0.1000

µs/timestep/point

Wall time (hours)

0.10

0.0100

0.0010

0.01

0.0001
102

103
proc #

104

105

102

103
proc #

104

105

Figure 15: Scaling results on Kraken at fixed load per processor, for a magnetized disk model in cylindrical
coordinates. The figure shows, after determining that 163 is the best load per processor for strong scaling,
how far one can push with weak scaling. The scaling index is found to be 0.7 for 163 and 0.93 for 643 , up
to 73 728 processors.

A.2

Running the code

119

Table 8: Like previous table, but for the versions from the ‘samples’ directory.

proc(s)

machine

µs
pt step

resol.

mem./proc

when

who

93 MB

23-jul-02
23-jul-02
23-jul-02
23-jul-02
23-jul-02
23-jul-02
23-jul-02

wd
wd
wd
wd
wd
wd
wd

29 MB
18 MB
11 MB
9 MB

23-jul-02
23-jul-02
23-jul-02
23-jul-02

wd
wd
wd
wd

conv-slab
1
1
1
1
1
1
1

Mhd
Cincinnatus
Cincinnatus
Cincinnatus
Kabul
Kabul
Kabul

323
323
643
1283
323
643
1283

6.45
4.82
11.6
20.8
3.91
3.88
4.16

4 MB
3 MB
14 MB
93 MB

conv-slab-flat
1
2
4
8

Kabul
Kabul
Kabul
Kabul

3.02
1.81
1.03
0.87

1282×32
1282×32
1282×32
1282×32

should not be so large that the number of mesh points per processor becomes comparable
to or less than the number of ghost zones (which is 6).
A.2

Running the code

To run the code, get one of the sample run directories, e.g.,
http://norlx51.nordita.org/~brandenb/pencil-code/timings/bforced/512_4x16x32.
The relevant file to be changed is src/cparam.local
ncpus=2048,nprocx=4,nprocy=16,nprocz=ncpus/(nprocx*nprocy)
nxgrid=512,nygrid=nxgrid,nzgrid=nxgrid

in particular the values of ncpus, nprocx, nprocy, and nxgrid. Once they are chosen, say
make, and submit start run.csh.
A.3

Triolith

On Triolith, strong scaling tests have been performed for three mesh sizes. The time per
time step and mesh point is given for different processor numbers and layouts. Generally, it is advantageous to keep the number of processors in the x direction small.
Comments. Although on Triolith the number of processors per node is 16, resolutions
with one or two powers of 3 (such as 576) still work well. Furthermore, the number
of processors above which the scaling becomes poor increases quadratically with the
number of mesh points. This implies that the RAM per processor increases linearly with
the problem size per direction. However, this will not be a limitation, because even for
23043 meshes, the required RAM is still below 100 MB.
A.4

Lindgren

On Lindgren, we have performed weak scaling tests and compare with weak scaling
results for Triolith. Triolith is about twice as fast as Lindgren.

120

T HE P ENCIL C ODE

Figure 16: Strong scaling on Triolith (2014).

Figure 17: Comparison Triolith (black, plus signs) and Lindgren (red, triangles). Weak scaling (2014).

A.4

Lindgren

121

Table 9: Triolith timings

proc

µs
pt step

16
0.075
16
0.065
16
0.0544
64
0.0146
64
0.0164
256
0.0049
512
0.0035
1024
0.00236
1024
0.00127
1024
0.00129
2048 9.34×10−4
2304
0.00107
4096 3.6×10−4
4096 3.8×10−4
4096 4.2×10−4
4608 7.38×10−4
4608 2.66×10−4
4608 3.03×10−4
4608 3.12×10−4
4608 2.36×10−4
8192 1.475×10−4
9216
0.00104
9216 1.276×10−4
9216 1.30×10−4

resol.

layout

1283
1283
2563
2563
2563
2563
2563
2563
5123
5123
5123
5763
10243
10243
10243
5763
11523
11523
11523
23043
20483
5763
23043
23043

2x2x4
1x4x4
1x4x4
1x8x8
2x4x8
1x16x16
2x16x16
2x16x32
2x16x32
4x16x16
4x16x32
4x18x32
4x32x32
8x16x32
4x16x64
8x18x32
4x32x36
4x36x32
4x18x64
2x32x72
4x32x64
16x18x32
4x36x64
4x32x72

Table 10: Lindgren timings

proc

µs
pt step

1536 0.00171
2048 0.00129
2048 0.00129
4096 4.6×10−4
9216 2.36×10−4

resol.

layout

5122 ×384
5122 ×1024
10242 ×2048
20483
23043

2x32x24
1x32x64
1x32x64
4x16x64
4x48x48

122

B

T HE P ENCIL C ODE

Coding standard

The numerous elements that make up the P ENCIL C ODE are written in a consistent
style that has evolved since it was first created. Many people have contributed their
knowledge and experience with in this and the result is what we believe is and extremely
readable and manageable code.
As well as improving the readability of the code, by having some naming conventions for
example aids greatly in understanding what the code does.
There is a standard for all aspects of the code, be it Fortran source, shell scripts, Perl
scripts, LaTeX source, Makefiles, or otherwise. Where nothing has been explicitly stated
it is recommended that similar existing examples found in the code are used as a template.
B.1

File naming conventions

All files with the exception of the ‘Makefile’s are given lowercase filenames.
Fortran source files all have the ‘.f90’ extension. Files that contain ‘non-executable code’
i.e. declarations that are included into other files are given the extension ‘.h’ and those
that are generated dynamically at compile time have an ‘.inc’ extension.
Fortran source code defining a module is placed in files whose names begin with the
Fortran module name in all lowercase. Where there exist multiple implementations of
a specific module the filenames are extended using and with an underscore ad a brief
name relating to what they do.
Text files containing parameters to be read by the code at run time are placed in files
with the extension ‘.in’
B.2

Fortran Code

The code should remain fully compatible with the Fortran90 standard. This ensures that
the code will run on all platforms. Indeed, an important aspect of P ENCIL C ODE philosophy is to be maximally flexible. This also means that useful non-standard extensions
to the code should be hidden in and be made accessible through suitable non-default
modules.
Fortran is not case-sensitive but in almost all instances we prescribe some form of capitalization for readability.
In general all Fortran code including keywords, variable names etc. are written in lowercase. Some of the coding standard has already been discussed in Sect. 9.1. Here we
discuss and amplify some remaining matters.
B.2.1 Indenting and whitespace
Whitespace should be removed from the end of lines.
Blank lines are kept to a minimum, and when occurring in subroutines or functions are
replaced by a single ‘!’ in the first column.

B.2

Fortran Code

123

Tab characters are not used anywhere in the code. Tab characters are not in fact allowed
by the Fortran standard and compilers that accept them do so as an extension.
All lines are kept to be not more than 80 characters long. Where lines are longer they
must be explicitly wrapped using the Fortran continuation character ‘&’. Longer lines
(up to 132 characters) and additional spaces are allowed in cases where the readability
of the code is enhanced, e.g. when one line is followed by a similar one with minor
differences in some places.
Code in syntactic blocks such as if–endif, do–enddo, subroutine–endsubroutine etc. is
always indented by precisely two spaces. The exception to this is that nested loops where
only the innermost loop contains executable code should be written with the do–enddo
pairs at the same level of indentation,
do n=n1,n2
do m=m1,m2
[...]
enddo
enddo
Alternatively nested loops may be written on a single line, i.e.
do n=n1,n2; do m=m1,m2
[...]
enddo; enddo
B.2.2 Comments
Descriptive comments are written on their own lines unless there is a strong reason to do
otherwise. Comments are never indented and the ‘!’ should appear in the first column
followed by two spaces and then the text of the comment. Extremely short comments
may follow at the end of a line of code, provided there is space.
Comments also must not exceed the 78 character line length and should be wrapped
onto more lines as needed.
Typically comments should appear with a blank commented line above and below the
wrapped text of the comment.
All subroutine/functions begin with a standard comment block describing what they do,
when and by whom they were created and when and by whom any non-trivial modifications were made.
Comments should be written in sentences using the usual capitalization and punctuation of English, similar to how text is formatted in an e-mail or a journal article.
For example:
some fortran code
some more fortran code
!
!
!
!

A descriptive comment explaining what the following few lines
of code do.
the fortran code being described

124

T HE P ENCIL C ODE

the fortran code being described
...
!
!
!

A final detail described here.
the final fortran code
the final fortran code
...

Subroutines and functions are started with a comment block describing what they do,
when and by whom they were created and when and by whom any non-trivial modifications were made. The layout of this comment block is a standard, for example:
!***********************************************************************
subroutine initialize_density(f,lstarting)
!
! Perform any post-parameter-read initialization i.e. calculate derived
! parameters.
!
! For compatibility with other applications, we keep the possibility
! of giving diffrho units of dxmin*cs0, but cs0 is not well defined general.
!
! 24-nov-02/tony: coded
!
1-aug-03/axel: normally, diffrho should be given in absolute units
!
where dates are written in dd-mmm-yy format as shown and names appearing after the
‘/’ are either the users cvs login name or, where such exists amongst the P ENCIL C ODE
community, the accepted short form (≈ 4 characters) of the authors name.
B.2.3 Module names
The names of modules are written with initial letter capitalization of each word and the
multiple words written consecutively without any separator.
B.2.4 Variable names
Variable are given short but meaningful names and written in all lowercase. Single
character names are avoided except for commonly used loop indices and the two code
data structures of the P ENCIL C ODE: ‘f’ the main state array (see 9.3) and ‘p’ the pencil
case structure (see 9.6).
Quantities commonly represented by a particular single character in mathematics are
typically given names formed by repeating the character (usually in lowercase), e.g. the
velocity u becomes ‘uu’, specific entropy s becomes ‘ss’ etc.
Temperature in variable names is denoted with a capital T so as not to be confused
with time as represented by a lowercase t. Note however the since Fortran is not case
sensitive the variables for example ‘TT’ and ‘tt’ are the same so distinct names must
be used. For this reason time is usually represented by a single t contrary to the above
guideline.

B.2

Fortran Code

125

The natural log of a quantity is represented by using adding ‘ln’ to its name, for example
log of temperature would be ‘lnTT’.
There are some standard prefixes used to help identify the type and nature of variables
they are as follows:
• i – Denotes integer variables typically used as array indices.
• i – Denotes pencil case array indices.
• idiag – Denotes diagnostic indices.
• l – Denotes logical/boolean flags
• cdt – Denotes timestep constraint parameters.
• unit – Denotes conversion code/physics unit conversion parameters.
B.2.5 Emacs settings
Here are some settings from wd’s ‘~/.emacs’ file:
;;;
;;;
;;;
;;;
;;;
;;;
;;;
;;;

~/.f90.emacs
Set up indentation and similar things for coding the {\sc Pencil Code}.
Most of this can probably be set through Emacs’ Customize interface
as well.
To automatically load this file, put the lines
(if (file-readable-p "~/.f90.emacs")
(load-file "~/.f90.emacs"))
into your ~/.emacs file.

;; F90-mode indentation widths
(setq f90-beginning-ampersand nil) ; no 2nd ampersand at continuation line
(setq f90-do-indent
2)
(setq f90-if-indent
2)
(setq f90-type-indent
2)
(setq f90-continuation-indent 4)
;; Don’t use any tabs for indentation (with TAB key).
;; This is actually already set for F90-mode.
(setq-default indent-tabs-mode nil)
;; Ensure Emacs uses F90-mode (and not Fortran-mode) for F90 files:
(setq auto-mode-alist
(append
’(
("\\.[fF]90$"
. f90-mode)
("\\.inc$"
. f90-mode)
)
auto-mode-alist))
;; Make M-Backspace behave in Xemacs as it does in GNU Emacs. The default
;; behavior is apparently a long-known bug the fix for which wasn’t

126

T HE P ENCIL C ODE

;; propagated from fortran.el to f90.el.
;; (http://list-archive.xemacs.org/xemacs-patches/200109/msg00026.html):
(add-hook ’f90-mode-hook
(function (lambda ()
(define-key f90-mode-map [(meta backspace)] ’backward-kill-word)
)))
B.3

Other best practices

When implementing IF or SELECT blocks always write code for all cases – including
the default or else case. This should be done even when that code is only a call to raise
an error that the case should not have been reached. If you see a missing case anywhere
then do add it. These failsafes are essential in a large multi-purpose multi-user code
like the P ENCIL C ODE.
If a case is supposed to do nothing and it may be unclear that the coder has recognized
this fact then make it explicit by adding the default case with a comment like
! Do Nothing. The compiler will clean away any such empty blocks.
B.4

General changes to the code

It is sometimes necessary to do major changes to the code. Since this may affect many people and may even be controversial among the developers, such
changes are restricted to the time of the next Pencil Code User Meeting. Such
meetings are advertised on http://www.nordita.org/software/pencil-code/ under
the news section. Notes about previous such meetings can be found under
http://www.nordita.org/software/pencil-code/UserMeetings/.
Major changes can affect those developers who have not checked in their latest changes
for some time. Before doing such changes it is therefore useful to contact the people who
have contributed to the latest developments on that module. If it is not functional or otherwise in bad shape, it should be moved to ‘experimental’, i.e. one says svn mv file.f90
experimental/file.f90. However, any such directory change constitutes a major change
in itself and should be performed in agreement with those involved in the development.
Otherwise any file that has been changed in the mean time will end up being outside
revision control, which is to be avoided at all cost.

C. Some specific initial conditions

C
C.1

127

Some specific initial conditions
Random velocity or magnetic fields

Obtained with inituu=’gaussian-noise’ (or initaa=’gaussian-noise’). The vector u (or
A) is set to normally distributed, uncorrelated random numbers in all meshpoints for
all three components. The power spectrum of u (A) increases then quadratically with
wavenumber k (without cutoff) and the power spectrum of ω (or B) increases like k 4 .
Note that a random initial condition contains significant power at the Nyquist frequency
(kNy = π/N , where N is the number of mesh points). In a decay calculation, because of
the discretization error, such power decays slower than it ought to; see Fig. 18, where
we show the evolution for a random initial velocity field for 643 meshpoints, ν = 5 × 10−2
(fairly large!), and nfilter=30.
It is clearly a good idea to filter the initial condition to prevent excess power at kNy . On
the other hand, such excess power is weak by comparison with the power at the energy
carrying scale, so one does not see it in visualizations in real space. Furthermore, as seen
from Fig. 18, for k < kNy /2 the power spectra for filtered and unfiltered initial conditions
is almost the same.

Figure 18: Velocity power spectra at three different times with and without filtering of the initial condition.

C.2

Turbulent initial with given spectrum

The most general procedure for producing an initial condition with a turbulent spectrum is inituu=’power_randomphase_hel’, which allows one to set two different slopes,
together with an exponential cutoff as well as a Gaussian peak within the spectrum. By
default, the field is solenoidal unless one puts lskip_projection=.true. and can have
fractional helicity by setting relhel_uu to a value between −1 and 1. By default it is 0,
which means it is nonhelical.

128

T HE P ENCIL C ODE

The spectral indices initpower and initpower2 refer to energy spectral indices. By default, initpower2=-5/3 , corresponding to a Kolmogorov spectrum. For a delta-correlated
spectrum, we have to put initpower=2 , corresponding to a k 2 energy spectrum for kinetic energy. This would be suitable for the subinertial range from k = 1 to k = kp
(corresponding to the variable kpeak ).
If cutoff=0 , no cutoff will be imposed. Otherwise, the spectrum will be multiplied by an
exponential function with exp(−k 2n ), where n = ’ncutoff=1’ by default.
Example, for ampluu=1e-1 , initpower=4., and kpeak=3., we get urms=3.981E-01 when
relhel uu=1 and urms=3.981E-01 when relhel uu=0 and urms=5.560E-01 when relhel uu=1 . The urms values scale linearly with ampluu and for initpower=2 also approximately linearly with kpeak . For the magnetic field, we initialize the magnetic vector
potential, so to get a k 4 spectrum, we have to put initpower aa=2. Everything else is
analogous; see, e.g.,
&hydro_init_pars
inituu=’power_randomphase_hel’, ampluu=1e-1, initpower=4., kpeak=3.
relhel_uu=0., cutoff=30.
/
&magnetic_init_pars
initaa=’power_randomphase_hel’, amplaa=1e-1, initpower_aa=2., kpeak_aa=3.
relhel_aa=0., cutoff_aa=30.
/
for which we get urms=3.981E-01 and brms=3.871E-01 .
C.3

Beltrami fields

Obtained with inituu=’Beltrami-z’ or initaa=’Beltrami-z’.
A = (cos z, sin z, 0),
C.4

or

u = (cos z, sin z, 0)

(116)

Magnetic flux rings: initaa=’fluxrings’

This initial condition sets up two interlocked thin magnetic tori (i. e. thin, torus-shaped
magnetic flux tubes). One torus of radius R lying in the plane z = 0 can be described in
cylindrical coordinates, (r, ϕ, z), by the vector potential


0
 ,
0
(117)
A = Φm 
−θ(r−R)δ(z)
resulting in a magnetic field




0
B = Φm δ(r−R)δ(z) .
0

(118)

Here Φm is the magnetic flux through the tube, θ(x) denotes the Heaviside function, and
δ(x) = θ′ (x)

(119)

C.5

Vertical stratification

129

is Dirac’s delta function.
Any smoothed versions of θ(x) and δ(x) will do, as long as the consistency condition (119)
is satisfied. E. g. the pairs


2
x
1
1
− x2
1 + erf √
(120)
e 2ε , θε (x) =
δε (x) = √
2
2ε
2πε2
or
1
x
1
1
δε (x) =
1 + tanh
(121)
, θε (x) =
2ε cosh2 xε
2
ε
are quite popular. Another possibility is a constant or box-like profile with

1
1
θ(|x| − ε) , θε (x) = {1 + max[−1, min(x/ε), 1]}
(122)
2ε
2
Note, however, that the Gaussian profile (120) is the only one that yields a radially
symmetric (withp
respect to the distance from the central line of the torus) magnetic field
profile Bϕ = Bφ ( (r−R)2 +z 2 ) if ε is sufficiently small.
δε (x) =

In Cartesian coordinates, the vector potential (117) takes the form


0


0
A = Φm  p
 .

2
2
x +y −R δ(z)
−θ
C.5

(123)

Vertical stratification

Gravity, g = −∇Φ, is specified in terms of a potential Φ. In slab geometry, Φ = Φ(z), we
have g = (0, 0, gz ) and gz = −dΦ/dz.
Use grav profile =’const’ together with gravz = −1 to get
Φ = (z − z∞ )(−gz ),

(−gz ) > 0.

(124)

gz = −νg2 z

(125)

Use grav profile =’linear’ to get
2
)νg2 ,
Φ = 12 (z 2 − z∞

where νg is the vertical epicyclic frequency. For a Keplerian accretion disc, νg = Ω. For
galactic discs, νg = 0.5Ω is representative of the solar neighborhood.
The value of z∞ is determined such that ρ = ρ0 and c2s = c2s0 at z = zref . This depends on
the values of γ and the polytropic index m (see below).
C.5.1 Isothermal atmosphere
Here we want cs = cs0 = const. Using initlnrho=’isothermal’ means
ρ
Φ
= −γ 2 .
ρ0
cs0

(126)

Φ
s
= (γ−1) 2 .
cp
cs0

(127)

ln
The entropy is then initialized to

In order that ρ = ρ0 and c2s = c2s0 at z = zref , we have to choose z∞ = zref .

130

T HE P ENCIL C ODE

C.5.2 Polytropic atmosphere
For a polytropic equation of state, p = KρΓ , where generally Γ 6= γ, we can write


1
ΓK Γ−1
≡ −∇h̃,
ρ
− ∇h + T ∇s = − ∇p = −∇
ρ
Γ−1
where we have introduced a pseudo enthalpy h̃ as
 


ΓK Γ−1
1
1
h̃ =
1−
h.
ρ
=
1−
Γ−1
γ
Γ

(128)

(129)

Obviously, for Γ = γ, the pseudo enthalpy h̃ is identical to h itself. Instead of specifying
Γ, one usually defines the polytropic index m = 1/(Γ−1). Thus, Γ = 1 + 1/m, and


1
h
(130)
h̃ = (m+1) 1 −
γ
This is consistent with a fixed entropy dependence, where s only depends on ρ like


s
ρ
Γ
− 1 ln
=
,
(131)
cp
γ
ρ0
and implies that
ln

c2s
ρ
.
= (Γ−1) ln
2
cs0
ρ0

(132)

For hydrostatic equilibrium we require h̃ + Φ = h̃0 = const. For gravity potentials that
vanish at infinity, we can have h̃0 6= 0, i.e. a finite pseudo enthalpy at infinity. For gz = −1
or gz = −z, this is not the case, so we put h̃0 = 0, and therefore h̃ = −Φ. Using c2s = (γ−1)h
together with (130) we find
γ
Φ.
(133)
c2s = −
m+1
In order that ρ = ρ0 and c2s = c2s0 at z = zref , we have to choose (remember that gz is
normally negative!)
z∞ = zref + (m+1)
and

c2s0
γ(−gz )
c2s0
γνg2
2

2
2
z∞
= zref
+ (m+1) 1

for grav profile =’const’,

(134)

for grav profile =’linear’.

(135)

Thus, when using initlnrho=’polytropic_simple’ we calculate


γΦ
c2s
ln 2 = ln −
cs0
(m+1)c2s0

(136)

and so the stratification is given by
ρ
c2
ln
= m ln 2s ,
ρ0
cs0

s
=
cp




c2
Γ
− 1 m ln 2s .
γ
cs0

(137)

C.5

Vertical stratification

131

C.5.3 Changing the stratification
Natural: measure length in units of c2s0 /gz . Can increase stratification by moving ztop
close to z∞ or, better still, keeping ztop = 0 and moving zbot → −∞. Disadvantage: in the
limit of weak stratification, the box size will be very small (in nondimensional units).
Box units: measure length in units of d. Can increase stratification by increasing gz to
gmax , which can be obtained by putting ztop = z∞ in (134), so
gmax =

m+1
c2s0
.
γ ztop − zref

(138)

For m = 1, γ = 5/3, ztop = 1, and zref = 0, for example, we have gmax = 6/5 = 1.2.
√
Gravitational box units: measure speed in units of gz d. The limit of vanishing stratification corresponds to cs0 → ∞. This seems optimal if we want to approach the Boussinesq case.
In Hurlburt et al. (1984), z increased downward and the atmosphere always terminated
at z = 0. In order to reproduce their case most directly, we put z∞ = 0 and consider only
negative values of z. To reproduce their case with a density stratification of 1:1.5, we
place the top of the model at z = −2 and the bottom at z = −3. In addition, the reference
height, zref , is chosen to be at the top of the box, i.e. zref = −2. From Eq. (134) we have
c2s0 = γ(−gz )(−zref )/(m + 1). Using (−gz ) = 1 and m = 1 we find c2s0 = γ, so cs0 = 1.291 (for
γ = 5/3). Values for other combinations are listed in Table 11.
Table 11: Correspondence between density contrast, top and bottom values of z, and cs0 for (−gz ) = 1,
m = 1, and γ = 5/3.

ρbot /ρtop

zbot

ztop

cs0

1.5
3
6
11
21

3
1.5
1.2
1.1
1.05

2
0.5
0.2
0.1
0.05

1.291
0.645
0.408
0.289
0.204

C.5.4 The Rayleigh number
In Ref. [9] the Rayleigh number is defined as


gd4
ds/cp
Ra =
−
,
νχ
dz
hydrostat

(139)

where the (negative) entropy gradient was evaluated in the middle of the box for the
associated hydrostatic reference solution, and χ = K/(ρcp ) and either ν = ν (if ν was
assumed constant) or ν = µ/ρ (if µ was assumed constant). Note that ρ is the average
mass in the box per volume, which is conserved. For a polytrope we have





ds/cp
1
1
−
,
(140)
= 1 − (m + 1) 1 −
dz
γ
z∞ − zm
hydrostat
where zm = (z1 + z2 )/2. This factor was also present in the definition of Hurlburt et al.
[20], but their definition differs slightly from Eq. (139), because they normalized the

132

T HE P ENCIL C ODE

density not with respect to the average value (which is constant for all times), but with
respect to the value at the top of the initial hydrostatic solution. Since the Rayleigh
number is proportional to ρ2 , their definition included the extra factor [(z∞ − zm )/d]2 .
Therefore
2m 
2

ρtop
z∞ − zm
Ra
(141)
RaHTM =
d
ρ
In the first model of Hurlburt et al. (1984), the Rayleigh number, RaHTM , was chosen to
be 310 times supercritical, and the critical Rayleigh number was around 400, so RaHTM =
1.25 × 105 . In their model the density contrast was 1:1.5 and m = 1. This turns out to
correspond to Ra = 4.9 × 104 , Fbot = 0.0025, and K = 0.002.

Another model that was considered by Hurlburt & Toomre (1988) had RaHTM = 105 ,
a density contrast of 11, and had a vertical imposed magnetic field (Chandrasekhar
number Q = 72). This corresponds to Ra = 3.6 × 108 , K = 0.0011, Fbot = 0.0014.
C.5.5 Entropy boundary condition
This discussion only applies to the case of convection in a slab. A commonly used lower
boundary condition is to prescribe the radiative flux at the bottom, i.e. Fbot = −KdT /dz.
Assuming that the density in the ghost zones has already been updated, we can calculate
the entropy gradient from


K c2s
d ln ρ
ds/cp
Fbot = −
(γ − 1)
,
(142)
+γ
cp γ − 1
dz
dz
which gives
γ−1
ds/cp
=−
dz
γ



Fbot d ln ρ
cp 2 +
Kcs
dz

(143)

for the derivative of the entropy at the bottom. This is implemented as the ‘c1’ boundary
condition at the bottom.
C.5.6 Temperature boundary condition at the top
In earlier papers the temperature at the top was set in terms of the quantity ξ0 , which
is the ratio of the pressure scale height relative to the depth of the unstable layer. Expressed in terms of the sound speed at the top we have
c2s,top = γξ0 gd.
c2s,bot

C.6


= ξ0 +

1
m+1

(144)


γgd.

(145)

Potential-field boundary condition

The ‘pot’ [or currently rather the ‘pwd’] boundary condition for the magnetic vector potential implements a potential-field boundary condition in z for the case of an x-y-periodic
box. In this section, we discuss the relevant formulas and their implementation in the
P ENCIL C ODE.

C.6

Potential-field boundary condition

133

Table 12: Correspondence between ξ0 and c2s,bot in single layer polytropes.

ξ0

c2s,bot

10.00 17.500
0.20
1.167
0.10
1.000
0.05
0.917
0.02
0.867
If the top boundary is at z = 0, the relevant potential field for z > 0 is given by
Ãx (kx , ky , z) = Cx (kxy ) e−κz ,
Ãy (kx , ky , z) = Cy (kxy ) e−κz ,
Ãx (kx , ky , z) = Cz (kxy ) e−κz ,
where
Ãi (kx , ky , z) ≡

Z

e−ikxy ·x Ai (x, y, z) dx dy

(146)
(147)
(148)

(149)

is the horizontal Fourier transform with kxy ≡ (kx , ky , 0), and k ≡ |kxy |. Note that this implies a certain gauge and generally speaking the z dependence in Eq. (148) is completely
arbitrary, but the form used here works well in terms of numerical stability.
At the very boundary, the potential field (146)–(148) implies
∂ Ã
+ κà = 0 ,
∂z

(150)

and, due to natural continuity requirements on the vector potential, these conditions
also hold for the interior field at the boundary.
Robin boundary conditions and ghost points To implement a homogeneous Robin
boundary condition, i. e. a condition of the form
df
+ κf = 0
dz

(151)

d
(f eκz ) = 0
dz

(152)

using ghost points, we first write it as

and implement this as symmetry condition for the variable φ(z) ≡ f (z) eκz :
φN −j = φN +j ,

j = 1, 2, 3

(153)

(where zN is the position of the top boundary and zN +1 , . . . are the boundary points). In
terms of f , this becomes
fN +j = fN −j e−κ(zN +j −zN −j ) .
(154)
Note that although the exponential term in Eq. (154) looks very much like the exterior
potential field (146)–(148), our ghost-zone values do not represent the exterior field –
they are rather made-up values that allow us to implement a local boundary condition
at z = 0.

134

C.7

T HE P ENCIL C ODE

Planet solution in the shearing box

In order to test the setup for accretion discs and the sliding periodic shearing sheet
boundary condition, a useful initial condition is the so-called planet solution of Goodman, Narayan, & Goldreich [17].
Assume s = 0 (isentropy), so the equations in 2-D are
ux ux,x + (u(0)
y + uy )ux,y = 2Ωuy − h,x

(155)

ux uy,x + (u(0)
y + uy )uy,y = −(2 − q)Ωux − h,y

(156)

(0)

where uy = −qΩx. Express u in terms of a stream function, so u = ∇ × (ψẑ), or
ux = ψ,y ,

uy = −ψ,x .

(157)

Ansatz for enthalpy
h = 21 δ 2 Ω 2 (R2 − x2 − ǫ2 y 2 − z 2 /δ 2 )

This implies

ψ = − 12 σΩ(R2 − x2 − ǫ2 y 2 ) − 12 qΩx2
ux = σΩǫ2 y,

(158)
(159)

uy = (q − σ)Ωx

(160)

(−q + q − σ)σǫ2 = 2(q − σ) + δ 2

(161)

σ(q − σ) = −(2 − q)σ + δ 2

(162)

− σ 2 ǫ2 = 2(q − σ) + δ 2

(163)

− σ 2 = −2σ + δ 2

(164)

δ 2 = (2 − σ)σ

(165)

σ 2 = 2q/(1 − ǫ2 )

(166)

and ux,x = uy,y = 0. Inserting into Eqs (155) and (156) yields

where we have already canceled common Ω 2 factors in both equations and common ǫ2
factors in the last equation. Simplifying both equations yields

The second equation yields
and subtracting the two yields

Table 13: Dependence of ǫ and δ on ǫ.

ǫ
0.1
0.2
0.3
0.4
0.48
0.5

σ

δ

1.74
1.77
1.82
1.89
1.97
2

0.67
0.64
0.58
0.46
0.22
0

D. Some specific boundary conditions

D

135

Some specific boundary conditions

In this section, we formulate and discuss the implementation of some common boundary
conditions in spherical and cylindrical coordinates.
D.1

Perfect-conductor boundary condition

This is a popular boundary condition for the magnetic field; it implies that
Bn = 0

(167)

Et = 0

(168)

and
on the boundary, where the subscript n denotes the normal component, and E t denotes
the tangential components of the electric field.
In Cartesian geometry, these conditions can be implemented by setting the two tangential components of the vector potential A to zero on the boundary. It is easy to see that
this also works in arbitrary curvilinear coordinates.
In particular, for spherical coordinates on a radial boundary we must have
r sin θ Br = ∂θ (sin θ Aφ ) − ∂φ Aθ = 0 .

(169)

This can be achieved by setting
(170)

Aφ = Aθ = 0

everywhere on the boundary. Note that this does not impose any condition on the radial
component of the vector potential.
Next, in spherical coordinates on a boundary with constant θ, we must have
Bθ =

1
1
∂φ Ar − ∂r (rAφ ) = 0 .
r sin θ
r

(171)

Again this can be achieved by Ar = Aφ = 0.
D.2

Stress-free boundary condition

On an impenetrable, stress-free boundary, we have
(172)

un = 0 ,

and the shear stress components Snt must vanish for any tangential direction t. At the
radial boundary, the relevant components of the strain tensor (required to vanish at the
boundary) are:
u 
1
θ
∂θ ur + r∂r
,
(173)
Srθ =
r
r
u 
1
φ
Srφ =
∂φ ur + r∂r
.
(174)
r sin θ
r
Both of them vanish if we require
ur = 0 ,

∂r (uθ /r) = 0 ,

∂r (uφ /r) = 0 .

(175)

136

T HE P ENCIL C ODE

We implement this by requiring ur to be antisymmetric and uθ /r and uφ /r to be symmetric with respect to the boundary.
The more general condition
rα ∂r (uθ /rα ) = ∂r uθ −

α
uθ = 0
r

(176)

(where α is a constant) can be implemented by requiring uθ /rα to be symmetric.
At a boundary θ = const, the stress-free boundary condition will take the form
u 
1
θ
Srθ =
∂θ ur + r∂r
=0,
r
r
 u 
1
φ
Sθφ =
∂φ uθ + sin θ ∂θ
=0.
r sin θ
r sin θ

(177)
(178)

With uθ = 0, the first condition gives ∂θ ur = 0, i.e. we require ur to be symmetric with
respect to the boundary. The second condition requires
sin θ  uφ 
∂θ
=0
r
sin θ

(179)

and is implemented by requiring uφ / sin θ to be symmetric.
D.3

Normal-field-radial boundary condition

While unphysical, this boundary condition is often used as a cheap replacement for a
potential-field condition for the magnetic field. It implies that the two tangential components of the magnetic field are zero at the boundary, while the normal component is
left unconstrained.
At a radial boundary, this gives:
1
1
∂φ Ar − ∂r (rAφ ) = 0 ,
r sin θ
r
1
1
=
∂r (rAθ ) − ∂θ Ar = 0 .
r
r

Bθ =

(180)

Bφ

(181)

Which are satisfied by setting
Ar = 0 ,

∂r (rAθ ) = 0 ,

∂r (rAφ ) = 0 ,

(182)

and these are implemented by requiring Ar to be antisymmetric, and rAθ and rAφ to be
symmetric.
On a boundary θ = const, we have
r sin θ Br = ∂θ (sin θ Aφ ) − ∂φ Aθ = 0 ,
rBφ = ∂r (rAθ ) − ∂θ Ar = 0

(183)
(184)

which can be achieved by setting
∂θ Ar = 0 ,

Aθ = 0 ,

∂θ (sin θ Aφ ) = 0 .

We thus require Ar and sin θ Aφ to be symmetric, and Aθ to be antisymmetric.

(185)

E. High-frequency filters

137

E High-frequency filters
Being high order, P ENCIL C ODE has much reduced numerical dissipation. In order to
perform inviscid simulations, high-frequency filters can be used to provide extra dissipation for modes approaching the Nyquist frequency. Usual Laplacian viscosity ν∇2 u is
equivalent to a multiplication by k 2 in Fourier space, where k is the wavenumber. Another tool is hyperviscosity, which replaces the k 2 dependency by a higher power-law, k n ,
n>2. The idea behind it is to provide large dissipation only where it is needed, at the grid
scale (high k), while minimizing it at the largest scales of the box (small k). In principle,
one can use as high n as desired, but in practice we are limited by the order of the code.
A multiplication by k n is equivalent to an operator ∇n in real space. As P ENCIL C ODE
is of sixth order, three ghost cells are available in each direction, thus the sixth-order
derivative is the highest we can compute. The hyperdissipation we use is therefore ∇6 ,
or k 6 is Fourier space. Figure 19 illustrates how such tool maximizes the inertial range
of a simulation.
Simplified hyperdiffusivity has been implemented for many dynamical variables and
can be found in the respective modules. A strict generalization of viscosity and resistivity
to higher order is implemented in the modules ‘hypervisc_strict_2nd’ and ‘hyperresi_strict_2nd’.
Hyperdiffusivity is meant purely as a numerical tool to dissipate energy at small scales
and comes with no guarantee that results are convergent with regular second order
dissipation. See Haugen & Brandenburg (2004) for a discussion. In fact, large-scale dynamo action is known to be seriously altered in simulations of closed systems where
magnetic helicity is conserved: this results in prolonged saturation times and enhanced
saturation amplitudes (Brandenburg & Sarson 2002).

E.1

Conservative hyperdissipation

It is desirable to have this high-frequency filter obeying the conservation laws. So, for
density we want a mass conserving term, for velocities we want a momentum conserving
term, for magnetic fields we want a term conserving magnetic flux, and for entropy we
want an energy conserving term. These enter as hyperdiffusion, hyperviscosity, hyperresistivity, and hyper heat conductivity terms in the evolution equations. To ensure conservation under transport, they must take the form of the divergence of the flux J of the
quantity ψ, so that Gauss theorem applies and we have
∂ψ
+∇·J =0
∂t

(186)

For density, the flow due to mass diffusion is usually taken as the phenomenological
Fick’s Law
J = −D∇ρ
(187)
i.e., proportional to the density gradient, in the opposite direction. This leads to the
usual Laplacian diffusion
∂ρ
= D∇2 ρ
∂t

(188)

138

T HE P ENCIL C ODE

Laplacian vs hyper dissipation
2.0

ψ

1.5
1.0
0.5
0.0
0.5

1.0

1.5

2.0

1.5

2.0

r

Dn∇2nψ

4

n=1

2
0
−2
n=3

−4
0.5

1.0

~
k2n Dn|ψ|

r
100
10−2
10−4
10−6
10−8
10−10
10−12
1

10

100
k

Figure 19: Dissipation acting on a scalar field ψ, for n=1 (Laplacian dissipation) and n=3 (third-order
hyperdissipation). The field is initially seeded with noise (upper panel). For n=3 the large scale is not
affected as much as in the n=1 case, which is seen by the larger wiggling of the latter in the middle panel.
In Fourier space (lower panel) we see that near the grid scale both formulations give strong dissipation.
It also illustrates that at the large scales (k≃1), the effect of n=3 is indeed negligible.

under the assumption that the diffusion coefficient D is isotropic. Higher order hyperdiffusion of order 2n involves a generalization of Eq. (187), to
J (n) = (−1)n D(n) ∇2n−1 ρ .

(189)

In our case, we are interested in the case n = 3, so that the hyperdiffusion term is
∂ρ
= D(3) ∇6 ρ.
∂t

(190)

The hyperdiffusion coefficient D(3) can be calculated from D assuming that at the
Nyquist frequency the two formulations (188) and (190) yield the same quenching. Considering a wave as a Fourier series in one dimension (x), one element of the series is
expressed as
ψk = Aei(kx+ωt)
(191)

E.2

Hyperviscosity

139

Plugging it into the second order diffusion equation (188) we have the dispersion condition iω = −Dk 2 . The sixth order version (190) yields iω = −D(3) k 6 . Equating both
we have D(3) = Dk −4 . This condition should hold at the grid scale, where k = π/∆x,
therefore

4
∆x
(3)
D =D
(192)
π
For the magnetic potential, resistivity has the same formulation as mass diffusion
∂A
= −η∇ × B = η∇2 A,
∂t

(193)

where we used the Coulomb gauge ∇ · A = 0. The algebra is the same as above, also
yielding η (3) = η(∆x/π)4 . For entropy, the heat conduction term is
1
∂S
=
∇ · (K∇T ) ,
∂t
ρT

(194)

and requiring that K be constant, we substitute it by
K (3) 6
∂S
=
∇ T.
∂t
ρT

(195)

also with K (3) = K(∆x/π)4 .
E.2

Hyperviscosity

Viscosity has some caveats where subtleties apply. The difference is that the momentum
flux due to viscosity is not proportional to the velocity gradient, but to the rate-of-strain
tensor
1
Sij =
2



∂ui ∂uj
2
+
− δij ∇ · u
∂xj
∂xi
3



,

(196)

which only allows the viscous acceleration to be reduced to the simple formulation ν∇2 u
under the condition of incompressibility and constant dynamical viscosity µ = νρ. Due
to this, the general expression for conservative hyperviscosity involves more terms. In
some cases, it is no great overhead, but for others, simpler formulations can be applied.
E.2.1 Conservative case
In the general case, the viscous acceleration is
fvisc = ρ−1 ∇ · (2ρνS)

(197)

So, for the hyperviscous force, we must replace the rate-of-strain tensor by a high order
version


(hyper)
(198)
= ρ−1 ∇ · 2ρνn S(n)
fvisc

where the nth -order rate of strain tensor is

S(n) = (−∇2 )n−1 S.

(199)

140

T HE P ENCIL C ODE

For the n = 3 case it is
(3)
Sij

1
=
2



∂ 5 uj
∂4
+
∂xi 5 ∂xi 4



∂ui
∂xj



1 ∂4
−
(∇ · u)
3 ∂xi 4



.

(200)

Plugging it into Eq. (198), and assuming µ3 = ρν3 = const
(hyper)
fvisc

= ν3



1
∇ u + ∇4 (∇(∇ · u))
3
6



.

(201)

For ν3 = const, we have to take derivatives of density as well
(hyper)
fvisc



1 4
(3)
6
= ν3 ∇ u + ∇ (∇(∇ · u)) + 2S · ∇ln ρ
3

(202)

E.2.2 Non-conservative cases
Equations (201) and (202) explicitly conserve linear and angular momentum. Although
desirable properties, such expressions are cumbersome and numerically expensive, due
to the fourth order derivatives of ∇(∇ · u).
This term, however, is only important when high compressibility is present (since it
depends on the divergence of u). In practice we drop this term and use a simple hyperviscosity
(
6
ν3 ∇
 u
 if µ = const
fvisc =
(203)
(3)
6
ν3 ∇ u + 2S · ∇ln ρ if ν = const
Notice that this can indeed be expressed as the divergence of a simple rate-of-strain
tensor
∂ 5 ui
(3)
,
(204)
Sij =
∂xj 5
so it does conserve linear momentum. It does not, however, conserve angular momentum, since the symmetry of the rate-of-strain tensor was dropped. Thus, vorticity sinks
and sources may be spuriously generated at the grid scale.
A symmetric tensor can be computed, that conserves angular momentum and can be
easily implemented


1 ∂ 5 ui ∂ 5 uj
Sij =
+
(205)
2 ∂xj 5 ∂xi 5
This tensor, however, is not traceless, and therefore accurate only for weak compressibility. It should work well if the turbulence is subsonic. Major differences are not expected,
since the spectral range in which hyperviscosity operates is very limited: as a numerical
tool, only its performance as a high-frequency filter is needed. This also supports the
usage of the highest order terms only, since these are the ones that provide quenching
at high k. Momentum conservation is a cheap bonus. Angular momentum conservation
is perhaps playing it too safe, at great computational expense.

E.2

Hyperviscosity

141

E.2.3 Choosing the coefficient
When changing the resolution, one wants to keep the grid Reynolds number, here defined as



2n−1
Regrid = urms νn kNy
(206)

approximately constant. Here, kNy = π/δx is the Nyquist wavenumber and δx is the
mesh spacing. Thus, when doubling the number of meshpoints, we can decrease the
viscosity by a factor of about 25 = 32 (Haugen & Brandenburg 2004). This shows that
hyperviscosity can allow a dramatic increase of the Reynolds number based on the scale
of the box.
By choosing idiff=’hyper3_mesh’ in density_run_pars the hyperdiffusion for density is
being set automatically in a mesh-independent way. A hyper-mesh Reynolds number of
30 corresponds to a coefficient diffrho_hyper3_mesh=2 if maxadvec is about 1, but in
practice we need a bit more (5 is currently the default).
E.2.4 Turbulence with hyperviscosity
When comparing hyperviscous simulations with non-hyperviscous ones, it turns out that
the Reynolds number at half the Nyquist frequency is usually in the range 5–7, i.e.


Rehalf−grid = urms νn (kNy /2)2n−1 ≈ 5–7
(207)

The following table gives some typical values used in simulations with forcing wavenumber kf = 1.5 and a forcing amplitude of f0 = 0.02. If hyperdiffusion D3 is used in the
continuity equation, the corresponding values are about 30 times smaller than those of
ν3 ; see Table 14.
Table 14: Empirical values of viscosity and hyperviscosity, as well as hyperdiffusion for density, at different numerical resolution, for simulations with forcing wavenumber kf = 1.5 and a forcing amplitude of
f0 = 0.02 in a 2π periodic domain. In all cases the half-mesh Reynolds number is about 5–7. For comparison, estimates of the numerical 4th order hyperdiffusion resulting from a third order time step are give
for two values of the CFL parameter.

N
16
32
64
128
256
512
1024

ν1
1 × 10−2
5 × 10−3
2 × 10−3
1 × 10−3
5 × 10−4
2 × 10−4
1 × 10−4

ν2
3 × 10−4
4 × 10−5
5 × 10−6
6 × 10−7
8 × 10−8
1 × 10−8
1 × 10−9

ν3
2 × 10−5
6 × 10−7
2 × 10−8
6 × 10−10
2 × 10−11
6 × 10−13
2 × 10−14

D3
6 × 10−7
2 × 10−8
6 × 10−10
2 × 10−11
6 × 10−13
2 × 10−14
6 × 10−16

κCFL=0.4
2
7 × 10−4
1 × 10−6
2 × 10−7
3 × 10−8
4 × 10−9
5 × 10−10
6 × 10−11

κCFL=0.9
2
1 × 10−4
2 × 10−5
3 × 10−6
4 × 10−7
5 × 10−8
6 × 10−9
8 × 10−10

For comparison, we give in Table 14 estimates of the numerical 4th order hyperdiffusion
resulting from a third order time step, for which we have
κCFL
=
2

1
urms (CCFL δx)3
24

(208)

where CCFL is the CFL parameter which is either 0.4 in the conservative case or 0.9 in
the more progressive case.

142

E.3

T HE P ENCIL C ODE

Anisotropic hyperdissipation

As we want quenching primarily at the Nyquist frequency, hyperdissipation depends
intrinsically on the resolution, according to Eq. (192). Because of this, isotropic hyperdissipation only gives equal quenching in all spatial directions if ∆x=∆y=∆z, i.e., if the
cells are cubic. For non-cubic cells, anisotropic dissipation is required as different directions may be better/worse sampled, thus needing less/more numerical smoothing. Such
generalization is straightforward. For that, we replace Eq. (189) by


∂ 5ρ
∂ 5ρ
∂ 5ρ
J = Dx 5 , Dy 5 , Dz 5 ,
(209)
∂x
∂y
∂z
so that different diffusion operates in different directions. Since Dx , Dy and Dz are constants, the divergence of this vector is
∇ · J = Dx

∂ 6ρ
∂ 6ρ
∂ 6ρ
+
D
+
D
.
y
z
∂x6
∂y 6
∂z 6

(210)

The formulation for resistivity and heat conductivity are strictly the same. For viscosity
it also assumes the same form if we consider the simple non-conservative rate-of-strain
tensor (204).
Mathematically, these operations can be written compactly by noticing that the coeffi(3)
(3)
cients in Eq. (210) transform like diagonal tensors χij = χk δijk , where δijk is the unit
diagonal third order tensor, χ(3) is the vector containing the dissipative coefficients (diffusion, viscosity, resistivity, or heat conductivity) in x, y, and z, and summation over
repeated indices applies.
Therefore, for a scalar quantity ψ (density, any of the three components of the velocity
or magnetic potential), we can write
X
∂ψ
∂6
(3)
ψ.
χ(3)
= −χij ∂ i ∂ 5j ψ = −
q
∂t
∂x6q
q
E.4

(211)

Hyperviscosity in Burgers shock

Figure 20: Left: Burgers shock from teach/PencilCode/material/BurgersShock (in the teaching material)
with −20 ≤ x ≤ 20, nx = 64 mesh points, ux = ∓1 on the two ends, ν = 0.4 and either ν3 = 0 (solid line)
or ν3 = 0.05 (dashed line). Right: similar to the left hand side, but with ν = 0 and ν3 = 0.05 (dashed line),
compared with the case ν = 0.4 and ν3 = 0 (solid line).

E.4

Hyperviscosity in Burgers shock

143

Hyperviscosity has the unfortunate property of introducing (numerically stable) wiggles,
even if one just adds a little bit of hyperviscosity to a run with normal viscosity; see left
hand side of Fig. 20. Running with just hyperviscosity give strong wiggles.

144

T HE P ENCIL C ODE

F Special techniques
F.1

After changing REAL PRECISION

To continue working in double precision (REAL_PRECISION=double), you just say lread_from_other_prec=T in run_pars. Afterwards, remember to put lread_from_other_prec=F.
If continuation is done in a new run directory, first execute start.csh there and then copy
the files var.dat (and if present global.dat) from the old to the new directory, using pc copyvar.

F.2

Remeshing (regridding)

[This should be written up in a better way and put somewhere else. But currently,
remeshing is only available for the Pencil developers anyway.]
Suppose you have a directory run 64 with a 643 run (running on N0 =ny ×nz =2×1 CPUs)
that you want to continue from ‘VAR1’ at 1283 (on ny ×nz =4 × 4 CPUs).
1. Get the remeshing stuff from repository:
unix>
unix>

cd $PENCIL_HOME; cvs co -d remesh pencil-remesh
setenv PATH ${PATH}:$PENCIL_HOME/remesh/bin

2. Create another run directory with current ‘VAR1’ as ‘var.dat’ (remesh.csh so far
only works with ‘var.dat’):
unix> cd run_64
run_64> pc_newrun ../tmp_64 or new tmp_64
run_64> mkdir -p ../tmp_64/data or (cd ../tmp_64/; crtmp)
run_64> (cd ../tmp_64/data ; mkproc-tree N0 )
run_64> restart-new-dir-VAR ../tmp_64 1
3. Create the new run directory (linking the executables with -s):
run_64> cd ../tmp_64
tmp_64> pc_newrun -s ../run_128 or new run_128
tmp_64> vi ../run_128/src/cparam.local
# set nxgrid=128, ncpus=16, nprocy=4
tmp_64> (cd ../run_128; crtmp; pc_setupsrc; make)
4. Setup and do remeshing
tmp_64> setup-remesh
tmp_64> vi remesh/common.local
# set muly=2, mulz=4, remesh par=2
tmp_64> (cd remesh; make)
tmp_64> vi remesh.in
# Replace line by ../run 128
tmp_64> remesh.csh
# Answer ‘yes’

F.3

F.3

Restarting from a run with less physics

145

Restarting from a run with less physics

First, prepare a new run directory with the new physics included. By new physics, we
mean that the new run wants to read in more fields (e.g. magnetic fields, if the old run
didn’t have magnetic fields).
Example for test fields:
1. Prepare ‘src/cparam.local’
Add the following 2 fragments into the ‘cparam.local’ file. The first piece comes in
the beginning and the second in the end of the file.
!** AUTOMATIC CPARAM.INC GENERATION ****************************************
! Declare (for generation of cparam.inc) the number of f array
! variables and auxiliary variables added by this module
! Use MVAR to reserve the appropriate workspace for testfield_z.f90
! The MAUX number must be equally big and is used for uxb in the f-array.
! At the end of this file, njtest must be set such that 3*njtest=MVAR.
!
! MVAR CONTRIBUTION 12
! MAUX CONTRIBUTION 12
!
!***************************************************************************
!
! note that MVAR=MAUX=3*njtest must be obeyed
!
integer, parameter :: njtest=4
!
2. Prepare ‘src/Makefile.local’
Add the line TESTFIELD=testfield_z to the file. Finally, compile the code.
3. Prepare restart data
Go into data directory of the new run and prepare the directory tree using, e.g., the
command pc_mkproctree 16. [In principle this could be automatized, but it isn’t
yet.]
Next, go into old run directory and say restart-new-dir ../32c, if ‘../32c’ is the
name of the new run directory. This procedure copies all the files from the processor
tree, plus files like ‘param.nml’, but this file may need some manual modification (or
you could just us one from another runs with the new physics included, which is
definitely the simplest!).
4. Prepare ‘run.in’
Set lread_oldsnap_notestfield=T in run pars . This means (as the name says) that
one reads an old snapshot that did not have test fields in it.
Reset boundary conditions and add stuff for the newly added fields, e.g.,
bcz=’a:s’,’a’,’a:s’,’a2’,’a’,’a’,’s’,’a’,’a’,’s’,’a’,’a’,’s’,’a’,’a’,’s’
in run pars . If you don’t do this, you would effectively use periodic boundary
conditions for the response to the test field, which is hardly correct once you set
non-periodic boundary conditions for the other variables.

146

T HE P ENCIL C ODE
Add something like the following text fragments in the right position (after grav run pars and magnetic run pars , but before shear run pars and viscosity run pars .
&testfield_run_pars
!linit_aatest=T, daainit=100.
itestfield=’B11-B22’
etatest=1e-4
lsoca=F
/
Make sure that the data above are correct. You may want to change the values of
daainit or etatest .
If you now run, and if you didn’t fix the file ‘data/param.nml’ you might get something like the following error:

forrtl: severe (24): end-of-file during read, unit 1, file /wkspace/brandenb/pencil-c
The reason for this is that it reads the old boundary data, but the corresponding array is too short. This includes stuff like FBCX1 to FBCX2 2 , but it is still
not enough. Therefore it is easiest to use the ‘data/param.nml’ file from another
run. You may well just use one from a single processor run with a different mesh.
But remember to fix the ‘start.in’ file by correcting the boundary conditions and
adding things like
&testfield_init_pars
luxb_as_aux=T
/
5. Prepare ‘print.in’, ‘xyaver.in’, and other obvious files such as ‘video.in’.
6. Once it works and is running, you must say explicitly
&run_pars
...
lread_oldsnap_notestfield=F
/
because otherwise you won’t read in your precious test field data next time you
restart the code! (If you instead just remove this line, it will remember lread_oldsnap_notestfield=T from the previous run, which is of course wrong!)
Comments: For large magnetic Reynolds numbers the solutions to the test-field equations can show a linear instability, which can introduce large fluctuations. In that case
it is best to reset the dependent test-field variable to zero in regular intervals. This is
done by setting linit aatest=T. Note that daainit=100 sets the reset interval to 100.

G. Runs and reference data

147

G Runs and reference data
For reference purposes we document here some results obtained with various samples
of the code.
G.1

Shock tests

G.1.1 Sod shock tube problem
Table 15: Combinations of ρ, p, and s/cp that are relevant for the Sod shock tube problem with constant
temperature and different pressure ratios on the left and right hand sides of the shock.

ρ

p

s

1.0
0.1
0.01

1.0
0.3065
0.1
1.2275
0.01 2.1486

G.1.2 Temperature jump
Table 16: Combinations of c2s , p, and s/cp that are relevant for the temperature shock problem with
constant density, ρ = 1, and different temperature ratios on the left and right hand sides of the shock.

G.2

c2s

s

1.0
0.1
0.01
10−4

0.0
−2.3
−4.6
−9.2

Random forcing function

A solenoidal random forcing function f can be invoked by putting iforce=’helical’ in
the forcing run pars namelist. This produces the forcing function f of the form
f (x, t) = Re{N f k(t) exp[ik(t) · x + iφ(t)]},

(212)

where k(t) = (kx , ky , kz ) is a time dependent wave vector, x = (x, y, z) is position, and
φ(t) with |φ| < π is a random phase. On dimensional grounds the normalization factor
is chosen to be N = f0 cs (kcs /δt)1/2 , where f0 is a nondimensional factor, k = |k|, and δt
is the length of the timestep. The δt−1/2 dependence ensures that the forcing, which is
delta-correlated in time, is properly normalized such that the correlator of the forcing
function is independent of the length of the time step, δt. We focus on the case where
|k| is around 5, and select at each timestep randomly one of the 350 possible vectors in
4.5 < |k| < 5.5. We force the system with eigenfunctions of the curl operator,
ik × (k × e) − σ|k|(k × e)
q
fk = √
,
2
2
2
2
1 + σ k 1 − (k · e) /k

(213)

148

T HE P ENCIL C ODE

where e is an arbitrary unit vector needed in order to generate a vector k × e that is
perpendicular to k. Note that |f k |2 = 1 and, for σ = 1, ik × f k = |k|f k , so the helicity
density of this forcing function satisfies
f · ∇ × f = |k|f 2 > 0

(214)

(for σ = 1)

at each point in space. We note that since the forcing function is like a delta-function in
k-space, this means that all points of f are correlated at any instant in time, but are different at the next timestep. Thus, the forcing function is delta-correlated in time (but the
velocity is not). This is the forcing function used in Brandenburg (2001), Brandenburg
& Dobler (2001), and other papers in that series.
For σ = 0, the forcing function is completely nonhelical and reduces to the simpler form
q
f k = (k × e) / k2 − (k · e)2 .
(215)

For 0 < |σ| < 1, the forcing function has fractional helicity, where σ ≈ hω · ui /(kf hu2 i);
see Sect. 4.5 of Ref. [7]. In the code and the forcing run pars namelist, σ is called relhel .

In the code, the possible wavevectors are pre-calculated and stored in ‘k.dat’, which
is being read in the beginning the code runs. To change the wavevectors (e.g.
the typical value of kf , you need to change the file. In the directory ‘$PENCIL_HOME/samples/helical-MHDturb/K_VECTORS/’ there are several such files prepared:
k10.dat
k15.dat

k1.dat
k27.dat

k2.dat
k30.dat

k3.dat
k4.dat

k5.dat
k8.dat

and more can be prepared in IDL with the procedure ‘$PENCIL_HOME/samples/helical-MHDturb/idl/generate_kvectors.pro’ There is also more help in
the ‘README’ file in ‘helical-MHDturb’.
Three-layered convection model
ln ρ

uz

Entropy s

Temperature T
1.5

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

z

1.5

z

1.5

z

1.5

z

G.3

0.0

0.0

0.0

0.0

−0.5

−0.5

−0.5

−0.5

−1

0

1

2

3

−0.15
−0.10
−0.050.000.05

−0.6−0.4−0.20.0 0.2 0.4

0.5

1.0

1.5

2.0

Figure 21: Like in Fig. 2, but at time t = 50.

In Sect. 3 we have shown the early stages of the convection model located in
‘samples/conv-slab’. To arrive at fully developed convection, you will need to run the

G.4

Magnetic helicity in the shearing sheet

149

1.000

u

0.100

0.010

umax
urms
0.001
0

10

20

30

40

50

t

Figure 22: Time evolution of rms and maximum velocity for the model ‘samples/conv-slab’. Similar plots
can be produced by running the IDL script ‘ts.pro’.

code for many more time steps. Figure 21 shows the vertical profiles of four basic quantities at time t = 50. Figure 22 shows the time evolution of rms and maximum velocity
for the model for 0 < t < 50.
Figures 23 and 24 show vertical and horizontal sections for time t = 50.
u at z=0.352258
0.4

0.2

0.2

0.2

0.0

0.0

0.0

y

0.4

−0.2

−0.2

−0.2

−0.4

−0.4

−0.4

−0.4−0.2 0.0 0.2 0.4
x

s and ρ at z=−0.292903

−0.4−0.2 0.0 0.2 0.4
x

s and ρ at z=0.352258

−0.4−0.2 0.0 0.2 0.4
x

s and ρ at z=0.932903

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

y

0.4

y

y

u at z=0.932903

0.4

y

y

u at z=−0.292903

−0.2

−0.2

−0.2

−0.4

−0.4

−0.4

−0.4−0.2 0.0 0.2 0.4
x

−0.4−0.2 0.0 0.2 0.4
x

−0.4−0.2 0.0 0.2 0.4
x

Figure 23: Horizontal sections for t = 50. Top: velocity field. Bottom: entropy (color coded) and density
(isocontours). Plots of this type can be produced by running the IDL script ‘hsections.pro’)

G.4

Magnetic helicity in the shearing sheet

To test magnetic helicity evolution in the shearing shear, we can choose as initial
condition initaa=’Beltrami-y’ with amplaa=1. in magnetic_init_pars together with
Sshear=-1. in shear_run_pars.

150

T HE P ENCIL C ODE

u at y=−0.296875

u at y=0.0156250

u at y=0.296875

0.5

0.5

0.5

z

1.0

z

1.0

z

1.0

0.0

0.0

0.0

−0.5

−0.5

−0.5

−0.4 −0.2 0.0 0.2 0.4
x

−0.4 −0.2 0.0 0.2 0.4
x

−0.4 −0.2 0.0 0.2 0.4
x

s and ρ at y=−0.296875

s and ρ at y=0.0156250

s and ρ at y=0.296875

0.5

0.5

0.5

z

1.0

z

1.0

z

1.0

0.0

0.0

0.0

−0.5

−0.5

−0.5

−0.4 −0.2 0.0 0.2 0.4
x

−0.4 −0.2 0.0 0.2 0.4
x

−0.4 −0.2 0.0 0.2 0.4
x

Figure 24: Vertical section y = 0.516 at t = 50. Top: velocity field. Bottom: entropy (color coded) and
density (isocontours). Plots of this type can be produced by running the IDL scripts ‘vsections.pro’) or
‘vsections2.pro’).

Thus, in ‘src/Makefile.local’ we just use
MAGNETIC=magnetic
HYDRO=nohydro
EOS=noeos
DENSITY=nodensity
SHEAR=shear
VISCOSITY=noviscosity
and put
&init_pars
cvsid=’$Id$’,
/
&magnetic_init_pars
initaa=’Beltrami-y’, amplaa=1.
/
&shear_init_pars

G.4

Magnetic helicity in the shearing sheet

151

Figure 25: Wind-up of the magnetic field leads to a linear increase in the rms magnetic field strength
until Ohmic diffusion begins to become important (top panel). During this time the magnetic helicity is
conserved. With Ohmic diffusion, the decay of hA·Bi is well described by integrating −2ηhJ ·Bi (indicated
by “from rhs” in the second panel).

/
in ‘start.in’ and, for example,
&run_pars
cvsid=’$Id$’
nt=150000, it1=10, cdt=0.9, isave=50, itorder=3
dsnap=100. dvid=5., ialive=1
/
&magnetic_run_pars
eta=0.
/
&shear_run_pars
Sshear=-1.
/
in ‘run.in’. The output includes, among other things
arms(f10.7)

152

T HE P ENCIL C ODE

brms(f12.7)
jrms(f14.7)
abm(f14.11)
jbm(f14.7)
The result is shown in Figure 25, where we show the wind-up of the magnetic field,
which leads to a linear increase in the rms magnetic field strength until Ohmic diffusion begins to become important (top panel). During this time the magnetic helicity is
conserved. With Ohmic diffusion, the decay of hA · Bi is well described by integrating
−2ηhJ · Bi (indicated by “from rhs” in the second panel).

H. Numerical methods

H
H.1

153

Numerical methods
Sixth-order spatial derivatives

Spectral methods are commonly used in almost all studies of ordinary (usually incompressible) turbulence. The use of this method is justified mainly by the high numerical
accuracy of spectral schemes. Alternatively, one may use high order finite differences
that are faster to compute and that can possess almost spectral accuracy. Nordlund &
Stein [26] and Brandenburg et al. [12] use high order finite difference methods, for example fourth and sixth order compact schemes [24].18
The sixth order first and second derivative schemes are given by
fi′ = (−fi−3 + 9fi−2 − 45fi−1 + 45fi+1 − 9fi+2 + fi+3 )/(60δx),
fi′′ = (2fi−3 − 27fi−2 + 270fi−1 − 490fi + 270fi+1 − 27fi+2 + 2fi+3 )/(180δx2 ),

(216)
(217)

In Fig. 26 we plot effective wavenumbers for different schemes. Apart from the different
explicit finite difference schemes given above, we also consider a compact scheme of 6th
order, which can be written in the form
1 ′
f
3 i−1

′
+ fi′ + 31 fi+1
= (fi−2 − 28fi−1 + 28fi+1 − fi+2 )/(36δx),

(218)

for the first derivative, and
2 ′′
f
11 i−1

+ fi′′ +

2 ′′
f
11 i+1

= (3fi−2 + 48fi−1 − 102fi + 48fi+1 + 3fi+2 )/(44δx2 ).

(219)

for the second derivative. As we have already mentioned in the introduction, this scheme
involves obviously solving tridiagonal matrix equations and is therefore effectively nonlocal.
In the second panel of Fig. 26 we have plotted effective wavenumbers for second derivatives, which were calculated as
2
(cos kx)′′num = −keff
cos kx.

(220)

Of particular interest is the behavior of the second derivative at the Nyquist frequency,
because that is relevant for damping zig-zag modes. For a second-order finite difference
2
scheme keff
is only 4, which is less than half the theoretical value of π 2 = 9.87. For
fourth, sixth, and tenth order schemes this value is respectively 5.33, 6.04, 6.83. The
last value is almost the same as for the 6th order compact scheme, which is 6.86. Significantly stronger damping at the Nyquist frequency can be obtained by using hyperviscosity, which Nordlund & Galsgaard (1995) treat as a quenching factor that diminishes
the value of the second derivative for wavenumbers that are small compared with the
Nyquist frequency. Accurate high order second derivatives (with no quenching factors)
are important when calculating the current J in the Lorentz force J × B from a vector
potential A using −µ0 J = ∇2 A−∇∇·A. This will be important in the MHD calculations
presented below.
18

The fourth order compact scheme is really identical to calculating derivatives from a cubic spline,
as was done in Ref. [26]. In the book by Collatz [13] the compact methods are also referred to as Hermitian methods or as Mehrstellen-Verfahren, because the derivative in one point is calculated using the
derivatives in neighboring points.

154

T HE P ENCIL C ODE

Figure 26: Effective wave numbers for first and second derivatives using different schemes. Note that
for the second derivatives the sixth order compact scheme is almost equivalent to the tenth order explicit
scheme. For the first derivative the sixth order compact scheme is still superior to the tenth order explicit
scheme.

H.2

Upwind derivatives to avoid ‘wiggles’

High-order centered-difference convection simulations often show “wiggles” (Nyquist
zigzag pattern) in ln ρ, which are apparently caused by a velocity profile where the velocity approaches zero on the boundary or inside the box.19 This causes the density profile
to be squeezed into a boundary layer where eventually numerical resolution is insufficient and, for centered differences, a spurious Nyquist signal is generated that almost
instantaneously propagates into much of the whole box.
Even if the stagnation point is on the boundary (and enforced by the boundary conditions), this behavior is hardly influenced by the boundary conditions on ln ρ at all.
A solution, however, is to apply upwinded derivative operators. The simplest upwind
derivative is a finite-difference derivative operator where the point furthest downwind
is excluded from the stencil. For u > 0, that means that instead of
f0′ =



−f−3 + 9f−2 − 45f−1
+ 45f1 − 9f2 + f3 δx6 f (7)
−
= D(cent,6) + O δx6 ,
60 δx
140

(221)

one takes
f0′ =



−2f−3 + 15f−2 − 60f−1 + 20f0 + 30f1 − 3f2 δx5 f (6)
+
= D(up,5) + O δx5 .
60 δx
60

(222)

A fourth-order upwind scheme (excluding two downwind points) would be
f0′ =
19



−f−3 + 6f−2 − 18f−1 + 10f0 + 3f1 δx4 f (5)
−
= D(up,4) + O δx4 .
12 δx
20

(223)

A simple one-dimensional test profile would be u(x) = 1 − x2 on x ∈ [−1, 1], which will accumulate
more and more mass near the right boundary x = 1.
In two- or three-dimensional settings, the presence of stagnation points of X-type leads to the same
configuration, this time with the possibility of a steady state (i.e. without accumulation of mass). Such
stagnation points occur e.g. at the top of an upwelling, or at the bottom of a downdraft in convection
simulations, where locally uz ∝ zX − z.

H.3

The bidiagonal scheme for cross-derivatives

155

The effect of upwinding is mostly to stop the Nyquist perturbations from spreading away
from the boundary or stagnation point. With the fourth-order formula they actually
hardly ever develop.
The difference between central and fifth-order upwind derivative is
[D(up,5) − D(cent,6) ]f0 =

δx5 (6)
−f−3 + 6f−2 − 15f−1 + 20f0 − 15f1 + 6f2 − f3
=−
f .
60 δx
60 0

(224)

In other words, 5th-order upwinding can be represented for any sign of u as hyperdiffusion (Dobler et al. 2006):
′
′
− uf(up,5th)
= −uf(centr,6th)
+

|u| δx5 (6)
f .
60

(225)

The advantage over adopting constant hyperdiffusion is that in the upwinding scheme
hyperdiffusion is only applied where it is needed (i.e. where advection is happening,
hence the factor |u|).
The form (225) also suggests an easy way to get ‘stronger’ upwinding: Rather than excluding more points in the downwind direction, we can simply treat the weight of the
hyperdiffusion term as a free parameter α:
′
′
− uf(up,5th,α)
= −uf(centr,6th)
+ α |u| δx5 f (6) .

(226)

If α is large, this may affect the time step, but for α = 1/60, the stability requirement for
the hyperdiffusive term should always be weaker than the advective Courant criterion.
H.3

The bidiagonal scheme for cross-derivatives

The old default scheme used for cross-derivatives of type ∂ 2 /(∂x∂y) used to read as follows:
df=fac*( &
270.*( f(l1+1:l2+1,m+1,n,k)-f(l1-1:l2-1,m+1,n,k)
+f(l1-1:l2-1,m-1,n,k)-f(l1+1:l2+1,m-1,n,k))
- 27.*( f(l1+2:l2+2,m+2,n,k)-f(l1-2:l2-2,m+2,n,k)
+f(l1-2:l2-2,m-2,n,k)-f(l1+2:l2+2,m-2,n,k))
+ 2.*( f(l1+3:l2+3,m+3,n,k)-f(l1-3:l2-3,m+3,n,k)
+f(l1-3:l2-3,m-3,n,k)-f(l1+3:l2+3,m-3,n,k))

&
&
&
&
&
&

)
and is “visualized” in the left part of Fig. 27. It is way more efficient than the straightforward approach of first taking the x and the y derivative consecutively. (shown in the
right part of Fig. 27).
Off-diagonal terms enter not only the diffusion terms through ∇∇ · u and ∇∇ · A terms,
but also through the J = ∇ × ∇ × A operator. The general formula is Ji = Aj,ij − Ai,jj ,
so in 2-D in the xy-plane we have
Jx = Ax,xx + Ay,xy − Ax,xx − Ax,yy = Ay,xy − Ax,yy ,
Jy = Ax,yx + Ay,yy − Ay,xx − Ay,yy = Ax,yx − Ay,xx

(227)
(228)

Figure 28 shows how the two schemes perform for the propagation of Alfvén waves,

156
-2
0
0
0
0
0
+2

T HE P ENCIL C ODE
0
+27
0
0
0
-27
0

0
0
-270
0
+270
0
0

0
0
0
0
0
0
0

0
0
+270
0
-270
0
0

0
-27
0
0
0
+27
0

+2
0
0
0
0
0
-2

9
-27
135
0
-135
27
-9

-27
81
-405
0
405
-81
27

135
-405
2025
0
-2025
405
-135

0
0
0
0
0
0
0

-135
405
-2025
0
2025
-405
135

27
-81
405
0
-405
81
-27

-9
27
-135
0
135
-27
9

Figure 27: Weights of bidiagonal scheme (left) and consecutive scheme (right) for mixed derivatives
∂ 2 /∂x∂y. The numbers shown need to be divided by 720 δx δy for the bidiagonal and by 3600 δx δy for
the consecutive scheme.

Figure 28: Alfvén wave for B 0 = (1, 2, 0) and k = (1, 2, 0) after t = 2π. The wave travels in the direction of
k. Red symbols are for the bidiagonal scheme, black symbols show results obtained using the consecutive
scheme. Already for 162 mesh points there are no clear differences. For 82 mesh points both schemes are
equally imprecise regarding the phase error, but the amplitude error is still quite small (but this is mainly
a property of the time stepping scheme).

u̇z = Jx B0y − Jy B0x ,
Ȧx = −uz B0y ,
Ȧy = +uz B0x .

(229)
(230)
(231)

The initial condition (as implemented in subroutine alfven_xy) is
uz ∼ cos(kx x + ky y − ωt) ,
Ax ∼ +B0y sin(kx x + ky y − ωt)/ω ,
Ay ∼ −B0x sin(kx x + ky y − ωt)/ω ,

(232)
(233)
(234)

where ω = k · B 0 . The figure shows that there is no clear advantage of either scheme, so
the code uses the more efficient bidiagonal one.
H.4

The 2N-scheme for time-stepping

For time stepping, higher-order schemes are necessary in order to reduce the amplitude
and phase errors of the scheme and, to some extent, to allow longer time steps. Usually

H.5

Diffusive error from the time-stepping

157

such schemes require large amounts of memory. However, we here use the memoryeffective 2N -schemes that require only two sets of variables to be held in memory. Such
schemes work for arbitrarily high order, although not all Runge-Kutta schemes can be
written as 2N -schemes [29, 28]. Consider the ordinary differential equation (ODE)
(235)

u̇ = F (u, t) ,

which can also be used as a prototype for a system of ODEs to be solved, like the ones
obtained by spatial discretization of PDEs. The 2N -schemes construct an approximation
to the solution
u(n) ≡ u(tn )
(236)
according to the iterative procedure

(237)
(238)

wi = αi wi−1 + δt F (ui−1 , ti−1 ) ,
ui = ui−1 + βi wi .

For a three-step (RK3-2N) scheme we have i = 1, ..., 3. In order to advance the variable
u from u(n) at time t(n) to u(n+1) at time t(n+1) = t(n) + δt we set in Eq. (238)
u0 = u(n)

and, after the last step,

u(n+1) = u3 ,

(239)

with u1 and u2 being intermediate steps. In order to be able to calculate the first step,
i = 1, for which no wi−1 ≡ w0 exists, we have to require α1 = 0. Thus, we are left with 5
unknowns, α2 , α3 , β1 , β2 , and β3 . Three conditions follow from the fact that the scheme be
third order for linear equations, so we have to have two more conditions. One possibility
is to choose the fractional times at which the right hand side is evaluated, for example
(0, 1/3, 2/3) or even (0, 1/2, 1). Yet another possibility is to require that inhomogeneous
equations of the form u̇ = tn with n = 1 and 2 are solved exactly. The corresponding
coefficients are listed in Table 17 and compared with those given by Williamson [29]. In
practice all of them are about equally good when it comes to real applications, although
we found the first one in Table 17 (‘symmetric’) marginally better in some simple test
problems where an analytic solution was known. In Ref. [3] the accuracy of some nonlinear equations is tested.
Table 17: Coefficients for different 2N -type third-order Runge-Kutta schemes. The coefficients ci (which
are determined by the αi , βi ) give the time for each substep, ti = t0 + ci δt

H.5

scheme

c1

c2

c3

α2

α3

β1

β2

β3

symmetric
[predictor/corrector]
inhomogeneous
Williamson (1980)

0
0
0
0

1/3
1/2
15/32
4/9

2/3
1
4/9
15/32

−2/3
−1/4
−17/32
−5/9

−1
−4/3
−32/27
−153/128

1/3
1/2
1/4
1/3

1
2/3
8/9
15/16

1/2
1/2
3/4
8/15

Diffusive error from the time-stepping

In many cases we use centered first derivatives for the advection operator, so the resulting discretization errors are only of dispersive nature (proportional to odd derivatives). A
diffusive error can be obtained from the discretization error of the time-stepping scheme.
For the RK3-2N scheme we have
 
 n+1 
 
df
d f
df
n
+ ...,
(240)
=
+ an δt
dt nth order
dt exact
dtn+1

158

T HE P ENCIL C ODE

where an = 1/(n + 1)! = 0.5. In particular, for n = 1 we have a1 = 1/2 = 0.2 and for
n = 3 we have a3 = 1/24 ≈ 0.04. The advection operator leads to a diffusive error equal
to a1δt(u · ∇)2 for n = 1 and a hyperdiffusive error equal to a3δt3 (u · ∇)4 for n = 3.
Substituting δt = cCFL δx/|u|, where cCFL is Courant-Friedrich-Lewy constant, we have a
diffusive error ν∇2 with negative ν = −a1cCFL |u|δx for n = 1, and a hyperdiffusive error
−νhyp ∇4 with positive νhyp = a3 c3CFL |u|δx3 for n = 3. The fact that the hyperdiffusive
error has a positive effective hyperdiffusivity is an important factor in the choice of this
scheme.
To decide whether the effective hyperdiffusivity from the diffusive error is significant,
we can compare with the error that would occur had we used a third-order upwinding
scheme (Sect. H.2). In that case we would have an effective hyperdiffusive coefficient
|u|δx/12 that is 1/(12a3 c3CFL ) ≈ 5.8 times larger than that from the time stepping scheme.
In this sense, the hyperdiffusive error can be regarded as small.
Since the hyperdiffusive error is proportional to −∇4 , we cannot directly compare with
the physical diffusion which is proportional to ∇2 . Therefore we define an effective vis2
cosity as νeff = νhyp kNy
with kNy = π/δx being the Nyquist wavenumber of the mesh
of the domain covered by N mesh points. We define Reynolds number based on the
Nyquist wavenumber as ReNy = |u|/νeff kNy , and find Re = −24/(πcCFL )3 ≈ 2.3 for our
favorite choice cCFL = 0.7. Thus, at the scale of the mesh, the effective Reynolds number
is comparable to the value often obtained in simulations. However, in turbulence simulations the viscous cutoff wavenumber is usually 5–10 times smaller than kNy , so the
relevant Reynolds number at the viscous scale is then another 2–3 orders of magnitude
larger and does therefore not impose a constraint in view of the physical viscosity that
is applied in such calculations.

H.6

Ionization

The specific entropy of each particle species (neutral hydrogen, electrons, protons and
neutral helium) may be written as
!
"
 3/2 #
si
5
1 ρi T
,
+
= xi ln
s0
xtot ρ T0
2

(241)

where
xH = 1 − y ,

xe = xp = y ,
s0 =

kB
,
µmH

xtot = 1 + y + xHe

T0 =

χH
,
kB

(242)
(243)

and
ρi = µmH
The specific entropy of mixing is

 m χ 3/2
i H
2π~2

.

X
smix
xi
=−
xi ln
.
s0
xtot
i

(244)

(245)

H.7

Radiative transfer

Summing up everything, we get the total specific entropy
"
!
 3/2 #
X si
5
1 ρi T
smix X
s
+
=
+
=
xi ln
s0
s0
s0
xi ρ T 0
2
i
i
!
"   #
3/2
X
5
1 T
ρi
.
+
=
xi ln + xtot ln
xi
ρ T0
2
i

159

(246)
(247)

Solving for T gives

P
s/s0 + i xi ln xi /ρi
3 T
5
ln
(248)
=
+ ln ρ − .
2 T0
xtot
2
Using this expression and the constants defined above, we may obtain the ionization
fraction y for given ln ρ and s by finding the root of
#
"  
3/2
T0
1−y
ρe T
−
F = ln
.
(249)
ρ T0
y2
T
The derivative with respect to y for Newton-Raphson is


∂F
1
2
3 T0 ∂ ln T /T0
=
+
−
− ,
∂y
2
T
∂y
1−y y

(250)

where

2
(ln ρH /ρp − F − T0 /T ) − 1
∂ ln T /T0
= 3
.
(251)
∂y
1 + y + xHe
In order to compute the pressure gradient in the momentum equation, the derivative of
y with respect to ln ρ and s needs to be known:

∂ ln P
∂y
1
∂ ln T
∂ ln T ∂y
=
+
+
+ 1,
∂ ln ρ
1 + y + xHe ∂ ln ρ
∂ ln ρ
∂y ∂ ln ρ
∂ ln P
∂y ∂ ln T
1
∂ ln T ∂y
=
+
+
.
∂s
1 + y + xHe ∂s
∂s
∂y ∂s
Since F = 0 for all desired solutions (y, ln ρ, s) we also have
dF =
and thus

and

H.7

∂F
∂F
∂F
d ln ρ +
ds +
dy = 0 ,
∂ ln ρ
∂s
∂y

∂y
=
∂ ln ρ



dy
d ln ρ

∂y
=
∂s



dy
ds





ds=0

d ln ρ=0

(253)

(254)

∂F/∂ ln ρ
∂F/∂y

(255)

∂F/∂s
.
∂F/∂y

(256)

=−
=−

(252)

Radiative transfer

H.7.1 Solving the radiative transfer equation
A formal solution of Eq. (76) is given by
I(τ ) = I(τ0 )e

−(τ −τ0 )

+

Zτ

τ0

′

e−(τ −τ ) S(τ ′ ) dτ ′ .

(257)

160

T HE P ENCIL C ODE

Using a generalization of the trapezoidal rule,
Zτ

e

−(τ −τ ′ )

′

f (τ ) dτ

′

Zτ

≈

τ0

τ0



=

e

−(τ −τ ′ )




f (τ )−f (τ0 ) ′
f (τ0 ) +
(τ − τ0 ) dτ ′
τ − τ0

(258)


1 − e−(τ −τ0 ) (1 + τ −τ0 )
[f (τ ) − f (τ0 )] , (259)
1−e−(τ −τ0 ) f (τ ) −
τ − τ0

which is exact for linear functions S(τ ), we discretize this as

1 − e−δτ (1 + δτ )
(Sk+1 − Sk ) ,
δτ
e−δτ − 1 + δτ
(Sk+1 − Sk ) .
+ (1−e−δτ )Sk +
δτ

Ik+1 = Ik e−δτ + (1−e−δτ )Sk+1 −
= Ik e−δτ

(260)
(261)

Here the simplest way to calculate δτ is
χk +χk+1
δx ;
2

(262)

δτ =

√

χk χk+1 δx

(263)

δτ =

χk+1 − χk
δx
ln χlnk+1
χk

(264)

δτ =
more accurate alternatives are
or

H.7.2 Angular integration
√
Table 18: Sums 4πYlm (θi , φi ) for special sets of directions. For all degrees and orders up to l = 8 not
mentioned in this table, the sums are 0. The label ‘Non-h. f-d.’ stands for ‘non-horizontal face-diagonals’,
i.e. the eight face diagonals that are not in the horizontal plane.
Y40

Y4±4

Y60

21
2
21
−
4
28
−
3

3√
70
4
3√
−
70
8
2√
70
−
3

3√
13
4
39 √
−
13
16
16 √
13
9

√
−2 5

39
4
9
2

3√
70
8
4√
70
3

√
2 5

6

0

Y00

Y20

6

0

12

0

Space diag.

8

0

Non-h. f-d.

8

√
2 5

Coord. x, y

4

Coord. z

2

Directions
Coord.
Face diag.

−

19 √
13
16
5√
−
13
4
√
2 13

−

Y6±4

Y80

Y8±4

Y8±8

3√
182
8
39 √
182
32
8√
−
182
9

99 √
17
32
891 √
17
256
11 √
17
9

3√
2618
32
27 √
2618
256
1√
2618
27

3√
24310
64
27 √
24310
512
1√
24310
54

27 √
182
32
3√
−
182
8

611 √
17
256
35 √
17
32
√
2 17

51 √
2618
256
3√
2618
32

3 √
24310
512
3√
24310
64

0

0

−

0

For angular integration over the full solid angle, we make the ansatz
Z

4π

N

dω X
=
f (θ, φ)
wi f (θi , φi ) + RN .
4π
i=1

(265)

H.7

Radiative transfer

161

√
Table 18 shows the sums 4πYlm (θi , φi ) over special sets of directions (θi , φi ). Using these
numbers and requiring that angular integration is exact for l ≤ lmax , we find the following weights wi for different sets of directions (see also [1], §25.4.65):
1. Axes
Coordinate axes:

1/6

lmax = 3
2. Face diagonals
Face diagonals:

1/12

lmax = 3
3. Space diagonals
Space diagonals:

1/8

lmax = 3
4. Axes + face diagonals
Coordinate axes:
Face diagonals:

1/30
1/15

lmax = 5
5. Axes + space diagonals
Coordinate axes:
Space diagonals:

1/15
3/40

lmax = 5
6. Face + space diagonals
Face diagonals:
Space diagonals:

2/15
-3/40

lmax = 5
7. Axes, face + space diagonals
Coordinate axes:
Face diagonals:
Space diagonals:

1/21
4/105
9/280

lmax = 7
8. Axes, non-horizontal face diagonals
Coordinate axes x, y:
Coordinate axes z:
Non-hor. face diagonals:

1/10
1/30
1/15

lmax = 3
9. Axes, non-horizontal face diagonals, space diagonals

162

T HE P ENCIL C ODE
Coordinate axes x, y:
Coordinate axes z:
Non-hor. face diagonals:
Space diagonals:
lmax = 5

12/215
10/129
-14/645
171/1720

J. Startup and run-time parameters

163

I Switchable modules
Module

Description

hydro.f90

This module takes care of most of the things related to velocity.
Pressure, for example, is added in the energy (entropy) module.

gpu astaroth.f90

This module contains GPU related types and functions to be used w
ASTAROTH nucleus.

noentropy.f90

Calculates pressure gradient term for
polytropic equation of state p = constρΓ .

nogpu.f90

This module contains GPU related dummy types and functions.

nohydro.f90

no variable u: useful for kinematic dynamo runs.

nopower spectrum.f90

reads in full snapshot and calculates power spetrum of u

noyinyang.f90

This module contains Yin-Yang related dummy types and functions.

noyinyang mpi.f90

This module contains Yin-Yang related dummy types and functions.

power spectrum.f90

reads in full snapshot and calculates power spetrum of u

timestep.f90

Runge-Kutta time advance, accurate to order itorder.
At the moment, itorder can be 1, 2, or 3.

timestep strang.f90

Runge-Kutta time advance, accurate to order itorder.
At the moment, itorder can be 1, 2, or 3.

timestep subcycle.f90

This is a highly specified timestep module currently only working
together with the special module coronae.f90.

yinyang.f90

This module contains Yin-Yang related types and functions
which are incompatible with FORTRAN 95.

yinyang mpi.f90

This module contains Yin-Yang related types and functions
which are incompatible with FORTRAN 95.

J Startup and run-time parameters
J.1

List of startup parameters for ‘start.in’

The following table lists all (at the time of writing, September 2002) namelists used
in ‘start.in’, with the corresponding parameters and their default values (in square
brackets). Any variable referred to as a flag can be set to any nonzero value to switch
the corresponding feature on. Not all parameters are used for a given scenario. This
list is not necessarily up to date; also, in many cases it can only give an idea of the
corresponding initial state; to get more insight and the latest set of parameters, you
need to look at the code.
The value ε corresponds to 5 times the smallest number larger than zero. For single
precision, this is typically about ε ≈ 5 × 1.2×10−7 = 6×10−7 ; for double precision, ε ≈
10−15 .
Variable [default value]

Meaning

164

T HE P ENCIL C ODE

Namelist init pars

cvsid [’’]
ip [14]
xyz0 [(−π, −π, −π)],
Lxyz [(2π, 2π, 2π)],
lperi [(T,T,T)]

lprocz slowest [T]

lwrite ic [F]

lnowrite [F]

lwrite aux [F]

lwrite 2d [F]
lread oldsnap [F]

the svn identification string, which allows you to
keep track of the version of ‘start.in’.
(anti-)verbosity level: ip=1 produces lots of diagnostic output, ip=14 virtually none.

determine the geometry of the box. All three are vectors of the form (x-comp., y-comp., z-comp.); xyz0 describes the left (lower) corner of the box, Lxyz the box
size. lperi specifies whether a direction is considered
periodic (in which case the last point is omitted) or
not. In all cases, three ghost zones will be added.
if set to F, the ordering of processor numbers is
changed, so the z processors are now in the inner
loop. Since nprocy =4 is optimal (see Sect. 5.20.2),
you may want to put lprocz slowest =T when
nygrid>nzgrid.
if set T, the initial data are written into the file ‘VAR0’.
This is generally useful, but doing this all the time
uses up plenty of disk space.
if set T, all initialization files are written, including
the param.nml file, except ‘var.dat’. This option allows you to use old filevar.dat files, but updates all
other initialization files. This could be useful after
having changed the code and, in particular, when the
‘var.dat’ files will be overwritten by ‘remesh.csh’.
if set T, auxiliary variables (those calculated at each
step, but not evolved mathematically) to ‘var.dat’ after the evolved quantities.
if set T, only 2D-snapshots are written into VAR files
in the case of 2D-runs with nygrid = 1 or nzgrid = 1.
if set T, the old snapshot will be read in before producing (overwriting) initial conditions. For example,
if you just want to add a perturbation to the magnetic field, you’d give no initial condition for density
and velocity (so you keep the data from a hopefully
relaxed run), and just add whatever you need for the
magnetic field. In this connection you may want to
touch NOERASE, so as not to erase the previous data.

J.1 Startup parameters for ‘start.in’

lread oldsnap nomag [F]

lread oldsnap nopscalar [F]

lshift origin [F,F,F]

unit system [’cgs’]

unit length [1]

unit velocity [1]
unit density [1]
unit temperature [1]
random gen [system]

bcx [(’p’, ’p’, . . . )],
bcy [(’p’, ’p’, . . . )],
bcz [(’p’, ’p’, . . . )]
pretend lnTT [F]

165

if set T, the old snapshot from a non-magnetic run
will be read in before producing (overwriting) initial conditions. This allows one to let a hydrodynamic run relax before adding a magnetic field.
However, for this to work one has to modify manually ‘data/param.nml’ by adding an entry for MAGNETIC INIT PARS or PSCALAR INIT PARS . In
addition, for idl to read correctly after the first
restarted run, you must adjust the value of mvar in
‘data/dim.dat’
if set T, the old snapshot from a run without passive
scalar will be read in before producing (overwriting)
initial conditions. This allows one to let a hydrodynamic run relax before adding a passive scalar.
if set T for any or some of the three directions, the
mesh is shifted by 1/2 meshpoint in that or those directions so that the mesh goes through the origin.
you can set this character string to ’SI’, which
means that you can give physical dimensions in SI
units. The default is cgs units.
allows you to set the unit length. Suppose you want
the unit length to be 1 kpc, then you would say unit_length=’3e21’. (Of course, politically correct would
be to say unit_system=’SI’ in which case you say
unit_length=’3e19’.)
Example: if you want km/s you say unit_length=’1e5’.
Example: if you want your unit density to be
10−24 g/cm3 you say unit_density=’1e-24’.
Example: unit_temperature=’1e6’ if you want megaKelvin.
choose random number generator; currently valid
choices are
’system’ (your compiler’s generator),
’min_std’ (the ‘minimal standard’ generator ran0()
from ‘Numerical Recipes’),
’nr_f90’ (the Parker-Miller-Marsaglia generator
ran() from ‘Numerical Recipes for F90’).

boundary conditions. See Sect. 5.16 for a discussion
of where and how to set these.
selects ln T as fundamental thermodynamic variable
in the entropy module
Namelist hydro init pars

166

T HE P ENCIL C ODE

inituu [’zero’]

initialization of velocity. Currently valid choices are
‘zero’ (u = 0 ),
‘gaussian-noise’ (random, normally-distributed
ux ,uz ),
‘gaussian-noise-x’ (random, normally-distributed
ux ),
‘sound-wave’ (sound wave in x direction),
‘shock-tube’ (polytropic standing shock),
‘bullets’ (blob-like velocity perturbations),
‘Alfven-circ-x’ (circularly polarized Alfven wave
in x direction),
‘const-ux’ (constant x-velocity),
‘const-uy’ (constant y-velocity),
‘tang-discont-z’ (tangential discontinuity: velocity is directed along x, jump is at z = 0),
‘Fourier-trunc’ (truncated Fourier series),
‘up-down’ (flow upward in one spot, downward in
another; not solenoidal).

ampluu [0.]
widthuu [0.1]
urand [0.]

amplitude for some types of initial velocities.
width for some types of initial velocities.
additional random perturbation of u. If urand>0, the
perturbation is additive, ui 7→ ui + urand U[0.5,0.5] ; if
urand<0, it is multiplicative, ui 7→ ui × urand U[0.5,0.5] ; in
both cases, U[0.5,0.5] is a uniformly distributed random
variable on the interval [−0.5, 0.5].

uu left [0.],
uu right [0.]

needed for inituu=’shock-tube’.
Namelist density init pars

J.1 Startup parameters for ‘start.in’

167

initlnrho [’zero’]

initialization of density. Currently valid choices are
‘zero’ (ln ρ = 0),
‘isothermal’ (isothermal stratification),
‘polytropic_simple’ (polytropic stratification),
‘hydrostatic-z-2’ (hydrostatic vertical stratification for isentropic atmosphere),
‘xjump’ (density jump in x of width widthlnrho ),
‘rho-jump-z’ (density jump in z of width
widthlnrho ),
‘piecew-poly’ (piecewise polytropic vertical stratification for solar convection),
‘polytropic’ (polytropic vertical stratification),
‘sound-wave’ (sound wave),
‘shock-tube’ (polytropic standing shock),
‘gaussian-noise’ (Gaussian-distributed, uncorrelated noise),
‘gaussian-noise’ (Gaussian-distributed, uncorrelated noise in x, but uniform in y and z),
‘hydrostatic-r’ (hydrostatic radial density stratification for isentropic or isothermal sphere),
‘sin-xy’ (sine profile in x and y),
‘sin-xy-rho’ (sine profile in x and y, but in ρ, not
ln ρ),
‘linear’ (linear profile in k · x),
‘planet’ (planet solution; see §C.7).

gamma [5./3]
cs0 [1.]

adiabatic index γ = cp /cv .
can be used to set the dimension of velocity; larger
values can be used to decrease stratification
reference values of sound speed and density, i. e. values at height zref .

rho0 [1.]
ampllnrho [0.],
widthlnrho [0.1]
rho left [1.],
rho right [1.]
cs2bot [1.],
cs2top [1.]

amplitude and width for some types of initial densities.
needed for initlnrho=’shock-tube’.
sound speed at bottom and top. Needed for some
types of stratification.
Namelist grav init pars

zref [0.]
gravz [−1.]
grav profile
[’const’]

z1 [0.],

reference height where in the initial stratification
c2s = c2s0 and ln ρ = ln ρ0 .
vertical gravity component gz .
constant gravity gz = gravz (grav_profile=’const’)
gravity or linear profile gz = gravz · z (grav_profile=’linear’, for accretion discs and similar).

168

z2 [1.]

nu epicycle [1.]

grav amp [0.], grav tilt [0.]

T HE P ENCIL C ODE
specific
to
the
solar
convection
case
initlnrho=’piecew-poly’. The stable layer is
z0 < z < z1 , the unstable layer z1 < z < z2 , and the
top (isothermal) layer is z2 < z < ztop .
vertical epicyclic frequency; for accretion discs it
should be equal to Omega, but not for galactic discs;
see Eq. (125) in Sect. C.5.
specific to the tilted gravity case (amplitude and angle wrt the vertical direction).
Namelist entropy init pars

initss [’nothing’]

initialization of entropy. Currently valid choices are
‘nothing’ (leaves the initialization done in the density module unchanged),
‘zero’ (put s = 0 explicitly; this may overwrite the
initialization done in the density module),
‘isothermal’ (isothermal stratification, T = const),
‘isobaric’ (isobaric, p = const),
‘isentropic’ (isentropic with superimposed hot [or
cool] bubble),
‘linprof’ (linear entropy profile in z),
‘piecew-poly’ (piecewise polytropic stratification
for convection),
‘polytropic’ (polytropic stratification, polytropic
exponent is mpoly0 ),
‘blob’ (puts a gaussian blob in entropy for buoyancy experiments; see Ref. [5] for details)
‘xjump’ (jump in x direction),
‘hor-tube’ (horizontal flux tube in entropy, oriented in the y-direction).

pertss [’zero’]

additional perturbation to entropy. Currently valid
choices are
’zero’ (no perturbation)
’hexagonal’ (hexagonal perturbation for convection).

ampl ss [0.],
widthss [2ε]
grads0 [0.]
radius ss [0.1]
mpoly0 [1.5],
mpoly1 [1.5],

amplitude and width for some types of initial entropy.
initial entropy gradient for initss=linprof.
radius of bubble for initss=isentropic.

J.1 Startup parameters for ‘start.in’

mpoly2 [1.5]

isothtop [0]
khor ss [1.]

169

specific
to
the
solar
convection
case
initss=piecew-poly: polytropic indices of unstable
(mpoly0 ), stable (mpoly1 ) and top layer (mpoly2 ). If
the flag isothtop is set, the top layer is initialized to
be isothermal, otherwise thermal (plus hydrostatic)
equilibrium is assumed for all three layers, which
results in a piecewise polytropic stratification.
flag for isothermal top layer for initss=piecew-poly.
horizontal wave number for pertss=hexagonal
Namelist magnetic init pars

initaa [’zero’]

initialization of magnetic field (vector potential).
Currently valid choices are
‘Alfven-x’ (Alfvén wave traveling in the xdirection; this also sets the velocity),
‘Alfven-z’ (Alfvén wave traveling in the zdirection; this also sets the velocity),
‘Alfvenz-rot’ (same as ‘Alfven-z’, but with rotation),
‘Alfven-circ-x’ (circularly polarized Alfven wave
in x direction),
‘Beltrami-x’ (x-dependent Beltrami wave),
‘Beltrami-y’ (y-dependent Beltrami wave),
‘Beltrami-z’ (z-dependent Beltrami wave),
‘Bz(x)’ (Bz ∝ cos(kx)),
‘crazy’ (for testing purposes).
‘diffrot’ ([needs to be documented]),
‘fluxrings’ (two interlocked magnetic fluxrings;
see § C.4),
‘gaussian-noise’ (white noise),
‘halfcos-Bx’ ([needs to be documented]),
‘hor-tube’ (horizontal flux tube in B, oriented in
the y-direction).
‘hor-fluxlayer’ (horizontal flux layer),
‘mag-support’ ([needs to be documented]),
‘mode’ ([needs to be documented]),
‘modeb’ ([needs to be documented]),
‘propto-ux’ ([needs to be documented]),
‘propto-uy’ ([needs to be documented]),
‘propto-uz’ ([needs to be documented]),
‘sinxsinz’ (sin x sin z),
‘uniform-Bx’ (uniform field in x direction),
‘uniform-By’ (uniform field in y direction),
‘uniform-Bz’ (uniform field in z direction),
‘zero’ (zero field),

170

T HE P ENCIL C ODE

initaa2 [’zero’]

additional perturbation of magnetic field. Currently
valid choices are
‘zero’ (zero perturbation),
‘Beltrami-x’ (x-dependent Beltrami wave),
‘Beltrami-y’ (y-dependent Beltrami wave),
‘Beltrami-z’ (z-dependent Beltrami wave).

amplaa [0.]
amplaa2 [0.]

amplitude for some types of initial magnetic fields.
amplitude for some types of magnetic field perturbation.

fring{1,2} [0.],
Iring{1,2} [0.],
Rring{1,2} [1.],
wr{1,2} [0.3]
radius [0.1]
epsilonaa [10−2 ]
widthaa [0.5]
z0aa [0.]
kx aa [1.],
ky aa [1.],
kz aa [1.]
lpress equil [F]

flux, current, outer and inner radius of flux ring 1/2;
see Sect. C.4.
used by some initial fields.
used by some initial fields.
used by some initial fields.
used by some initial fields.

wavenumbers used by some initial fields.
flag for pressure equilibrium (can be used in connection with all initial fields)
Namelist pscalar init pars

initlncc [’zero’]

initialization of passive scalar (concentration per
unit mass, c). Currently valid choices (for ln c) are
‘zero’ (ln c = 0.),
‘gaussian-noise’ (white noise),
‘wave-x’ (wave in x direction),
‘wave-y’ (wave in y direction),
‘wave-z’ (wave in z direction),
‘tang-discont-z’ (Kelvin-Helmholtz instability),
‘hor-tube’ (horizontal tube in concentration; used
as a marker for magnetic flux tubes).

initlncc2 [’zero’]

additional perturbation of passive scalar concentration c. Currently valid choices are
‘zero’ (δ ln c = 0.),
‘wave-x’ (add x-directed wave to ln c).

ampllncc [0.1]
ampllncc2 [0.]

amplitude for some types of initial concentration.
amplitude for some types of concentration perturbation.

kx lncc [1.],
ky lncc [1.],
kz lncc [1.]

wave numbers for some types of initial concentration.
Namelist shear init pars

J.2 Runtime parameters for ‘run.in’

qshear [0.]

171

degree of shear for shearing-box simulations (the
shearing-periodic boundaries are the x-boundaries
and are sheared in the y-direction). The shear velocity is U = −qΩx ŷ.

Namelist particles ads init pars

init ads mol frac [0.]

initial adsorbed mole fraction
Namelist particles surf init pars

init surf mol frac [0.]

initial surface mole fraction
Namelist particles chem init pars

total carbon sites [1.08e − 8] carbon sites per surface area [mol/cm]2
Namelist particles stalker init pars

dstalk [0.1]
lstalk xx [F]
lstalk vv [F]
lstalk uu [F]
lstalk guu [F]
lstalk rho [F]
lstalk grho [F]
lstalk ap [F]
lstalk bb [T]
lstalk relvel [F]

J.2

times between printout of stalker data
particles position
particles velocity
gas velocity at particles position
gas velocity gradient at particles position
gas density at particles position
gas density gradient at particles position
particles diameter
magnetic field at particles position
particles relative velocity to gas

List of runtime parameters for ‘run.in’

The following table lists all (at the time of writing, September 2002) namelists used
in file ‘run.in’, with the corresponding parameters and their default values (in square
brackets). Default values marked as [start] are taken from ‘start.in’. Any variable referred to as a flag can be set to any nonzero value to switch the corresponding feature
on. Not all parameters are used for a given scenario. This list is not necessarily up to
date; also, in many cases it can only give an idea of the corresponding setup; to get more
insight and the latest set of parameters, you need to look at the code.
Once you have changed any of the ‘*.in’ files, you may want to first execute the command
pc_configtest in order to test the correctness of these configuration files, before you
apply them in an active simulation run.
Variable [default value]

Meaning
Namelist run pars

cvsid [’’]
ip [14]
nt [0]

svn identification string, which allows you to keep
track of the version of ‘run.in’.
(anti-)verbosity level: ip=1 produces lots of additional diagnostic output, ip=14 virtually none.
number of time steps to run. This number can be
increased or decreased during the run by touch
RELOAD.

172

it1 [10]
it1d [it1]
cdt [0.4]
cdtv [0.08]
dt [0.]
dtmin [10−6 ]
tmax [1033 ]

isave [100]
itorder [3]
dsnap [100.]

dvid [100.]

iwig [0]

ix [−1], iy [−1], iz [−1], iz2 [−1]

slice position [’p’]

zbot slice [value]

ztop slice [value]

T HE P ENCIL C ODE
write diagnostic output every it1 time steps (see
Sect. 5.5).
write averages every it1d time steps (see Sect. 5.8.1).
it1d has to be greater than or equal to it1 .
Courant coefficient for advective time step; see §5.15.
Courant coefficient for diffusive time step; see §5.15.
time step; if 6= 0., this overwrites the Courant time
step. See §5.15 for a discussion of the latter.
abort if time step δt < δtmin .
don’t run time steps beyond this time. Useful if you
want to run for a given amount of time, but don’t
know the necessary number of time steps.
update current snapshot ‘var.dat’ every isave time
steps.
order of time step (1 for Euler; 2 for 3nd-order, 3 for
3rd-order Runge–Kutta).
save permanent snapshot every dsnap time units
to files ‘VARN ’, where N counts from N = 1
upward. (This information is stored in the file
‘data/tsnap.dat’; see the module wsnaps.f90 , which
in turn uses the subroutines out1 and out2 ).
write two-dimensional sections for generation of
videos every dvid time units (not timesteps; see the
subroutines out1 and out2 in the code).
if 6= 0, apply a Nyquist filter (a filter eliminating any
signal at the Nyquist frequency, but affecting large
scales as little as possible) every iwig time steps to
logarithmic density (sometimes necessary with convection simulations).
position of slice planes for video files. Any negative
value of some of these variables will be overwritten
according to the value of slice position . See § 5.7) for
details.
symbolic specification of slice position. Currently
valid choices are
’p’ (periphery of the box)
’m’ (middle of the box)
’e’ (equator for half-sphere calculations, i. e. x, y
centered, z bottom)
These settings are overridden by explicitly setting ix ,
iy , iz or iz2 . See § 5.7) for details.
z position of slice xy-plane. The value can be any float
number inside the z domain. These settings are overridden by explicitly setting ix , iy , iz or iz2 . Saved as
slice with the suffix xy . See § 5.7) for details.
z position of slice xy-plane. The value can be any float
number inside the z domain. These settings are overridden by explicitly setting ix , iy , iz or iz2 . Saved as
slice with the suffix xy2 . See § 5.7) for details.

J.2 Runtime parameters for ‘run.in’

tavg [0]

idx tavg [(0, 0, . . . , 0)]
d2davg [100.]
ialive [0]

bcx [(’p’, ’p’, . . . )],
bcy [(’p’, ’p’, . . . )],
bcz [(’p’, ’p’, . . . )]
random gen [start]
lwrite aux [start]

173

averaging time τavg for time averages (if 6= 0); at the
same time, time interval for writing time averages.
See § 5.8.4 for details.
indices of variables to time-average. See § 5.8.4 for
details.
time interval for azimuthal and z-averages, i.e. the
averages that produce 2d data. See § 5.8.3 for details.
if 6= 0, each processor writes the current time step
to ‘alive.info’ every ialive time steps. This provides
the best test that the job is still alive. (This can be
used to find out which node has crashed if there is a
problem and the run is hanging.)

boundary conditions. See Sect. 5.16 for a discussion
of where and how to set these.
see start parameters, p. 165
if set T, auxiliary variables (those calculated at each
step, but not evolved mathematically) to ‘var.dat’
and ‘VAR’ files after the evolved quantities.
Namelist hydro run pars

Omega [0.]

theta [0.]

ttransient [0.]

dampu [0.],
tdamp [0.],
ldamp fade [F]

dampuint [0.],
dampuext [0.],
rdampint [0.],
rdampext [impossible],

magnitude of angular velocity for Coriolis force
(note: the centrifugal force is turned off by default,
unless lcentrifugal_force=T is set).
direction of angular velocity in degrees (θ = 0 for zdirection, θ = 90 for the negative x-direction, corresponding to a box located at the equator of a rotating
sphere. Thus, e.g., θ = 60 corresponds to 30◦ latitude.
(Note: prior to April 29, 2007, there was a minus sign
in the definition of θ.)
initial time span for which to do something special
(transient). Currently just used to smoothly switch
on heating [Should be in run pars , rather than here].

damp motions during the initial time interval 0 < t <
tdamp with a damping term −dampu (u). If ldamp fade is set, smoothly reduce damping to zero over
the second half of the time interval tdamp . Initial velocity damping is useful for situations where initial
conditions are far from equilibrium.
weighting of damping external to spherical region
(see wdamp , damp u below).
weighting of damping in internal spherical region
(see wdamp , damp u below).
radius of internal damping region
radius of external damping region, used in place of
former variable rdamp

174

wdamp [0.2],

ampl forc [0.],

k forc [0.],
w forc [0.]

T HE P ENCIL C ODE
permanently damp motions in |x| < rdampint
with damping term −damp u int u χ(r−rdampint )
or |x|
>
rdampext with damping term
−damp u ext u χ(r−rdampext ), where χ(·) is a smooth
profile of width wdamp .
amplitude of the ux-forcing or uy-forcing on the vertical boundaries that is of the form u x(t) = ampl f orc∗
sin(k f orc ∗ x) ∗ cos(w f orc ∗ t) [must be used in
connection with bcx=’g’ or bcz=‘g’ and force lower bound=‘vel time’ or force upper bound=‘vel time’]
corresponding horizontal wavenumber
corresponding frequency
Namelist density run pars

cs0 [start],
rho0 [start],
gamma [start]
cdiffrho [0.]
cs2bot [start],
cs2top [start]
lupw lnrho [.false.]

see start parameters, p. 167
Coefficient for mass diffusion (diffusion term will be
cdiffrho δx cs0 .
squared sound speed at bottom and top for boundary
condition ‘c2’.
use 5th-order upwind derivative operator for the advection term u · ∇ ln ρ to avoid spurious Nyquist signal (‘wiggles’); see §H.2.

Namelist entropy run pars

hcond0 [0.],
hcond1 [start],
hcond2 [start]

iheatcond [’K-const’]

lcalc heatcond constchi [F]

specific
to
the
solar
convection
case
initss=piecew-poly: heat conductivities K in
the individual layers. hcond0 is the value Kunst in
the unstable layer, hcond1 is the ratio Kstab /Kunst for
the stable layer, and hcond2 is the ratio Ktop /Kunst
for the top layer. The function K(z) is not discontinuous, as the transition between the different values
is smoothed over the width widthss . If hcond1 or
hcond2 are not set, they are calculated according to
the polytropic indices of the initial profile, K ∝ m+1.
select type of heat conduction. Currently valid
choices are
‘K-const’ (constant heat conductivity),
‘K-profile’ (vertical or radial profile),
‘chi-const’ (constant thermal diffusivity),
‘magnetic’ (heat conduction by electrons in magnetic field – currently still experimental).
flag for assuming thermal diffusivity χ = K/(cp ρ) =
const, rather than K = const (which is the
default). This is currently only correct with
‘noionization.f90’. Superseded by iheatcond .

J.2 Runtime parameters for ‘run.in’

chi [0.]
widthss [start]
isothtop [start]
luminosity [0.],
wheat [0.1]
cooltype [’Temp’]

175

value of χ when lcalc_heatcond_constchi=T.
width of transition region between layers. See start
parameters, p. 169.
flag for isothermal top layer for solar convection case.
See start parameters, p. 169.
strength and width of heating region.
type of cooling; currently only implemented for spherical geometry. Currently valid choices are
‘Temp’,‘cs2’ (cool temperature toward c2s
=
cs2cool ) with a cooling term
−C = −ccool

c2s − c2s cool
c2s cool

)
‘Temp-rho’,cs2-rho (cool temperature toward c2s =
cs2cool ) with a cooling term
−C = −ccool ρ

c2s − c2s cool
c2s cool

— this avoids numerical instabilities in lowdensity regions [currently, the cooling coefficient ccool ≡cool is not taken into account when
the time step is calculated])
‘entropy’ (cool entropy toward 0.).

cool [0.],
wcool [0.1]
rcool [1.]
Fbot [start]

chi t [0.]

lupw ss [.false.]

tauheat buffer [0.]
zheat buffer [0.]
dheat buffer1 [0.]
TTheat buffer [0.]

strength ccool and smoothing width of cooling region.
radius of cooling region: cool for |x| ≥ rcool .
heat flux for bottom boundary condition ‘c1’. For
polytropic atmospheres, if Fbot is not set, it will be
calculated from the value of hcond0 in ‘start.x’, provided the entropy boundary condition is set to ‘c1’.
entropy diffusion coefficient for diffusive term
∂s/∂t = . . . + χt ∇2 s in the entropy equation, that can
represent some kind of turbulent (sub-grid) mixing.
It is probably a bad idea to combine this with heat
conduction hcond0 6= 0.
use 5th-order upwind derivative operator for the advection term u · ∇s to avoid spurious Nyquist signal
(‘wiggles’); see §H.2.
time scale for heating to target temperature
(=TTheat buffer ); zero disables the buffer zone.
z coordinate of the thermal buffer zone. Buffering is
active in |z| >TTheat buffer .
Inverse thickness of transition to buffered layer.
target temperature in thermal buffer zone (z direction only).

176

lhcond global [F]

T HE P ENCIL C ODE
flag for calculating the heat conductivity K (and
also ∇ log K) globally using the global arrays facility.
Only valid when iheatcond =‘K-profile’.
Namelist magnetic run pars

B ext [(0., 0., 0.)]

lignore Bext in b2 [F]
or luse Bext in b2 [T]
eta [0.]
height eta [0.],
eta out [0.]
eta int [0.]
eta ext [0.]
kinflow [’’]

kx [1.],
ky [1.],
kz [1.]
ABC A [1.],
ABC B [1.],
ABC C [1.]

uniform background magnetic field (for fully periodic
boundary conditions, uniform fields need to be explicitly added, since otherwise the vector potential A has
a linear x-dependence which is incompatible with periodicity).
add uniform background magnetic field when
computing b2 pencils
magnetic diffusivity η = 1/(µ0 σ), where σ is the electric conductivity.
used to add extra diffusivity in a halo region.
used to add extra diffusivity inside sphere of radius
r int .
used to add extra diffusivity outside sphere of radius
r ext .
set type of flow fixed with ‘nohydro’. Currently the
only recognized value is ’ABC’ for an ABC flow; all
other values lead to u = 0.

wave numbers for ABC flow.

amplitudes A, B and C for ABC flow.
Namelist pscalar run pars

pscalar diff [0.]
tensor pscalar diff [0.]
reinitialize lncc [F]

diffusion for passive scalar concentration c.
coefficient for non-isotropic diffusion of passive
scalar.
reinitialize the passive scalar to the value of cc const
in start.in at next run
Namelist forcing run pars

iforce [2]

select form of forcing in the equation of motion; currently valid choices are
’zero’ (no forcing),
’irrotational’ (irrotational forcing),
’helical’ (helical forcing),
’fountain’ (forcing of “fountain flow”; see Ref. [11]),
’horizontal-shear’ (forcing localized horizontal sinusoidal shear).
’variable_gravz’ (time-dependent vertical gravity
for forcing internal waves),

iforce2 [0]

select form of additional forcing in the equation of
motion; valid choices are as for iforce .

J.2 Runtime parameters for ‘run.in’

force [0.]
relhel [1.]

height ff [0.]
r ff [0.]

width ff [0.5]
kfountain [5]
fountain [1.]
omega ff [1.]
ampl ff [1.]

177

amplitude of forcing.
helicity of forcing. The parameter relhel corresponds
to σ introduced in Sect. G.2. (σ = ±1 corresponds to
maximum helicity of either sign).
multiply forcing by z-dependent profile of width
height ff (if 6= 0) .
if 6= 0, multiply forcing by spherical cutoff profile (of
radius r ff ) and flip signs of helicity at equatorial
plane.
width of vertical and radial profiles for modifying
forcing.
horizontal wavenumber of the fountain flow.
amplitude of the fountain flow.
frequency of the cos or sin forcing [e.g. cos(omega ff*t)].
amplitude of forcing in front of cos or sin [e.g. ampl ff*cos(omega ff*t)].
Namelist grav run pars

zref [start],
gravz [start],
grav profile [start]
nu epicycle [start]

see p. 167.
see Eq. (125) in Sect. C.5.
Namelist viscosity run pars

nu [0.]
nu hyper2 [0.]
nu hyper3 [0.]
zeta [0.]
ivisc [’nu-const’]

kinematic viscosity.
kinematic hyperviscosity (with ∇4 u).
kinematic hyperviscosity (with ∇6 u).
bulk viscosity.
select form of viscous term (see §6.2); currently valid
choices are
’nu-const’ – viscous force for ν = const, F visc =
ν(∇2 u + 31 ∇∇ · u + 2S · ∇ ln ρ)
’rho_nu-const’ – viscous force for µ ≡ ρν = const,
F visc = (µ/ρ)(∇2 u + 13 ∇∇ · u). With this option, the input parameter nu actually sets the
value of µ/ρ0 (rho0 =ρ0 is another input parameter, see pp. 167 and 174)
’simplified’ – simplified viscous force F visc =
ν∇2 u
Namelist shear run pars

qshear [start]

See p. 171.
Namelist particles run pars

ldragforce dust par [F]
ldragforce gas par [F]
ldraglaw steadystate [F]
lpscalar sink [F]
pscalar sink rate [0]

dragforce on particles
particle-gas friction force
particle forces only with τ1 ∆v
particles consume passive scalar
volumetric pscalar consumption rate

178

T HE P ENCIL C ODE

lbubble [F]

addition of the virtual mass term
Namelist particles ads run pars

placeholder [start]

placeholder
Namelist particles surf run pars

lspecies transfer [T]

Species transfer from solid to fluid phase
Namelist particles chem run pars

lthiele [T]

J.3

Modeling of particle porosity by application of Thiele
modulus

List of parameters for ‘print.in’

The following table lists all possible inputs to the file ‘print.in’ that are documented.
Variable

Meaning
Module ‘cdata.f90’

it
t
dt
walltime
Rmesh
Rmesh3
maxadvec

number of time step (since beginning of job only)
time t (since start.csh)
time step δt
wall clock time since start of run.x, in seconds
Rmesh
(3)
Rmesh
maxadvec

u2tm

D

uotm
outm
fkinzm
u2m
uxpt
uypt
uzpt
uxp2
uyp2
uzp2
urms
urmsx
urmsz
durms
umax
umin
uxrms
uyrms

Module ‘hydro.f90’
E
Rt
u(t) · 0 u(t′ )dt′
D
E
Rt
u(t) · 0 ω(t′ )dt′
E
D
Rt
′
′
ω(t) · 0 u(t )dt
1
̺u2 uz
2
hu2 i
ux (x1 , y1 , z1 , t)
uy (x1 , y1 , z1 , t)
uz (x1 , y1 , z1 , t)
ux (x2 , y2 , z2 , t)
uy (x2 , y2 , z2 , t)
uz (x2 , y2 , z2 , t)
1/2
hu2 i
1/2
hu2 i for the hydro xaver range
1/2
hu2 i for the hydro zaver range
1/2
hδu2 i
max(|u|)
min(|u|)
1/2
hu2x i
1/2
u2y

J.3 Parameters for ‘print.in’

uzrms
uxmin
uymin
uzmin
uxmax
uymax
uzmax
uxm
uym
uzm
ux2m
uy2m
uz2m
ux2ccm
ux2ssm
uy2ccm
uy2ssm
uxuycsm
uxuym
uxuzm
uyuzm
umx
umy
umz
omumz
umamz
umbmz
umxbmz
rux2m
ruy2m
ruz2m
divum
rdivum
divu2m
gdivu2m
u3u21m
u1u32m
u2u13m
u2u31m
u3u12m
u1u23m
ruxm
ruym
ruzm
ruxtot
rumax

1/2

hu2z i
min(|ux |)
min(|uy |)
min(|uz |)
max(|ux |)
max(|uy |)
max(|uz |)
hux i
huy i
huz i
hu2x i
u2y
hu2z i
hu2x cos2 kzi
u2x sin2 kz
u2y cos2 kz
u2y sin2 kz
hux uy cos kz sin kzi
hux uy i
hux uz i
huy uz i
hux i
huy i
hu
D zi
E
hW ixy · hU ixy
tion)
D
E
huixy · hAixy

E
hU ixy · hBixy
tion)
D
E
hU ixy × hBixy
D

(xy-averaged mean cross helicity produc-

hρu2x i
ρu2y
hρu2z i

(xy-averaged mean cross helicity produc-

z

(xy-averaged mean emf)

hdivu)i
h̺divu)i
h(divu)2 i
h(grad divu)2 i
hu3 u2,1 i
hu1 u3,2 i
hu2 u1,3 i
hu2 u3,1 i
hu3 u1,2 i
hu1 u2,3 i
h̺ux i (mean x-momentum density)
h̺uy i (mean y-momentum density)
h̺uz i (mean z-momentum density)
hρ|u|i (mean absolute x-momentum density)
max(̺|u|) (maximum modulus of momentum)

179

180

T HE P ENCIL C ODE

ruxuym
ruxuzm
ruyuzm
divrhourms
divrhoumax
rlxm
rlym
rlzm
rlx2m
rly2m
rlz2m
tot ang mom
dtu
oum
ou int
fum
odel2um
o2m
orms
omax
ox2m
oy2m
oz2m
oxuzxm
oyuzym
oxoym
oxozm
oyozm
qfm
q2m
qrms
qmax
qom
quxom
pvzm
oumphi
ugurmsx
ugu2m
dudx
Marms
Mamax
ekin
ekintot
uxglnrym
uyglnrxm
uzdivum
uxuydivum

h̺ux uy i (mean Reynolds stress)
h̺ux uz i (mean Reynolds stress)
h̺uy uz i (mean Reynolds stress)
|∇ · (̺u)|rms
|∇ · (̺u)|max
hρyuz − zuy i
hρzux − xuz i
hρxuy − yux i
h(ρyuz − zuy )2 i
h(ρzux − xuz )2 i
h(ρxuy − yux )2 i
Total angular momentum in spherical coordinates about the
axis.
δt/[cδt δx/ max |u|] (time step relative to advective time step;
see § 5.15)
hω
R · ui
ω · u dV
V
hf · ui
hω∇2 ui
hω 2 i ≡ h(∇ × u)2 i
1/2
hω 2 i
max(|ω|)
hωx2 i
ωy2
hωz2 i
hωx uz,x i
hωy uz,y i
hωx ωy i
hωx ωz i
hωy ωz i
hq · f i
hq 2 i
1/2
hq 2 i
max(|q|)
hq · ωi
hq · (u × ω)i
hωz + 2Ω/̺i (z component of potential vorticity)
hω · uiϕ
(u∇u)2
hu∇ui2
δu
δx

1/2

for the hydro xaver range

hu2 /c2s i (rms Mach number)
max |u|/cs (maximum Mach number)
1
2
R 2 ̺u
1
̺u2 dV
V 2
hux ∂y ln ̺i
huy ∂x ln ̺i
huz ∇ · ui
hux uy ∇ · ui

J.3 Parameters for ‘print.in’

divuHrms
uxxrms
uyyrms
uxzrms
uyzrms
uzyrms
dtF
udpxxm

(∇H · uH )rms
urms
x,x
urms
y,y
urms
x,z
urms
y,z
urms
z,y
δt/[cδt δx/ max
R |F|] (time step relative to max force time step;
see § 5.15) ur (θ, φ)Yℓm (θ, φ) sin(θ)dθdφ
components of symmetric tensor hui ∂j p + uj ∂i pi
Module ‘density.f90’

rhom
rhomxmask
rhomzmask
drho2m
drhom
rhomin
rhomax
ugrhom
uglnrhom
totmass
mass
vol
grhomax

h̺i (mean density)
h̺i for the density xaver range
h̺i for the density zaver range
< (̺ − ̺0 )2 >
< ̺ − ̺0 >
min(ρ)
max(ρ)
hu · ∇̺i
hu
R · ∇ ln ̺i
R ̺ dV
R ̺ dV
dV (volume)
max(|∇̺|)

Module ‘entropy.f90’

dtc
ethm
ssruzm
ssuzm
ssm
ss2m
eem
ppm
csm
pdivum
fradbot
fradtop
TTtop
ethtot
dtchi
Hmax
tauhmin
dtH
yHm
yHmax
TTm

δt/[cδt δx / max cs ] (time step relative to acoustic time step;
see § 5.15)
h̺ei (mean thermal [=internal] energy)
hs̺uz /cp i
hsuz /cp i
hs/cp i (mean entropy)
h(s/cp )2 i (mean squared entropy)
hei
hpi
hcs i
hp∇
R · ui
R Fbot · dS
R Ftop · dS
R Ttop dS
̺e dV (total thermal [=internal] energy)
V
δt/[cδt,v δx2 /χmax ] (time step relative to time step based on
heat conductivity; see § 5.15)
Hmax (net heat sources summed see § 5.15)
Hmax (net heat sources summed see § 5.15)
δt/[cδt,s cv T /Hmax ] (time step relative to time step based on
heat sources; see § 5.15)
mean hydrogen ionization
max of hydrogen ionization
hT i

181

182

T HE P ENCIL C ODE

TTmax
TTmin
gTmax
ssmax
ssmin
gTrms
gsrms
gTxgsrms
fconvm
ufpresm
Kkramersm

Tmax
Tmin
max(|∇T |)
smax
smin
(∇T )rms
(∇s)rms
(∇T × ∇s)rms
hcp ̺uz T i
h−u/ρ∇pi
hKkramers i

Module ‘magnetic.f90’

eta tdep
ab int
jb int
b2tm
bjtm
jbtm
b2ruzm
b2uzm
ubbzm
b1m
b2m
b4m
bm2
j2m
jm2
abm
abumx
abumy
abumz
abmh
abmn
abms
abrms
jbrms
ajm
jbm
jbmh
jbmn
jbms
ubm
dubrms
dobrms
uxbxm
uybxm
uzbxm

t-dependent
η
R
R A · B dV
E
D j · BRdV
t
b(t) · 0 b(t′ )dt′
D
E
Rt
b(t) · 0 j(t′ )dt′
D
R t ′ ′E
j(t) · 0 b(t )dt
B 2 ρuz
B 2 uz
h(u · B)Bz i
h|B|i
B2
B4
max(B 2 )
j2
max(j 2 )
hA · Bi
hux A · Bi
huy A · Bi
huz A · Bi
hA · Bi (temp)
hA · Bi (north)
hA · Bi (south)
1/2
h(A · B)2 i
1/2
h(j · B)2 i
hj · Ai
hj · Bi
hJ · Bi (temp)
hJ · Bi (north)
hJ · Bi (south)
hu · Bi
1/2
h(u − B)2 i
1/2
h(ω − B)2 i
hux Bx i
huy Bx i
huz Bx i

J.3 Parameters for ‘print.in’

uxbym
uybym
uzbym
uxbzm
uybzm
uzbzm
cosubm
jxbxm
jybxm
jzbxm
jxbym
jybym
jzbym
jxbzm
jybzm
jzbzm
uam
ujm
fbm
fxbxm
epsM
epsAD
bxpt
bypt
bzpt
jxpt
jypt
jzpt
Expt
Eypt
Ezpt
axpt
aypt
azpt
bxp2
byp2
bzp2
jxp2
jyp2
jzp2
Exp2
Eyp2
Ezp2
axp2
ayp2
azp2
exabot
exatop
emag
brms

hux By i
huy By i
huz By i
hux Bz i
huy Bz i
huz Bz i
hU · B/(|U | |B|)i
hjx Bx i
hjy Bx i
hjz Bx i
hjx By i
hjy By i
hjz By i
hjx Bz i
hjy Bz i
hjz Bz i
hu · Ai
hu · J i
hf · Bi
hfx Bx i
ηµ0 j 2
hρ−1 tAD (J × B)2 i (heating by ion-neutrals friction)
Bx (x1 , y1 , z1 , t)
By (x1 , y1 , z1 , t)
Bz (x1 , y1 , z1 , t)
Jx (x1 , y1 , z1 , t)
Jy (x1 , y1 , z1 , t)
Jz (x1 , y1 , z1 , t)
Ex (x1 , y1 , z1 , t)
Ey (x1 , y1 , z1 , t)
Ez (x1 , y1 , z1 , t)
Ax (x1 , y1 , z1 , t)
Ay (x1 , y1 , z1 , t)
Az (x1 , y1 , z1 , t)
Bx (x2 , y2 , z2 , t)
By (x2 , y2 , z2 , t)
Bz (x2 , y2 , z2 , t)
Jx (x2 , y2 , z2 , t)
Jy (x2 , y2 , z2 , t)
Jz (x2 , y2 , z2 , t)
Ex (x2 , y2 , z2 , t)
Ey (x2 , y2 , z2 , t)
Ez (x2 , y2 , z2 , t)
Ax (x2 , y2 , z2 , t)
Ay (x2 , y2 , z2 , t)
A
R z (x2 , y2 , z2 , t)
R E × A dS|bot
R E 1× A2 dS|top
B dV
V 2µ0
B2

1/2

183

184

T HE P ENCIL C ODE

bfrms
bf2m
bf4m
bmax
bxmin
bymin
bzmin
bxmax
bymax
bzmax
bbxmax
bbymax
bbzmax
jxmax
jymax
jzmax
jrms
hjrms
jmax
vArms
vAmax
dtb

D
D
D

B′

2

B′

2

′4

E1/2
E
E

bmy

B
max(|B|)
min(|Bx |)
min(|By |)
min(|Bz |)
max(|Bx |)
max(|By |)
max(|Bz |)
max(|Bx |)excludingBvext
max(|By |)excludingBvext
max(|Bz |)excludingBvext
max(|jvx |)
max(|jvy |)
max(|jvz |)
1/2
j2
1/2
j2
max(|j|)
1/2
B 2 /̺
max(B 2 /̺)1/2
δt/[cδt δx/vA,max ] (time step relative to Alfvén time step; see
§ 5.15)
δt/[cδt,v δx2 /ηmax ] (time step relative to resistive time step;
see § 5.15)
A2
1/2
A2
max(|A|)
h(∇ · A)2 i1/2
B 2 /(2µ0 p)
(mean inverse plasma beta)
2
max[B /(2µ0 p)] (maximum inverse plasma beta)
hβi
max β
min β
hBx i
hBy i
hBz i
hBx By i
D
E1/2
hBi2yz
(energy of yz-averaged mean field)

bmz

(energy of xy-averaged mean field)

dteta
a2m
arms
amax
divarms
beta1m
beta1max
betam
betamax
betamin
bxm
bym
bzm
bxbym
bmx

bmzS2
bmzA2
jmx

1/2

hBi2xz
D
E1/2
hBi2xy
E
D
hB S i2xy
E
D
2
hB A ixy
D
E1/2
2
hJ iyz

(energy of xz-averaged mean field)

(energy of yz-averaged mean current density)

J.3 Parameters for ‘print.in’

jmy
jmz
bmzph
bmzphe
bsinphz
bcosphz
emxamz3
embmz
ambmz
ambmzh
ambmzn
ambmzs
jmbmz
Rmmz
kx aa
kmz
bx2m
by2m
bz2m
uxbm
jxbm
magfricmax
b3b21m
b3b12m
b1b32m
b1b23m
b2b13m
b2b31m
uxbmx
uxbmy
uxbmz
jxbmx
jxbmy
jxbmz
examx
examy
examz
exjmx
exjmy
exjmz

1/2

(energy of xz-averaged mean current density)
hJ i2
D xz E1/2
hJ i2xy
(energy of xy-averaged mean current density)
Phase of a Beltrami field
Error of phase of a Beltrami field
sine of phase of a Beltrami field
cosine
of phaseEof a Beltrami field
D
D
)
D

hEixy × hAixy
E
hEixy · hBixy

(xy-averaged mean field helicity flux)

(xy-averaged mean field helicity production

E

hAixy · hBixy
field)
D
E

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, temp) E
D

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, north) E
D

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, south) E
D

(magnetic helicity of xy-averaged mean

D

hJ ixy · hBixy
E
|u×B |
|η J |
xy

(current helicity of xy-averaged mean field)

k
Dx
E D
E
hJ ixy · hBixy / hBi2xy
hBx2 i
By2
hBz2 i
hu × Bi · B 0 /B02
hj × Bi · B 0 /B02
hj × Bi · B 2
hB3 B2,1 i
hB3 B1,2 i
hB1 B3,2 i
hB1 B2,3 i
hB2 B1,3 i
hB2 B3,1 i
h(u × B)x i
h(u × B)y i
h(u × B)z i
h(j × B)x i
h(j × B)y i
h(j × B)z i
hE × Ai |x
hE × Ai |y
hE × Ai |z
hE × J i |x
hE × J i |y
hE × J i |z

185

186

T HE P ENCIL C ODE

dexbmx
dexbmy
dexbmz
phibmx
phibmy
phibmz
b2divum
jdel2am
ujxbm
jxbrmax
jxbr2m
bmxy rms
etasmagm
etasmagmin
etasmagmax
etavamax
etajmax
etaj2max
etajrhomax
cosjbm
jparallelm
jperpm
hjparallelm
hjperpm
brmsx
brmsz
Exmxy
Eymxy
Ezmxy

h∇ × E × Bi |x
h∇ × E × Bi |y
h∇ × E × Bi |z
hφBi |x
hφBi |y
hφBi |z
B2∇ · u
hJ · ∇2 A)i
hu · (J × B)i
max(|J × B/ρ|)
2
h(J
p × B/ρ) i
[hbx iz (x, y)]2 + [hby iz (x, y)]2 + [hbz iz (x, y)]2
Mean of Smagorinsky resistivity
Min of Smagorinsky resistivity
Max of Smagorinsky resistivity
Max of artificial resistivity η ∼ vA
√
Max of artificial resistivity η ∼ J/ ρ
Max of artificial resistivity η ∼ J 2 /ρ
Max of artificial resistivity η ∼ J/ρ
hJ · B/(|J | |B|)i
Mean value of the component of J parallel to B
Mean value of the component of J perpendicular to B
Mean value of the component of Jhyper parallel to B
Mean value of the component of Jhyper perpendicular to B
1/2
B2
for the magnetic xaver range
1/2
B2
for the magnetic zaver range
hEx iz
hEy iz
hEz iz
Module ‘pscalar.f90’

rhoccm
ccmax
ccglnrm
dtchi2
dtrad
dtspitzer
qmax
qrms

h̺ci
max(c)
hc∇z ̺i

Module ‘1D_loop.f90’

heatconduction
radiative loss from RTV
Spitzer heat conduction time step
max of heat flux vector
rms of heat flux vector
Module ‘advective_gauge.f90’

Lamm
Lampt
Lamp2
Lamrms
Lambzm
Lambzmz
gLambm

hΛi
Λ(x1, y1, z1)
Λ(x2, y2, z2)
1/2
hΛ2 i
hΛBz i
hΛBz ixy
hΛBi

J.3 Parameters for ‘print.in’

apbrms
jxarms
jxaprms
jxgLamrms
gLamrms
divabrms
divapbrms
d2Lambrms
d2Lamrms

1/2

h(A′ B)2 i
1/2
h(J × A)2 i
1/2
h(J × A′ )2 i
1/2
h(J × ∇Λ)2 i
1/2
h(∇Λ)2 i
1/2
h[(∇ · A)B]2 i
1/2
h[(∇ · A′ )B]2 i
1/2
h[(∇2 Λ)B]2 i
1/2
h[∇2 Λ]2 i

Module ‘anelastic.f90’

rhom
ugrhom
mass
divrhoum
divrhourms
divrhoumax

h̺i (mean density)
hu
R · ∇̺i
̺ dV
h∇ · (̺u)i
|∇ · (̺u)|rms
|∇ · (̺u)|max

Module ‘bfield.f90’

bmax
bmin
brms
bm
b2m
bxmax
bymax
bzmax
bxm
bym
bzm
bx2m
by2m
bz2m
bxbym
bxbzm
bybzm
dbxmax
dbymax
dbzmax
dbxm
dbym
dbzm
dbx2m
dby2m
dbz2m
jmax
jmin
jrms
jm

max B
min B
hB 2 i1/2
hBi
hB 2 i
max |Bx |
max |By |
max |Bz |
hBx i
hBy i
hBz i
hBx2 i
hBy2 i
hBz2 i
hBx By i
hBx Bz i
hBy Bz i
max |Bx − Bext,x |
max |By − Bext,y |
max |Bz − Bext,z |
hBx − Bext,x i
hBy − Bext,y i
hBz − Bext,z i
h(Bx − Bext,x )2 i
h(By − Bext,y )2 i
h(Bz − Bext,z )2 i
max J
min J
hJ 2 i1/2
hJi

187

188

T HE P ENCIL C ODE

j2m
jxmax
jymax
jzmax
jxm
jym
jzm
jx2m
jy2m
jz2m
divbmax
divbrms
betamax
betamin
betam
vAmax
vAmin
vAm

hJ 2 i
max |Jx |
max |Jy |
max |Jz |
hJx i
hJy i
hJz i
hJx2 i
hJy2 i
hJz2 i
max |∇ · B|
h(∇ · B)2 i1/2
max β
min β
hβi
max vA
min vA
hvA i

Module ‘chemistry.f90’

dtchem

dtchem
Module ‘chemistry_simple.f90’

dtchem

dtchem
Module ‘chiral_fluids.f90’

mu5m
mu5rms
bgmu5rms
bgtheta5rms
theta5m
theta5rms

hµ5 i
1/2
hµ25 i
1/2
h(B · ∇µ5 )2 i
1/2
h(B · ∇θ5 )2 i
hθ5 i
1/2
hθ52 i

Module ‘chiral_fluids_gradtheta.f90’

mu5m
mu5rms
bgmu5rms
bgtheta5rms
gtheta5rms
gmu5rms
gtheta5mx
gtheta5my
gtheta5mz

hµ5 i
1/2
hµ25 i
1/2
h(B · ∇µ5 )2 i
1/2
h(B · ∇θ5 )2 i
1/2
h(∇θ5 )2 i
1/2
h(∇µ5 )2 i
h∇θ5x i
h∇θ5y i
h∇θ5z i

Module ‘chiral_mhd.f90’

mu5m
mu5rms
gmu5rms
gmu5mx

hµ5 i
1/2
hµ25 i
1/2
h(∇µ5 )2 i
h∇µ5 ix

J.3 Parameters for ‘print.in’

gmu5my
gmu5mz
bgmu5rms
dt mu5 1
dt mu5 2
dt mu5 3
dt bb 1
dt chiral

h∇µ5 iy
h∇µ5 iz
1/2
h(B · ∇µ5 )2 i
min(µ5 /B 2 )δx/(λη)
(ληmin(B 2 ))−1
δx2 /D5
δx/(ηmax(µ5 ))
total time-step contribution from chiral MHD
Module ‘coronae.f90’

dtchi2
dtspitzer
dtrad

δt/[cδt,v δx2 /χmax ] (time step relative to time step based on
heat conductivity; see § 5.15)
Spitzer heat conduction time step
radiative loss from RTV
Module ‘cosmicray_current.f90’

ekincr
ethmcr

1
̺u2cr
2

h̺cr ecr i

Module ‘density_stratified.f90’

mass
rhomin
rhomax
drhom
drho2m
drhorms
drhomax

ρ d3 x
min |ρ|
max |ρ|
h∆ρ/ρ0 i
h(∆ρ/ρ0 )2 i
h∆ρ/ρ0 irms
max |∆ρ/ρ0 |
R

Module ‘detonate.f90’

detn
dettot

KKm
KK2m
MMxm
MMym
MMzm

Number of detonated sites (summed over time steps between
adjacent outputs)
Total energy input (summed over time steps between adjacent
outputs)
Module ‘dustdensity.f90’
P coag
P Tkcoag
P Tk x
P Mk,coag
My
P k,coag
Mzk,coag

Module ‘entropy_anelastic.f90’

dtc
ethm
ssm
ss2m
eem
ppm
csm
pdivum

δt/[cδt δx / max cs ] (time step relative to acoustic time step;
see § 5.15)
h̺ei (mean thermal [=internal] energy)
hs/cp i (mean entropy)
h(s/cp )2 i (mean squared entropy)
hei
hpi
hcs i
hp∇ui

189

190

T HE P ENCIL C ODE

fradbot
fradtop
TTtop
ethtot
dtchi
ssmxy
ssmxz

R
R Fbot · dS
R Ftop · dS
R Ttop dS
̺e dV (total thermal [=internal] energy)
V
δt/[cδt,v δx2 /χmax ] (time step relative to time step based on
heat conductivity; see § 5.15)
hsiz
hsiy
Module ‘gravitational_waves.f90’

hhT2m
hhX2m
hhThhXm
ggTpt
strTpt
strXpt

hh2T i
hh2X i
hhT hX i
gT (x1 , y1 , z1 , t)
ST (x1 , y1 , z1 , t)
SX (x1 , y1 , z1 , t)
Module ‘gravitational_waves_hij6.f90’

h22rms
h33rms
h23rms
g11pt
g22pt
g33pt
g12pt
g23pt
g31pt
hhTpt
hhXpt
ggTpt
ggXpt
hhTp2
hhXp2
ggTp2
ggXp2
hrms
gg2m
hhT2m
hhX2m
hhTXm
ggT2m
ggX2m
ggTXm
ggTm
ggXm

hrms
22
hrms
33
hrms
23
g11 (x1 , y1 , z1 , t)
g22 (x1 , y1 , z1 , t)
g33 (x1 , y1 , z1 , t)
g12 (x1 , y1 , z1 , t)
g23 (x1 , y1 , z1 , t)
g31 (x1 , y1 , z1 , t)
hT (x1 , y1 , z1 , t)
hX (x1 , y1 , z1 , t)
ḣT (x1 , y1 , z1 , t)
ḣX (x1 , y1 , z1 , t)
hT (x1 , y1 , z1 , t)
hX (x1 , y1 , z1 , t)
ḣT (x1 , y1 , z1 , t)
ḣX (x1 , y1 , z1 , t)
hh2T + h2X i1/2
2
hgT2 + gX
i
2
hhT i
hh2X i
hhT hX i
hgT2 i
2
hgX
i
hgT gX i
hgT i
hgX i

Module ‘gravity_simple.f90’

epot
epottot
ugm

h̺Φ
R grav i (mean potential energy)
̺Φgrav dV (total potential energy)
V
hu · gi

J.3 Parameters for ‘print.in’
Module ‘heatflux.f90’

dtspitzer
dtq
dtq2
qmax
qxmin
qymin
qzmin
qxmax
qymax
qzmax
qrms
qsatmin
qsatrms

Spitzer heat conduction time step
heatflux time step
heatflux time step due to tau
max(|q|)
min(|qx |)
min(|qy |)
min(|qz |)
max(|qx |)
max(|qy |)
max(|qz |)
rms of heat flux vector
minimum of qsat/qabs
rms of qsat/abs
Module ‘lorenz_gauge.f90’

phim
phipt
phip2
phibzm
phibzmz
ab int
jb int
b2tm
bjtm
jbtm
b2ruzm
b2uzm
ubbzm
b1m
b2m
bm2
j2m
jm2
abm
abumx
abumy
abumz
abmh
abmn
abms
abrms
jbrms
ajm
jbm
jbmh

hφi
φ(x1, y1, z1)
φ(x2, y2, z2)
hφBz i
hφBz ixy
Module ‘magnetic_shearboxJ.f90’
R
R A · B dV
E
D j · BRdV
t
b(t) · 0 b(t′ )dt′
E
D
Rt
b(t) · 0 j(t′ )dt′
E
D
Rt
j(t) · 0 b(t′ )dt′
B 2 ρuz
B 2 uz
h(u · B)Bz i
h|B|i
B2
max(B 2 )
j2
max(j 2 )
hA · Bi
hux A · Bi
huy A · Bi
huz A · Bi
hA · Bi (temp)
hA · Bi (north)
hA · Bi (south)
1/2
h(A · B)2 i
1/2
h(j · B)2 i
hj · Ai
hj · Bi
hJ · Bi (temp)

191

192

T HE P ENCIL C ODE

jbmn
jbms
ubm
dubrms
dobrms
uxbxm
uybxm
uzbxm
uxbym
uybym
uzbym
uxbzm
uybzm
uzbzm
cosubm
jxbxm
jybxm
jzbxm
jxbym
jybym
jzbym
jxbzm
jybzm
jzbzm
uam
ujm
fbm
fxbxm
epsM
epsAD
bxpt
bypt
bzpt
jxpt
jypt
jzpt
Expt
Eypt
Ezpt
axpt
aypt
azpt
bxp2
byp2
bzp2
jxp2
jyp2
jzp2
Exp2
Eyp2

hJ · Bi (north)
hJ · Bi (south)
hu · Bi
1/2
h(u − B)2 i
1/2
h(ω − B)2 i
hux Bx i
huy Bx i
huz Bx i
hux By i
huy By i
huz By i
hux Bz i
huy Bz i
huz Bz i
hU · B/(|U | |B|)i
hjx Bx i
hjy Bx i
hjz Bx i
hjx By i
hjy By i
hjz By i
hjx Bz i
hjy Bz i
hjz Bz i
hu · Ai
hu · J i
hf · Bi
hfx Bx i
ηµ0 j 2
hρ−1 tAD (J × B)2 i (heating by ion-neutrals friction)
Bx (x1 , y1 , z1 , t)
By (x1 , y1 , z1 , t)
Bz (x1 , y1 , z1 , t)
Jx (x1 , y1 , z1 , t)
Jy (x1 , y1 , z1 , t)
Jz (x1 , y1 , z1 , t)
Ex (x1 , y1 , z1 , t)
Ey (x1 , y1 , z1 , t)
Ez (x1 , y1 , z1 , t)
Ax (x1 , y1 , z1 , t)
Ay (x1 , y1 , z1 , t)
Az (x1 , y1 , z1 , t)
Bx (x2 , y2 , z2 , t)
By (x2 , y2 , z2 , t)
Bz (x2 , y2 , z2 , t)
Jx (x2 , y2 , z2 , t)
Jy (x2 , y2 , z2 , t)
Jz (x2 , y2 , z2 , t)
Ex (x2 , y2 , z2 , t)
Ey (x2 , y2 , z2 , t)

J.3 Parameters for ‘print.in’

Ezp2
axp2
ayp2
azp2
exabot
exatop
emag

Ez (x2 , y2 , z2 , t)
Ax (x2 , y2 , z2 , t)
Ay (x2 , y2 , z2 , t)
A
R z (x2 , y2 , z2 , t)
R E × A dS|bot
R E 1× A2 dS|top
B dV
V 2µ0
1/2

bmy

B2
D
E1/2
2
B′
max(|B|)
min(|Bx |)
min(|By |)
min(|Bz |)
max(|Bx |)
max(|By |)
max(|Bz |)
max(|Bx |)excludingBvext
max(|By |)excludingBvext
max(|Bz |)excludingBvext
max(|jvx |)
max(|jvy |)
max(|jvz |)
1/2
j2
1/2
j2
max(|j|)
1/2
B 2 /̺
max(B 2 /̺)1/2
δt/[cδt δx/vA,max ] (time step relative to Alfvén time step; see
§ 5.15)
δt/[cδt,v δx2 /ηmax ] (time step relative to resistive time step;
see § 5.15)
A2
1/2
A2
max(|A|)
h(∇ · A)2 i1/2
B 2 /(2µ0 p)
(mean inverse plasma beta)
2
max[B /(2µ0 p)] (maximum inverse plasma beta)
hβi
max β
min β
hBx i
hBy i
hBz i
hBx By i
D
E1/2
hBi2yz
(energy of yz-averaged mean field)

bmz

(energy of xy-averaged mean field)

brms
bfrms
bmax
bxmin
bymin
bzmin
bxmax
bymax
bzmax
bbxmax
bbymax
bbzmax
jxmax
jymax
jzmax
jrms
hjrms
jmax
vArms
vAmax
dtb
dteta
a2m
arms
amax
divarms
beta1m
beta1max
betam
betamax
betamin
bxm
bym
bzm
bxbym
bmx

1/2

hBi2xz
D
E1/2
2
hBixy

(energy of xz-averaged mean field)

193

194

T HE P ENCIL C ODE

bmzS2
bmzA2
jmx
jmy
jmz
bmzph
bmzphe
bsinphz
bcosphz
emxamz3
embmz
ambmz
ambmzh
ambmzn
ambmzs
jmbmz
kx aa
kmz
bx2m
by2m
bz2m
uxbm
jxbm
magfricmax
b3b21m
b3b12m
b1b32m
b1b23m
b2b13m
b2b31m
uxbmx
uxbmy
uxbmz
jxbmx
jxbmy
jxbmz
examx
examy
examz

D
D

hB S i2xy

E

E

hB A i2xy
D
E1/2
2
hJ iyz
1/2
hJ i2xz
D
E1/2
hJ i2xy

(energy of yz-averaged mean current density)
(energy of xz-averaged mean current density)

(energy of xy-averaged mean current density)
Phase of a Beltrami field
Error of phase of a Beltrami field
sine of phase of a Beltrami field
cosine
of phaseEof a Beltrami field
D
D
)
D

hEixy × hAixy
E
hEixy · hBixy

(xy-averaged mean field helicity flux)

(xy-averaged mean field helicity production

E

hAixy · hBixy
field)
D
E

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, temp) E
D

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, north) E
D

(magnetic helicity of xy-averaged mean

hAixy · hBixy
field, south) E
D

(magnetic helicity of xy-averaged mean

hJ ixy · hBixy

(current helicity of xy-averaged mean field)

k
Dx
E D
E
hJ ixy · hBixy / hBi2xy
hBx2 i
By2
hBz2 i
hu × Bi · B 0 /B02
hj × Bi · B 0 /B02
Magneto-Frictional velocity hj × Bi · B 2
hB3 B2,1 i
hB3 B1,2 i
hB1 B3,2 i
hB1 B2,3 i
hB2 B1,3 i
hB2 B3,1 i
h(u × B)x i
h(u × B)y i
h(u × B)z i
h(j × B)x i
h(j × B)y i
h(j × B)z i
hE × Ai |x
hE × Ai |y
hE × Ai |z

J.3 Parameters for ‘print.in’

exjmx
exjmy
exjmz
dexbmx
dexbmy
dexbmz
phibmx
phibmy
phibmz
b2divum
ujxbm
jxbrmax
jxbr2m
bmxy rms
etasmagm
etasmagmin
etasmagmax
etavamax
etajmax
etaj2max
etajrhomax
cosjbm
jparallelm
jperpm
hjparallelm
hjperpm
brmsx
brmsz
Exmxy
Eymxy
Ezmxy

hE × J i |x
hE × J i |y
hE × J i |z
h∇ × E × Bi |x
h∇ × E × Bi |y
h∇ × E × Bi |z
hφBi |x
hφBi |y
hφBi |z
B2∇ · u
hu · (J × B)i
max(|J × B/ρ|)
2
h(J
p × B/ρ) i
[hbx iz (x, y)]2 + [hby iz (x, y)]2 + [hbz iz (x, y)]2
Mean of Smagorinsky resistivity
Min of Smagorinsky resistivity
Max of Smagorinsky resistivity
Max of artificial resistivity η ∼ vA
√
Max of artificial resistivity η ∼ J/ ρ
Max of artificial resistivity η ∼ J 2 /ρ
Max of artificial resistivity η ∼ J/ρ
hJ · B/(|J | |B|)i
Mean value of the component of J parallel to B
Mean value of the component of J perpendicular to B
Mean value of the component of Jhyper parallel to B
Mean value of the component of Jhyper perpendicular to B
1/2
B2
for the magnetic xaver range
2 1/2
B
for the magnetic zaver range
hEx iz
hEy iz
hEz iz
Module ‘meanfield.f90’

qsm
qpm
qem
qam
alpm
etatm
EMFmz1
EMFmz2
EMFmz3
EMFdotBm
EMFdotB int

qp (B)
qp (B)
qe (B) , in the paper referred to as qg (B)
qa (B)
hαi
hηt i
hEixy |x
hEixy |y
hEixy |z
hE
R · Bi
E · BdV
Module ‘meanfield_demfdt.f90’

EMFrms
EMFmax
EMFmin

(hEi)rms
max(hEi)
min(hEi)

195

196

T HE P ENCIL C ODE
Module ‘noentropy.f90’

dtc
ethm
pdivum

δt/[cδt δx / max cs ] (time step relative to acoustic time step;
see § 5.15)
h̺ei (mean thermal [=internal] energy)
hp∇ui
Module ‘particles_chemistry.f90’

Shchm

meanparticleSherwoodnumber
Module ‘particles_dust.f90’

xpm
xpmin
xpmax
xp2m
vrelpabsm
vpxm
vpx2m
ekinp
vpxmax
vpxmin
npm

xpart
xpart
xpart
x2part
Absolutevalueofmeanrelativevelocity
upart
u2part
Ekin,part
M AX(upart )
M IN (upart )
meanparticlenumberdensity
Module ‘particles_dust_brdeplete.f90’

xpm
xp2m
vrelpabsm
vpxm
vpx2m
ekinp
vpxmax
vpxmin
npm

xpart
x2part
Absolutevalueofmeanrelativevelocity
upart
u2part
Ekin,part
M AX(upart )
M IN (upart )
meanparticlenumberdensity
Module ‘particles_lagrangian.f90’

xpm
xp2m
vrelpabsm
vpxm
vpx2m
ekinp
vpxmax
vpxmin
npm

xpart
x2part
Absolutevalueofmeanrelativevelocity
upart
u2part
Ekin,part
M AX(upart )
M IN (upart )
meanparticlenumberdensity
Module ‘particles_mass_swarm.f90’

mpm
mpmin
mpmax

mp
minj mp,j
maxj mp,j
Module ‘particles_surfspec.f90’

dtpchem

dtparticle,chemistry

J.3 Parameters for ‘print.in’

Module ‘polymer.f90’

polytrm
frmax

hT r[Cij ]i
max(f (r))
Module ‘shear.f90’

dtshear
deltay

advec shear/cdt
deltay
Module ‘shock.f90’

shockmax

Max shock number
Module ‘shock_highorder.f90’

gshockmax

max |∇νshock |

Module ‘solar_corona.f90’

dtvel
dtnewt
dtradloss
dtchi2
dtspitzer
mag flux

Velocity driver time step
Radiative cooling time step
Radiative losses time step
δt/[cδt,v δx2 /χmax ] (time step relative to time step based on
heat conductivity; see § 5.15)
Spitzer heat conduction time step
Total vertical magnetic flux at
Module ‘solid_cells_CGEO.f90’
Module ‘solid_cells_reactive.f90’
Module ‘temperature_idealgas.f90’

TTmax
gTmax
TTmin
TTm
TTzmask
TT2m
TugTm
Trms
uxTm
uyTm
uzTm
gT2m
guxgTm
guygTm
guzgTm
Tugux uxugTm
Tuguy uyugTm
Tuguz uzugTm
Tdxpm
Tdypm
Tdzpm
fradtop

max(T )
max(|∇T |)
min(T )
hT i
hT i for the temp zaver range
hT 2 i
hT
p u · ∇T i
hT 2 i
hux T i
huy T i
huz T i
h(∇T )2 i
h∇ux · ∇T i
h∇uy · ∇T i
h∇uz · ∇T i
hT u · ∇ux + ux u · ∇T i = hu · ∇(ux T )i
hT u · ∇uy + uy u · ∇T i = hu · ∇(uy T )i
hT u · ∇uz + uz u · ∇T i = hu · ∇(uz T )i
hT dp/dxi
hT dp/dyi
hT dp/dzi
>top (top radiative flux)
< −K dT
dz

197

198

T HE P ENCIL C ODE

fradbot
yHmax
yHmin
yHm
ethm
eem
ethtot
ssm
thcool
ppm
csm
dtc
dtchi
Emzmask

< −K dT
>bot (bottom radiative flux)
dz
DOCUMENT ME
DOCUMENT ME
DOCUMENT ME
heth i = hcv ρT i (mean thermal energy)
hei
R = hcv T i (mean internal energy)
̺e dV (total thermal energy)
V
S
τcool
P
cs
δt/[cδt δx / max cs ] (time step relative to acoustic time step;
see § 5.15)
δt/[cδt,v δx2 /χmax ] (time step relative to time step based on
heat conductivity; see § 5.15)
hn2 exp −(log T − log T0 )2 /(δ log T )2 i the emiss zaver range
Module ‘temperature_ionization.f90’

TTmax
TTmin
TTm
ethm
eem
ppm

max(T )
min(T )
hT i
heth i = hcv ρT i (mean thermal energy)
hei (mean internal energy)
hpi
Module ‘testfield_axisym.f90’

alpPERP
alpPARA
gam
betPERP
betPARA
del
kapPERP
kapPARA
mu
alpPERPz
alpPARAz
gamz
betPERPz
betPARAz
delz
kapPERPz
kapPARAz
muz
bx1pt
bx2pt
bx3pt
b1rms
b2rms
b3rms

α⊥
α⊥
γ
β⊥
β⊥
δ
κ⊥
κ⊥
µ
α⊥ (z)
α⊥ (z)
γ(z)
β⊥ (z)
β⊥ (z)
δ(z)
κ⊥ (z)
κ⊥ (z)
µ(z)
b1x
b2x
b3x
1/2
hb21 i
1/2
hb22 i
1/2
hb23 i

J.3 Parameters for ‘print.in’

Module ‘testfield_axisym2.f90’

alpPERP
alpPARA
gam
betPERP
betPARA
del
kapPERP
kapPARA
mu
bx1pt
bx2pt
bx3pt
b1rms
b2rms
b3rms

α⊥
α⊥
γ
β⊥
β⊥
δ
κ⊥
κ⊥
µ
b1x
b2x
b3x
1/2
hb21 i
1/2
hb22 i
1/2
hb23 i

Module ‘testfield_axisym4.f90’

alpPERP
alpPARA
gam
betPERP
betPERP2
betPARA
del
del2
kapPERP
kapPERP2
kapPARA
mu
mu2
alpPERPz
alpPARAz
gamz
betPERPz
betPARAz
delz
kapPERPz
kapPARAz
muz
bx1pt
bx2pt
bx3pt
b1rms
b2rms
b3rms

α⊥
α⊥
γ
β⊥
(2)
β⊥
β⊥
δ
δ (2)
κ⊥
(2)
κ⊥
κ⊥
µ
µ(2)
α⊥ (z)
α⊥ (z)
γ(z)
β⊥ (z)
β⊥ (z)
δ(z)
κ⊥ (z)
κ⊥ (z)
µ(z)
b1x
b2x
b3x
1/2
hb21 i
1/2
hb22 i
1/2
hb23 i

Module ‘testfield_compress_z.f90’

199

200

T HE P ENCIL C ODE

alp11
alp21
alp31
alp12
alp22
alp32
eta11
eta21
eta12
eta22
alpK
alpM
alpMK
phi11
phi21
phi12
phi22
phi32
psi11
psi21
psi12
psi22
phiK
phiM
phiMK
alp11cc
alp21sc
alp12cs
alp22ss
eta11cc
eta21sc
eta12cs
eta22ss
s2kzDFm
M11
M22
M33
M11cc
M11ss
M22cc
M22ss
M12cs
bx11pt
bx21pt
bx12pt
bx22pt
bx0pt
by11pt
by21pt
by12pt

α11
α21
α31
α12
α22
α32
η11 k
η21 k
η12 k
η22 k
αK
αM
αM K
φ11
φ21
φ12
φ22
φ32
ψ11 k
ψ21 k
ψ12 k
ψ22 k
φK
φM
φM K
α11 cos2 kz
α21 sin kz cos kz
α12 cos kz sin kz
α22 sin2 kz
η11 cos2 kz
η21 sin kz cos kz
η12 cos kz sin kz
η22 sin2 kz
hsin 2kz∇ · F i
M11
M22
M33
M11 cos2 kz
M11 sin2 kz
M22 cos2 kz
M22 sin2 kz
M12 cos kz sin kz
b11
x
b21
x
b12
x
b22
x
b0x
b11
y
b21
y
b12
y

J.3 Parameters for ‘print.in’

by22pt
by0pt
u11rms
u21rms
u12rms
u22rms
j11rms
b11rms
b21rms
b12rms
b22rms
ux0m
uy0m
ux11m
uy11m
u0rms
b0rms
jb0m
E11rms
E21rms
E12rms
E22rms
E0rms
Ex11pt
Ex21pt
Ex12pt
Ex22pt
Ex0pt
Ey11pt
Ey21pt
Ey12pt
Ey22pt
Ey0pt
bamp
E111z
E211z
E311z
E121z
E221z
E321z
E112z
E212z
E312z
E122z
E222z
E322z
E10z
E20z

b22
y
b0y
1/2
hu211 i
1/2
hu221 i
1/2
hu212 i
1/2
hu222 i
2 1/2
hj11
i
2 1/2
hb11 i
1/2
hb221 i
1/2
hb212 i
1/2
hb222 i
hu0x i
u0y
hu11x i
u11y
1/2
hu20 i
1/2
hb20 i
hjb0 i
2 1/2
hE11
i
2 1/2
hE21 i
2 1/2
hE12
i
2 1/2
hE22 i
1/2
hE02 i
Ex11
Ex21
Ex12
Ex22
Ex0
Ey11
Ey21
Ey12
Ey22
Ey0
bamp
E111
E211
E311
E121
E221
E321
E112
E212
E312
E122
E222
E322
E10
E20

201

202

T HE P ENCIL C ODE

E30z
EBpq
E0Um
E0Wm
bx0mz
by0mz
bz0mz
M11z
M22z
M33z

E30
E · B pq
E0 · U
E0 · W
hbx ixy
hby ixy
hbz ixy
hM11 ixy
hM22 ixy
hM33 ixy
Module ‘testfield_meri.f90’

E11xy
E12xy
E13xy
E21xy
E22xy
E23xy
E31xy
E32xy
E33xy
E41xy
E42xy
E43xy
E51xy
E52xy
E53xy
E61xy
E62xy
E63xy
E71xy
E72xy
E73xy
E81xy
E82xy
E83xy
E91xy
E92xy
E93xy
a11xy
a12xy
a13xy
a21xy
a22xy
a23xy
a31xy
a32xy
a33xy
b111xy
b121xy

E11xy
E12xy
E13xy
E21xy
E22xy
E23xy
E31xy
E32xy
E33xy
E41xy
E42xy
E43xy
E51xy
E52xy
E53xy
E61xy
E62xy
E63xy
E71xy
E72xy
E73xy
E81
E82
E83
E91
E92
E93
α11
α12
α13
α21
α22
α23
α31
α32
α33
111
¯121
¯

J.3 Parameters for ‘print.in’

b131xy
b211xy
b221xy
b231xy
b311xy
b321xy
b331xy
b112xy
b122xy
b132xy
b212xy
b222xy
b232xy
b312xy
b322xy
b332xy
alp11
alp21
alp31
alp12
alp22
alp32
eta11
eta21
eta12
eta22
alpK
alpM
alpMK
phi11
phi21
phi12
phi22
phi32
psi11
psi21
psi12
psi22
phiK
phiM
phiMK
alp11cc
alp21sc
alp12cs
alp22ss
eta11cc
eta21sc
eta12cs

131
¯211
¯221
¯231
¯311
¯321
¯331
¯112
¯122
¯132
¯212
¯222
¯232
¯312
¯322
¯332
¯

Module ‘testfield_nonlin_z.f90’

α11
α21
α31
α12
α22
α32
η11 k
η21 k
η12 k
η22 k
αK
αM
αM K
φ11
φ21
φ12
φ22
φ32
ψ11 k
ψ21 k
ψ12 k
ψ22 k
φK
φM
φM K
α11 cos2 kz
α21 sin kz cos kz
α12 cos kz sin kz
α22 sin2 kz
η11 cos2 kz
η21 sin kz cos kz
η12 cos kz sin kz

203

204

T HE P ENCIL C ODE

eta22ss
s2kzDFm
M11
M22
M33
M11cc
M11ss
M22cc
M22ss
M12cs
bx11pt
bx21pt
bx12pt
bx22pt
bx0pt
by11pt
by21pt
by12pt
by22pt
by0pt
u11rms
u21rms
u12rms
u22rms
j11rms
b11rms
b21rms
b12rms
b22rms
ux0m
uy0m
ux11m
uy11m
u0rms
b0rms
jb0m
E11rms
E21rms
E12rms
E22rms
E0rms
Ex11pt
Ex21pt
Ex12pt
Ex22pt
Ex0pt
Ey11pt
Ey21pt

η22 sin2 kz
hsin 2kz∇ · F i
M11
M22
M33
M11 cos2 kz
M11 sin2 kz
M22 cos2 kz
M22 sin2 kz
M12 cos kz sin kz
b11
x
b21
x
b12
x
b22
x
b0x
b11
y
b21
y
b12
y
b22
y
b0y
1/2
hu211 i
1/2
hu221 i
1/2
hu212 i
1/2
hu222 i
2 1/2
hj11
i
1/2
hb211 i
1/2
hb221 i
1/2
hb212 i
1/2
hb222 i
hu0x i
u0y
hu11x i
u11y
1/2
hu20 i
1/2
hb20 i
hjb0 i
2 1/2
hE11
i
2 1/2
hE21 i
2 1/2
hE12
i
2 1/2
hE22 i
1/2
hE02 i
Ex11
Ex21
Ex12
Ex22
Ex0
Ey11
Ey21

J.3 Parameters for ‘print.in’

Ey12pt
Ey22pt
Ey0pt
bamp
E111z
E211z
E311z
E121z
E221z
E321z
E112z
E212z
E312z
E122z
E222z
E322z
E10z
E20z
E30z
EBpq
E0Um
E0Wm
bx0mz
by0mz
bz0mz
M11z
M22z
M33z

Ey12
Ey22
Ey0
bamp
E111
E211
E311
E121
E221
E321
E112
E212
E312
E122
E222
E322
E10
E20
E30
E · B pq
E0 · U
E0 · W
hbx ixy
hby ixy
hbz ixy
hM11 ixy
hM22 ixy
hM33 ixy
Module ‘testfield_x.f90’

alp11
alp21
alp31
alp12
alp22
alp32
eta11
eta21
eta12
eta22
alp11cc
alp21sc
alp12cs
alp22ss
eta11cc
eta21sc
eta12cs
eta22ss
alp11 x
alp21 x

α11
α21
α31
α12
α22
α32
η11 k
η21 k
η12 k
η22 k
α11 cos2 kx
α21 sin kx cos kx
α12 cos kx sin kx
α22 sin2 kx
η11 cos2 kx
η21 sin kx cos kx
η12 cos kx sin kx
η22 sin2 kx
α11 x
α21 x

205

206

T HE P ENCIL C ODE

alp12 x
alp22 x
eta11 x
eta21 x
eta12 x
eta22 x
alp11 x2
alp21 x2
alp12 x2
alp22 x2
eta11 x2
eta21 x2
eta12 x2
eta22 x2
b11rms
b21rms
b12rms
b22rms
b0rms
E11rms
E21rms
E12rms
E22rms
E0rms
E111z
E211z
E311z
E121z
E221z
E321z
E112z
E212z
E312z
E122z
E222z
E322z
E10z
E20z
E30z
EBpq
bx0mz
by0mz
bz0mz
alp11x
alp21x
alp12x
alp22x
eta11x

α12 x
α22 x
η11 kx
η21 kx
η12 kx
η22 kx
α11 x2
α21 x2
α12 x2
α22 x2
η11 kx2
η21 kx2
η12 kx2
η22 kx2
1/2
hb211 i
1/2
hb221 i
1/2
hb212 i
1/2
hb222 i
1/2
hb20 i
2 1/2
hE11
i
2 1/2
hE21 i
2 1/2
hE12
i
2 1/2
hE22 i
1/2
hE02 i
E111
E211
E311
E121
E221
E321
E112
E212
E312
E122
E222
E322
E10
E20
E30
E · B pq
hbx ixy
hby ixy
hbz ixy
α11 (x, t)
α21 (x, t)
α12 (x, t)
α22 (x, t)
η11 (x, t)

J.3 Parameters for ‘print.in’

eta21x
eta12x
eta22x

η21 (x, t)
η12 (x, t)
η22 (x, t)
Module ‘testfield_xz.f90’

E111z
E211z
E311z
E121z
E221z
E321z
alp11
alp21
eta11
eta21
b11rms
b21rms
alp11
alp21
alp31
alp12
alp22
alp32
alp13
alp23
eta11
eta21
eta31
eta12
eta22
eta32
alp11cc
alp21sc
alp12cs
alp22ss
eta11cc
eta21sc
eta12cs
eta22ss
s2kzDFm
M11
M22
M33
M11cc
M11ss
M22cc
M22ss
M12cs

E111
E211
E311
E121
E221
E321
α11
α21
η113 k
η213 k
hb211 i
hb221 i

Module ‘testfield_z.f90’

α11
α21
α31
α12
α22
α32
α13
α23
η113 k or η11 k if leta
η213 k or η21 k if leta
η313 k
η123 k or η12 k if leta
η223 k or η22 k if leta
η323 k
α11 cos2 kz
α21 sin kz cos kz
α12 cos kz sin kz
α22 sin2 kz
η11 cos2 kz
η21 sin kz cos kz
η12 cos kz sin kz
η22 sin2 kz
hsin 2kz∇ · F i
M11
M22
M33
M11 cos2 kz
M11 sin2 kz
M22 cos2 kz
M22 sin2 kz
M12 cos kz sin kz

rank2=T
rank2=T
rank2=T
rank2=T

207

208

T HE P ENCIL C ODE

bx11pt
bx21pt
bx12pt
bx22pt
bx0pt
by11pt
by21pt
by12pt
by22pt
by0pt
b11rms
b21rms
b12rms
b22rms
b0rms
jb0m
E11rms
E21rms
E12rms
E22rms
E0rms
Ex11pt
Ex21pt
Ex12pt
Ex22pt
Ex0pt
Ey11pt
Ey21pt
Ey12pt
Ey22pt
Ey0pt
bamp
alp11z
alp21z
alp12z
alp22z
alp13z
alp23z
eta11z
eta21z
eta12z
eta22z
E111z
E211z
E311z
E121z
E221z
E321z

b11
x
b21
x
b12
x
b22
x
b0x
b11
y
b21
y
b12
y
b22
y
b0y
1/2
hb211 i
1/2
hb221 i
1/2
hb212 i
1/2
hb222 i
1/2
hb20 i
hjb0 i
2 1/2
hE11
i
2 1/2
hE21 i
2 1/2
hE12
i
2 1/2
hE22
i
2 1/2
hE0 i
Ex11
Ex21
Ex12
Ex22
Ex0
Ey11
Ey21
Ey12
Ey22
Ey0
bamp
α11 (z, t)
α21 (z, t)
α12 (z, t)
α22 (z, t)
α13 (z, t)
α23 (z, t)
η11 (z, t)
η21 (z, t)
η12 (z, t)
η22 (z, t)
E111
E211
E311
E121
E221
E321

J.3 Parameters for ‘print.in’

E112z
E212z
E312z
E122z
E222z
E322z
E10z
E20z
E30z
EBpq
E0Um
E0Wm
bx0mz
by0mz
bz0mz
M11z
M22z
M33z

E112
E212
E312
E122
E222
E322
E10
E20
E30
E · B pq
E0 · U
E0 · W
hbx ixy
hby ixy
hbz ixy
hM11 ixy
hM22 ixy
hM33 ixy
Module ‘testflow_z.f90’

gal
aklam
gamma
nu
zeta
xi
aklamQ
gammaQ
nuQ
zetaQ
xiQ
ux0mz
uy0mz
uz0mz

GAL-coefficients, couple F and U
AKA-λ-tensor, couples F and W = ∇ × U
γ-vector, couples F and ∇ · U
ν-tensor, couples F and ∂ 2 U /∂z 2
ζ-vector, couples F and Gz = ∇z H
ξ-vector, couples F and ∂ 2 H/∂z 2
aklamQ -vector, couples Q and W
γ Q -scalar, couples Q and ∇ · U = dUz /dz
ν Q -vector, couples Q and ∂ 2 U /∂z 2
ζ Q -scalar, couples Q and Gz
ξ Q -scalar, couples Q and ∂ 2 H/∂z 2
pq
pq
αK,ij γi νij ζi ξi νiQ aklamQ
upq2 hpq 2
i Fi Q
hux ixy
huy ixy
huz ixy
Module ‘testperturb.f90’

alp11
alp21
alp31
alp12
alp22
alp32
eta11
eta21
eta31
eta12
eta22
eta32

α11
α21
α31
α12
α22
α32
η113 k
η213 k
η313 k
η123 k
η223 k
η323 k

209

210

T HE P ENCIL C ODE
Module ‘testscalar.f90’

gam11
gam12
gam13
gam21
gam22
gam23
gam31
gam32
gam33
kap11
kap21
kap31
kap12
kap22
kap32
kap13
kap23
kap33
gam11z
gam12z
gam13z
gam21z
gam22z
gam23z
gam31z
gam32z
gam33z
kap11z
kap21z
kap31z
kap12z
kap22z
kap32z
kap13z
kap23z
kap33z
mgam33
mkap33
ngam33
nkap33
c1rms
c2rms
c3rms
c4rms
c5rms

(1)

γ1
(1)
γ2
(1)
γ3
(2)
γ1
(2)
γ2
(2)
γ3
(3)
γ1
(3)
γ2
(3)
γ3
κ11
κ21
κ31
κ12
κ22
κ32
κ13
κ23
κ33
(1)
γ1 (z, t)
(1)
γ2 (z, t)
(1)
γ3 (z, t)
(2)
γ1 (z, t)
(2)
γ2 (z, t)
(2)
γ3 (z, t)
(3)
γ1 (z, t)
(3)
γ2 (z, t)
(3)
γ3 (z, t)
κ11 (z, t)
κ21 (z, t)
κ31 (z, t)
κ12 (z, t)
κ22 (z, t)
κ32 (z, t)
κ13 (z, t)
κ23 (z, t)
κ33 (z, t)
γ̃33
κ̃33
γ̂33
κ̂33
1/2
hc21 i
1/2
hc22 i
1/2
hc23 i
1/2
hc24 i
1/2
hc25 i

J.3 Parameters for ‘print.in’

c6rms
c1pt
c2pt
c3pt
c4pt
c5pt
c6pt
F11z
F21z
F31z
F12z
F22z
F32z
muc1
muc2
gamc
kapcPERP1
kapcPERP2
kapcPARA
mucz
gamcz
kapcPERPz
kapcPARAz
gam11
gam12
gam13
gam21
gam22
gam23
gam31
gam32
gam33
kap11
kap21
kap31
kap12
kap22
kap32
kap13
kap23
kap33
gam11z
gam12z
gam13z
gam21z
gam22z

hc26 i
c1
c2
c3
c4
c5
c6
F11
F21
F31
F12
F22
F32

1/2

Module ‘testscalar_axisym.f90’

µ(c1)
µ(c2)
γ (c)
(1)
κ⊥
(2)
κ⊥
κk
µ(c) (z, t)
γ (c) (z, t)
κ⊥ (z, t)
κk (z, t)
(1)
γ1
(1)
γ2
(1)
γ3
(2)
γ1
(2)
γ2
(2)
γ3
(3)
γ1
(3)
γ2
(3)
γ3
κ11
κ21
κ31
κ12
κ22
κ32
κ13
κ23
κ33
(1)
γ1 (z, t)
(1)
γ2 (z, t)
(1)
γ3 (z, t)
(2)
γ1 (z, t)
(2)
γ2 (z, t)

211

212

T HE P ENCIL C ODE

gam23z
gam31z
gam32z
gam33z
gam3z
kap11z
kap21z
kap31z
kap12z
kap22z
kap32z
kap13z
kap23z
kap33z
mgam33
mkap33
ngam33
nkap33
c1rms
c2rms
c3rms
c4rms
c5rms
c6rms
c1pt
c2pt
c3pt
c4pt
c5pt
c6pt
F11z
F21z
F31z
F12z
F22z
F32z

(2)

γ3 (z, t)
(3)
γ1 (z, t)
(3)
γ2 (z, t)
(3)
γ3 (z, t)
γ (c) (z, t)
κ11 (z, t)
κ21 (z, t)
κ31 (z, t)
κ12 (z, t)
κ22 (z, t)
κ32 (z, t)
κ13 (z, t)
κ23 (z, t)
κ33 (z, t)
γ̃33
κ̃33
γ̂33
κ̂33
1/2
hc21 i
1/2
hc22 i
1/2
hc23 i
1/2
hc24 i
1/2
hc25 i
1/2
hc26 i
c1
c2
c3
c4
c5
c6
F11
F21
F31
F12
F22
F32

Module ‘testscalar_simple.f90’

gam11
gam12
gam13
gam21
gam22
gam23
gam31
gam32
gam33
kap11

(1)

γ1
(1)
γ2
(1)
γ3
(2)
γ1
(2)
γ2
(2)
γ3
(3)
γ1
(3)
γ2
(3)
γ3
κ11

J.3 Parameters for ‘print.in’

kap21
kap31
kap12
kap22
kap32
kap13
kap23
kap33
gam11z
gam12z
gam13z
gam21z
gam22z
gam23z
gam31z
gam32z
gam33z
kap11z
kap21z
kap31z
kap12z
kap22z
kap32z
kap13z
kap23z
kap33z
mgam33
mkap33
ngam33
nkap33
c1rms
c2rms
c3rms
c4rms
c5rms
c6rms
c1pt
c2pt
c3pt
c4pt
c5pt
c6pt
F11z
F21z
F31z
F12z
F22z
F32z

κ21
κ31
κ12
κ22
κ32
κ13
κ23
κ33
(1)
γ1 (z, t)
(1)
γ2 (z, t)
(1)
γ3 (z, t)
(2)
γ1 (z, t)
(2)
γ2 (z, t)
(2)
γ3 (z, t)
(3)
γ1 (z, t)
(3)
γ2 (z, t)
(3)
γ3 (z, t)
κ11 (z, t)
κ21 (z, t)
κ31 (z, t)
κ12 (z, t)
κ22 (z, t)
κ32 (z, t)
κ13 (z, t)
κ23 (z, t)
κ33 (z, t)
γ̃33
κ̃33
γ̂33
κ̂33
1/2
hc21 i
1/2
hc22 i
1/2
hc23 i
1/2
hc24 i
1/2
hc25 i
1/2
hc26 i
c1
c2
c3
c4
c5
c6
F11
F21
F31
F12
F22
F32

213

214

T HE P ENCIL C ODE

Module ‘thermal_energy.f90’

TTmax
TTmin
ppm
TTm
ethm
ethtot
ethmin
ethmax
eem
etot

max(T )
min(T )
hpi
hT i
he
R th i = hcv ρT i (mean thermal energy)
e dV (total thermal energy)
V th
mineth
maxeth
hei = hcv T i (mean internal energy)
heth + ρu2 /2i
Module ‘visc_smagorinsky.f90’

nu LES

Mean value of Smagorinsky viscosity
Module ‘viscosity.f90’

nu tdep
fviscm
fviscmin
fviscmax
fviscrmsx
num
nusmagm
nusmagmin
nusmagmax
nu LES
visc heatm
qfviscm
ufviscm
Sij2m
epsK
dtnu
meshRemax
Reshock

J.4

time-dependent viscosity
Mean value of viscous acceleration
Min value of viscous acceleration
Max value of viscous acceleration
Rms value of viscous acceleration for the vis xaver range
Mean value of viscosity
Mean value of Smagorinsky viscosity
Min value of Smagorinsky viscosity
Max value of Smagorinsky viscosity
Mean value of Smagorinsky viscosity
Mean value of viscous heating
hq · f visc i
hu · f visc i
S2
2ν̺S2
δt/[cδt,v δx2 /νmax ] (time step relative to viscous time step; see
§ 5.15)
Max mesh Reynolds number
Mesh Reynolds number at shock

List of parameters for ‘video.in’

The following table lists all (at the time of writing, May 10, 2018) possible inputs to the
file ‘video.in’.
Variable

Meaning
Module ‘hydro.f90’

uu
u2

velocity vector u; writes all three components separately to
files ‘u[xyz].{xz,yz,xy,xy2}’
kinetic energy density u2 ; writes ‘u2.{xz,yz,xy,xy2}’

J.4 Parameters for ‘video.in’

oo
o2
divu
mach

215

vorticity vector ω = ∇ × u; writes all three components separately to files ‘oo[xyz].{xz,yz,xy,xy2}’
enstrophy ω 2 = |∇ × u|2 ; writes ‘o2.{xz,yz,xy,xy2}’
∇ · u; writes ‘divu.{xz,yz,xy,xy2}’
Mach number squared Ma2 ; writes ‘mach.{xz,yz,xy,xy2}’
Module ‘density.f90’

lnrho
rho

logarithmic density ln ρ; writes ‘lnrho.{xz,yz,xy,xy2}’
density ρ; writes ‘rho.{xz,yz,xy,xy2}’
Module ‘entropy.f90’

ss
pp

entropy s; writes ‘ss.{xz,yz,xy,xy2}’
pressure p; writes ‘pp.{xz,yz,xy,xy2}’
Module ‘temperature_idealgas.f90’

lnTT
TT

logarithmic temperature ln T ; writes ‘lnTT.{xz,yz,xy,xy2}’
temperature T ; writes ‘TT.{xz,yz,xy,xy2}’
Module ‘shock.f90’

shock

shock viscosity νshock ; writes ‘shock.{xz,yz,xy,xy2}’
Module ‘eos_ionization.f90’

yH

ionization fraction yH ; writes ‘yH.{xz,yz,xy,xy2}’
Module ‘radiation_ray.f90’

Qrad
Isurf

radiative heating rate Qrad ; writes ‘Qrad.{xz,yz,xy,xy2}’
surface intensity Isurf (?); writes ‘Isurf.xz’
Module ‘magnetic.f90’

aa
bb
b2
jj
j2
jb
beta1
Poynting
ab

magnetic vector potential A; writes ‘aa[xyz].{xz,yz,xy,xy2}’
magnetic flux density B; writes ‘bb[xyz].{xz,yz,xy,xy2}’
magnetic energy density B 2 ; writes ‘b2.{xz,yz,xy,xy2}’
current density j; writes ‘jj[xyz].{xz,yz,xy,xy2}’
current density squared j 2 ; writes ‘j2.{xz,yz,xy,xy2}’
jB; writes ‘jb.{xz,yz,xy,xy2}’
inverse plasma beta B 2 /(2µ0 p); writes ‘beta1.{xz,yz,xy,xy2}’
Poynting vector ηj × B − (u × B) × B/µ0 ; writes
‘Poynting[xyz].{xz,yz,xy,xy2}’
magnetic
helicity
density
A
·
B;
writes
‘ab[xyz].{xz,yz,xy,xy2}’
Module ‘pscalar.f90’

lncc

logarithmic density
‘lncc.{xz,yz,xy,xy2}’

of

passive

scalar

ln c;

writes

Module ‘cosmicray.f90’

ecr

energy ecr of cosmic rays (?); writes ‘ec.{xz,yz,xy,xy2}’

216

J.5

T HE P ENCIL C ODE

List of parameters for ‘phiaver.in’

The following table lists all (at the time of writing, November 2003) possible inputs to
the file ‘phiaver.in’.
Variable

rcylmphi
phimphi
zmphi
rmphi

Meaning
Module ‘cdata.f90’
p
cylindrical radius ̟ =
x2 + y 2 (useful for debugging azimuthal averages)
azimuthal angle ϕ = arctan xy (useful for debugging)
z-coordinate (useful for
√ debugging)
spherical radius r = ̟2 + z 2 (useful for debugging)
Module ‘hydro.f90’

urmphi
upmphi
uzmphi
ursphmphi
uthmphi
uumphi
uusphmphi
u2mphi

hu̟ iϕ [cyl. polar coords (̟, ϕ, z)]
huϕ iϕ
huz iϕ
hur iϕ
huϑ iϕ
shorthand for urmphi , upmphi and uzmphi together
shorthand for ursphmphi , uthmphi and upmphi together
hu2 iϕ
Module ‘density.f90’

lnrhomphi
rhomphi

hln ̺iϕ
h̺iϕ
Module ‘entropy.f90’

ssmphi
cs2mphi

hsiϕ
hc2s iϕ
Module ‘magnetic.f90’

jbmphi
brmphi
bpmphi
bzmphi
bbmphi
bbsphmphi
b2mphi
brsphmphi
bthmphi

hJ · Biϕ
hB̟ iϕ [cyl. polar coords (̟, ϕ, z)]
hBϕ iϕ
hBz iϕ
shorthand for brmphi , bpmphi and bzmphi together
shorthand for brsphmphi , bthmphi and bpmphi together
B2 ϕ
hBr iϕ
hBϑ iϕ
Module ‘anelastic.f90’

lnrhomphi
rhomphi

hln ̺iϕ
h̺iϕ
Module ‘entropy_anelastic.f90’

ssmphi

hsiϕ

J.6 Parameters for ‘xyaver.in’

cs2mphi

217

hc2s iϕ
Module ‘magnetic_shearboxJ.f90’

jbmphi
brmphi
bpmphi
bzmphi
bbmphi
bbsphmphi
b2mphi
brsphmphi
bthmphi

J.6

hJ · Biϕ
hB̟ iϕ [cyl. polar coords (̟, ϕ, z)]
hBϕ iϕ
hBz iϕ
shorthand for brmphi , bpmphi and bzmphi together
shorthand for brsphmphi , bthmphi and bpmphi together
B2 ϕ
hBr iϕ
hBϑ iϕ

List of parameters for ‘xyaver.in’

The following table lists possible inputs to the file ‘xyaver.in’. This list is not complete
and maybe outdated.
Variable

Meaning
Module ‘hydro.f90’

u2mz
o2mz
divu2mz
curlru2mz
divru2mz
fmasszmz
fkinzmz
uxmz
uymz
uzmz
uzupmz
uzdownmz
ruzupmz
ruzdownmz
divumz
uzdivumz
oxmz
oymz
ozmz
ux2mz
uy2mz
uz2mz
ox2mz
oy2mz
oz2mz

hu2 ixy
W 2 xy
h(∇ · u)2 ixy
h(∇ × ̺U )2 ixy
h(∇ · ̺u)2 ixy
h̺uz ixy
1
̺u2 uz xy
2
hux ixy (horiz. averaged x velocity)
huy ixy
huz ixy
huz↑ ixy
huz↓ ixy
h̺uz↑ ixy
h̺uz↓ ixy
hdivuixy
huz divuixy
hωx ixy
hωy ixy
hωz ixy
hu2x ixy
u2y xy
hu2z ixy
hωx2 ixy
ωy2 xy
hωz2 ixy

218

T HE P ENCIL C ODE

ruxmz
ruymz
ruzmz
rux2mz
ruy2mz
ruz2mz
uxuymz
uxuzmz
uyuzmz
ruxuymz
ruxuzmz
ruyuzmz
ruxuy2mz
ruxuz2mz
ruyuz2mz
oxuxxmz
oyuxymz
oxuyxmz
oyuyymz
oxuzxmz
oyuzymz
uyxuzxmz
uyyuzymz
uyzuzzmz
ekinmz
oumz
Remz
oguxmz
oguymz
oguzmz
ogux2mz
oguy2mz
oguz2mz
oxdivumz
oydivumz
ozdivumz
oxdivu2mz
oydivu2mz
ozdivu2mz
accpowzmz
accpowzupmz
accpowzdownmz
fkinxmx

h̺ux ixy
h̺uy ixy
h̺uz ixy
h̺u2x ixy
̺u2y xy
h̺u2z ixy
hux uy ixy
hux uz ixy
huy uz ixy
hρux uy ixy
hρux uz ixy
hρuy uz ixy
hρux uy ixy
hρux uz ixy
hρuy uz ixy
hωx ux,x ixy
hωy ux,y ixy
hωx uy,x ixy
hωy uy,y ixy
hωx uz,x ixy
hωy uz,y ixy
huy,x uz,x ixy
huy,y uz,y ixy
huy,z uz,z ixy
1
̺u2 xy
2
hω · uixy
h |u·u| ixy
∂
∂xj

(νSij )

h(ω · ∇u)x ixy
h(ω · ∇u)y ixy
h(ω · ∇u)z ixy
h(ω · ∇u)2x ixy
(ω · ∇u)2y xy
h(ω · ∇u)2z ixy
hωx ∇ · uixy
hωy ∇ · uixy
hωz ∇ · uixy
h(ωx nabla · u)2 ixy
h(ωy ∇ · u)2 ixy
h(ωz ∇ · u)2 ixy
h(uz Duz /Dt)2 ixy
h(uz Duz /Dt)2 ixy+
h(uz Duz /Dt)2 ixy−
1
̺u2 ux yz
2
Module ‘density.f90’

rhomz
rho2mz

h̺ixy
h̺2 ixy

J.6 Parameters for ‘xyaver.in’

gzlnrhomz
uglnrhomz
ugrhomz
uygzlnrhomz
uzgylnrhomz
rho2mx

h∇z ln ̺ixy
hu · ∇ ln ̺ixy
hu · ∇̺ixy
huy ∇z ln ̺ixy
huz ∇y ln ̺ixy
h̺2 iyz
Module ‘entropy.f90’

fradz
fconvz
ssmz
ss2mz
ppmz
TTmz
TT2mz
uxTTmz
uyTTmz
uzTTmz
gTxgsxmz
gTxgsymz
gTxgszmz
gTxgsx2mz
gTxgsy2mz
gTxgsz2mz
fradz kramers
fradz Kprof
fradz constchi
fturbz
fturbtz
fturbmz
fturbfz
dcoolz
heatmz
Kkramersmz
ethmz

hFrad ixy
hcp ̺uz T ixy
hsixy
hs2 ixy
hpixy
hT ixy
hT 2 ixy
hux T ixy
huy T ixy
huz T ixy
h(∇T × ∇s)x ixy
h(∇T × ∇s)y ixy
h(∇T × ∇s)z ixy
h(∇T × ∇s)2x ixy
(∇T × ∇s)2y xy
h(∇T × ∇s)2z ixy
Frad (from Kramers’ opacity)
Frad (from Kprof)
Frad (from chi const)
h̺T χt ∇z sixy (turbulent heat flux)
h̺T χt 0∇z sixy (turbulent heat flux)
h̺T χt 0∇z sixy (turbulent heat flux)
h̺T χt 0∇z s′ ixy (turbulent heat flux)
surface cooling flux
heating
K0 T ( 3 − b)/rho( a + 1) xy
h̺eixy
Module ‘magnetic.f90’

axmz
aymz
azmz
abuxmz
abuymz
abuzmz
uabxmz
uabymz
uabzmz
bbxmz
bbymz

hAx ixy
hAy ixy
hAz ixy
h(A · B)ux ixy
h(A · B)uy ixy
h(A · B)uz ixy
h(u · A)Bx ixy
h(u · A)By ixy
h(u · A)Bz ixy
hBx′ ixy
By′ xy

219

220

T HE P ENCIL C ODE

bbzmz
bxmz
bymz
bzmz
jxmz
jymz
jzmz
Exmz
Eymz
Ezmz
bx2mz
by2mz
bz2mz
bx2rmz
by2rmz
bz2rmz
beta1mz
betamz
beta2mz
jbmz
d6abmz
d6amz1
d6amz2
d6amz3
abmz
ubmz
uamz
uxbxmz
uybxmz
uzbxmz
uxbymz
uybymz
uzbymz
uxbzmz
uybzmz
uzbzmz
examz1
examz2
examz3
e3xamz1
e3xamz2
e3xamz3
etatotalmz
bxbymz
bxbzmz
bybzmz
b2mz
bf2mz

hBz′ ixy
hBx ixy
hBy ixy
hBz ixy
hJx ixy
hJy ixy
hJz ixy
hEx ixy
hEy ixy
hEz ixy
hBx2 ixy
By2 xy
hBz2 ixy
hBx2 /̺ixy
By2 /̺ xy
hBz2 /̺ixy
h(B 2 /2µ0 )/pixy
hβixy
hβ 2 ixy
hJ · Bi |xy
h∇6 A · Bi |xy
h∇6 Aixy |x
h∇6 Aixy |y
h∇6 Aixy |z
hA · Bi |xy
hu · Bi |xy
hu · Ai |xy
hux bx i |xy
huy bx i |xy
huz bx i |xy
hux by i |xy
huy by i |xy
huz by i |xy
hux bz i |xy
huy bz i |xy
huz bz i |xy
hE × Aixy |x
hE × Aixy |y
hE × Aixy |z
hE hyper3 × Aixy |x
hE hyper3 × Aixy |y
hE hyper3 × Aixy |z
hηixy
hBx By ixy
hBx Bz ixy
hBy Bz ixy
B 2 xy
B ′2 xy

J.6 Parameters for ‘xyaver.in’

j2mz
poynzmz
epsMmz

j 2 xy
Averaged poynting flux in z direction
ηµ0 j 2 xy
Module ‘bfield.f90’

bmz
b2mz
bxmz
bymz
bzmz
bx2mz
by2mz
bz2mz
bxbymz
bxbzmz
bybzmz
betamz
beta2mz

hBixy
hB 2 ixy
hBx ixy
hBy ixy
hBz ixy
hBx2 ixy
hBy2 ixy
hBz2 ixy
hBx By ixy
hBx Bz ixy
hBy Bz ixy
hβixy
hβ 2 ixy

Module ‘density_stratified.f90’

drhomz
drho2mz

h∆ρ/ρ0 ixy
h(∆ρ/ρ0 )2 ixy

Module ‘gravity_simple.f90’

epotmz
epotuzmz

h̺Φgrav ixy
h̺Φgrav uz ixy

(potential energy flux)

Module ‘magnetic_shearboxJ.f90’

axmz
aymz
azmz
abuxmz
abuymz
abuzmz
uabxmz
uabymz
uabzmz
bbxmz
bbymz
bbzmz
bxmz
bymz
bzmz
jxmz
jymz
jzmz
Exmz
Eymz
Ezmz

hAx ixy
hAy ixy
hAz ixy
h(A · B)ux ixy
h(A · B)uy ixy
h(A · B)uz ixy
h(u · A)Bx ixy
h(u · A)By ixy
h(u · A)Bz ixy
hBx′ ixy
By′ xy
hBz′ ixy
hBx ixy
hBy ixy
hBz ixy
hJx ixy
hJy ixy
hJz ixy
hEx ixy
hEy ixy
hEz ixy

221

222

T HE P ENCIL C ODE

bx2mz
by2mz
bz2mz
bx2rmz
by2rmz
bz2rmz
beta1mz
betamz
beta2mz
jbmz
d6abmz
d6amz1
d6amz2
d6amz3
abmz
ubmz
uamz
uxbxmz
uybxmz
uzbxmz
uxbymz
uybymz
uzbymz
uxbzmz
uybzmz
uzbzmz
examz1
examz2
examz3
e3xamz1
e3xamz2
e3xamz3
etatotalmz
bxbymz
bxbzmz
bybzmz
b2mz
bf2mz
j2mz
poynzmz
epsMmz

hBx2 ixy
By2 xy
hBz2 ixy
hBx2 /̺ixy
By2 /̺ xy
hBz2 /̺ixy
h(B 2 /2µ0 )/pixy
hβixy
hβ 2 ixy
hJ · Bi |xy
h∇6 A · Bi |xy
h∇6 Aixy |x
h∇6 Aixy |y
h∇6 Aixy |z
hA · Bi |xy
hu · Bi |xy
hu · Ai |xy
hux bx i |xy
huy bx i |xy
huz bx i |xy
hux by i |xy
huy by i |xy
huz by i |xy
hux bz i |xy
huy bz i |xy
huz bz i |xy
hE × Aixy |x
hE × Aixy |y
hE × Aixy |z
hE hyper3 × Aixy |x
hE hyper3 × Aixy |y
hE hyper3 × Aixy |z
hηixy
hBx By ixy
hBx Bz ixy
hBy Bz ixy
B 2 xy
B ′2 xy
j 2 xy
Averaged poynting flux in z direction
ηµ0 j 2 xy
Module ‘meanfield.f90’

qpmz

hqp ixy
Module ‘shock_highorder.f90’
Module ‘temperature_idealgas.f90’

ppmz

hpixy

J.7 Parameters for ‘xzaver.in’

TTmz
ethmz
fpresxmz
fpresymz
fpreszmz
TT2mz
uxTmz
uyTmz
uzTmz
fradmz
fconvmz

223

hT ixy
heth ixy
h(∇p)x ixy
h(∇p)y ixy
h(∇p)z ixy
hT 2 ixy
hux T ixy
huy T ixy
huz T ixy
hFrad ixy
hcp ̺uz T ixy
Module ‘temperature_ionization.f90’

puzmz
pr1mz
eruzmz
ffakez
mumz
TTmz
ssmz
eemz
ppmz

hpuz ixy
hp/̺ixy
he̺uz ixy
h̺uz cp T ixy
hµixy
hT ixy
hsixy
heixy
hpixy
Module ‘thermal_energy.f90’

ppmz
TTmz

hpixy
hT ixy
Module ‘viscosity.f90’

fviscmz
fviscsmmz
epsKmz

J.7

h2ν̺ui Siz ixy (z-component of viscous flux)
h2νSmag ̺ui Siz ixy (z-component of viscous flux)
2ν̺S2 xy

List of parameters for ‘xzaver.in’

The following table lists possible inputs to the file ‘xzaver.in’. This list is not complete
and maybe outdated.
Variable

Meaning
Module ‘hydro.f90’

uxmy
uymy
uzmy
oumy
rhomy

hux ixz
huy ixz
huz ixz
hω · uixz
h̺ixz

Module ‘density.f90’

224

T HE P ENCIL C ODE
Module ‘entropy.f90’

ssmy
ppmy
TTmy
bxmy
bymy
bzmy
bx2my
by2my
bz2my
bxbymy
bxbzmy
bybzmy

hsixz
hpixz
hT ixz

Module ‘magnetic.f90’

hBx ixz
hBy ixz
hBz ixz
hBx2 ixz
By2 xz
hBz2 ixz
hBx By ixz
hBx Bz ixz
hBy Bz ixz

Module ‘density_stratified.f90’

drhomy
drho2my

h∆ρ/ρ0 ixz
h(∆ρ/ρ0 )2 ixz

Module ‘gravity_simple.f90’

epotmy

h̺Φgrav ixz

Module ‘magnetic_shearboxJ.f90’

bxmy
bymy
bzmy
bx2my
by2my
bz2my
bxbymy
bxbzmy
bybzmy

hBx ixz
hBy ixz
hBz ixz
hBx2 ixz
By2 xz
hBz2 ixz
hBx By ixz
hBx Bz ixz
hBy Bz ixz

Module ‘shock_highorder.f90’

Module ‘temperature_idealgas.f90’

ppmy
TTmy
ppmy
TTmy

J.8

hpixz
hT ixz

Module ‘thermal_energy.f90’

hpixz
hT ixz

List of parameters for ‘yzaver.in’

The following table lists possible inputs to the file ‘yzaver.in’. This list is not complete
and maybe outdated.

J.8 Parameters for ‘yzaver.in’

Variable

Meaning
Module ‘hydro.f90’

uxmx
uymx
uzmx
ruxmx
ruymx
ruzmx
rux2mx
ruy2mx
ruz2mx
ruxuymx
ruxuzmx
ruyuzmx
ux2mx
uy2mx
uz2mx
ox2mx
oy2mx
oz2mx
uxuymx
uxuzmx
uyuzmx
oumx
ekinmx
rhomx

hux iyz
huy iyz
huz iyz
h̺ux iyz
h̺uy iyz
h̺uz iyz
hρu2x iyz
hρu2y iyz
hρu2z iyz
hρux uy iyz
hρux uz iyz
hρuy uz iyz
hu2x iyz
u2y yz
hu2z iyz
hωx2 iyz
ωy2 yz
hωz2 iyz
hux uy iyz
hux uz iyz
huy uz iyz
hω · uiyz
h 12 ρu2 ixy

Module ‘density.f90’

h̺iyz
Module ‘entropy.f90’

ssmx
ss2mx
ppmx
TTmx
TT2mx
uxTTmx
uyTTmx
uzTTmx
fconvxmx
fradmx
fturbmx
Kkramersmx
dcoolx
fradx kramers

hsiyz
hs2 iyz
hpiyz
hT iyz
hT 2 iyz
hux T iyz
huy T iyz
huz T iyz
hcp ̺ux T iyz
hFrad iyz
h̺T χt ∇x siyz (turbulent heat flux)
K0 T ( 3 − b)/rho( a + 1) yz
surface cooling flux
Frad (from Kramers’ opacity)
Module ‘magnetic.f90’

b2mx

hB 2 iyz

225

226

T HE P ENCIL C ODE

bxmx
bymx
bzmx
bx2mx
by2mx
bz2mx
bxbymx
bxbzmx
bybzmx
betamx
beta2mx
etatotalmx

hBx iyz
hBy iyz
hBz iyz
hBx2 iyz
By2 yz
hBz2 iyz
hBx By iyz
hBx Bz iyz
hBy Bz iyz
hβiyz
hβ 2 iyz
hηiyz
Module ‘bfield.f90’

bmx
b2mx
bxmx
bymx
bzmx
bx2mx
by2mx
bz2mx
bxbymx
bxbzmx
bybzmx
betamx
beta2mx

hBiyz
hB 2 iyz
hBx iyz
hBy iyz
hBz iyz
hBx2 iyz
hBy2 iyz
hBz2 iyz
hBx By iyz
hBx Bz iyz
hBy Bz iyz
hβiyz
hβ 2 iyz

Module ‘density_stratified.f90’

drhomx
drho2mx

h∆ρ/ρ0 iyz
h(∆ρ/ρ0 )2 iyz

Module ‘gravity_simple.f90’

epotmx
epotuxmx

h̺Φgrav iyz
h̺Φgrav ux iyz

(potential energy flux)

Module ‘magnetic_shearboxJ.f90’

b2mx
bxmx
bymx
bzmx
bx2mx
by2mx
bz2mx
bxbymx
bxbzmx
bybzmx
betamx
beta2mx
etatotalmx

hB 2 iyz
hBx iyz
hBy iyz
hBz iyz
hBx2 iyz
By2 yz
hBz2 iyz
hBx By iyz
hBx Bz iyz
hBy Bz iyz
hβiyz
hβ 2 iyz
hηiyz

J.9 Parameters for ‘yaver.in’

227

Module ‘shock_highorder.f90’
Module ‘temperature_idealgas.f90’

ppmx
TTmx

hpiyz
hT iyz
Module ‘thermal_energy.f90’

ppmx
TTmx

hpiyz
hT iyz
Module ‘viscosity.f90’

fviscmx
numx

J.9

h2ν̺ui Six iyz (x-component of viscous flux)
hνiyz (yz-averaged viscosity)

List of parameters for ‘yaver.in’

The following table lists possible inputs to the file ‘yaver.in’. This list is not complete
and maybe outdated.
Variable

Meaning
Module ‘hydro.f90’

uxmxz
uymxz
uzmxz
ux2mxz
uy2mxz
uz2mxz
uxuymxz
uxuzmxz
uyuzmxz
oumxz

hux iy
huy iy
huz iy
hu2x iy
u2y y
hu2z iy
hux uy iy
hux uz iy
huy uz iy
hω · uiy
Module ‘density.f90’

rhomxz

h̺iy
Module ‘entropy.f90’

TTmxz
ssmxz

hT iy
hsiy
Module ‘magnetic.f90’

b2mxz
axmxz
aymxz
azmxz
bx1mxz

2

B y
hAx iy
hAy iy
hAz iy
h|Bx |iy

228

T HE P ENCIL C ODE

by1mxz
bz1mxz
bxmxz
bymxz
bzmxz
jxmxz
jymxz
jzmxz
bx2mxz
by2mxz
bz2mxz
bxbymxz
bxbzmxz
bybzmxz
uybxmxz
uybzmxz
Exmxz
Eymxz
Ezmxz
vAmxz

h|By |iy
h|Bz |iy
hBx iy
hBy iy
hBz iy
hJx iy
hJy iy
hJz iy
hBx2 iy
By2 y
hBz2 iy
hBx By iy
hBx Bz iy
hBy Bz iy
hUy Bx iy
hUy Bz iy
hEx iy
hEy iy
hEz iy
hvA2 iy
Module ‘density_stratified.f90’

drhomxz
drho2mxz

h∆ρ/ρ0 iy
h(∆ρ/ρ0 )2 iy

Module ‘magnetic_shearboxJ.f90’

b2mxz
axmxz
aymxz
azmxz
bx1mxz
by1mxz
bz1mxz
bxmxz
bymxz
bzmxz
jxmxz
jymxz
jzmxz
bx2mxz
by2mxz
bz2mxz
bxbymxz
bxbzmxz
bybzmxz
uybxmxz
uybzmxz
Exmxz

B2 y
hAx iy
hAy iy
hAz iy
h|Bx |iy
h|By |iy
h|Bz |iy
hBx iy
hBy iy
hBz iy
hJx iy
hJy iy
hJz iy
hBx2 iy
By2 y
hBz2 iy
hBx By iy
hBx Bz iy
hBy Bz iy
hUy Bx iy
hUy Bz iy
hEx iy

J.10 Parameters for ‘zaver.in’

Eymxz
Ezmxz
vAmxz

229

hEy iy
hEz iy
hvA2 iy
Module ‘meanfield.f90’

peffmxz
alpmxz

hPeff iy
hαiy
Module ‘temperature_idealgas.f90’

TTmxz
Emymxz

hT iy
hEmy iy

Emission in y-direction
Module ‘thermal_energy.f90’

TTmxz

J.10

hT iy

List of parameters for ‘zaver.in’

The following table lists possible inputs to the file ‘zaver.in’. This list is not complete
and maybe outdated.
Variable

Meaning
Module ‘hydro.f90’

uxmxy
uymxy
uzmxy
uxuymxy
uxuzmxy
uyuzmxy
oxmxy
oymxy
ozmxy
oumxy
pvzmxy
uguxmxy
uguymxy
uguzmxy
ruxmxy
ruymxy
ruzmxy
ux2mxy
uy2mxy
uz2mxy
rux2mxy
ruy2mxy
ruz2mxy
ruxuymxy
ruxuzmxy

hux iz
huy iz
huz iz
hux uy iz
hux uz iz
huy uz iz
hωx iz
hωy iz
hωz iz
hω · uiz
h(ωz + 2Ω)/̺iz
h(u · ∇u)x iz
h(u · ∇u)y iz
h(u · ∇u)z iz
hρux iz
hρuy iz
hρuz iz
hu2x iz
u2y z
hu2z iz
hρu2x iz
ρu2y z
hρu2z iz
hρux uy iz
hρux uz iz

(z component of potential vorticity)

230

T HE P ENCIL C ODE

ruyuzmxy
fkinxmxy
fkinymxy

hρuy uz iz
1
̺u2 ux
2
1
̺u2 uy
2

z
z

Module ‘density.f90’

rhomxy
TTmxy
ssmxy
uxTTmxy
uyTTmxy
uzTTmxy
gTxmxy
gTymxy
gTzmxy
gsxmxy
gsymxy
gszmxy
gTxgsxmxy
gTxgsymxy
gTxgszmxy
gTxgsx2mxy
gTxgsy2mxy
gTxgsz2mxy
fconvxy
fconvyxy
fconvzxy
fradxy Kprof
fradymxy Kprof
fradxy kramers
fturbxy
fturbymxy
fturbrxy
fturbthxy
dcoolxy

h̺iz

Module ‘entropy.f90’

hT iz
hsiz
hux T iz
huy T iz
huz T iz
h∇x T iz
h∇y T iz
h∇z T iz
h∇x siz
h∇y siz
h∇z siz
h(∇T
× ∇s)x iEz
D
(∇T × ∇s)y

z

(∇T × ∇s)2y

z

h(∇T × ∇s)z iz
2
D(∇T × ∇s)x Ez

(∇T × ∇s)2z z
hcp ̺ux T iz
hcp ̺uy T iz
hcp ̺uz T iz
Fxrad (x-component of radiative flux, z-averaged, from Kprof)
Fyrad (y-component of radiative flux, z-averaged, from Kprof)
Frad (z-averaged, from Kramers’ opacity)
h̺T χt ∇x siz
h̺T χt ∇y siz
h̺T χri ∇i siz (radial part of anisotropic turbulent heat flux)
h̺T χθi ∇i siz (latitudinal part of anisotropic turbulent heat
flux)
surface cooling flux
Module ‘magnetic.f90’

bxmxy
bymxy
bzmxy
jxmxy
jymxy
jzmxy
axmxy
aymxy
azmxy
bx2mxy

hBx iz
hBy iz
hBz iz
hJx iz
hJy iz
hJz iz
hAx iz
hAy iz
hAz iz
hBx2 iz

J.10 Parameters for ‘zaver.in’

by2mxy
bz2mxy
bxbymxy
bxbzmxy
bybzmxy
poynxmxy
poynymxy
poynzmxy
jbmxy
abmxy
examxy1
examxy2
examxy3
StokesImxy
StokesQmxy
StokesUmxy
StokesQ1mxy
StokesU1mxy
beta1mxy

By2 z
hBz2 iz
hBx By iz
hBx Bz iz
hBy Bz iz
hE × Bix
hE × Biy
hE × Biz
hJ · Biz
hA · Biz
hE × Aiz |x
hE × Aiz |y
hE × Aiz |z
hǫB⊥ iz |z
− hǫB⊥ cos 2χiz |z
− hǫB⊥ sin 2χiz |z
+ hF ǫB⊥ sin 2χiz |z
− hF ǫB⊥ cos 2χiz |z
B 2 /(2µ0 p) z |z

Module ‘density_stratified.f90’

drhomxy
drho2mxy

h∆ρ/ρ0 iz
h(∆ρ/ρ0 )2 iz

Module ‘gravity_simple.f90’

epotmxy
epotuxmxy

h̺Φgrav iz
h̺Φgrav ux iz

(potential energy flux)

Module ‘magnetic_shearboxJ.f90’

bxmxy
bymxy
bzmxy
jxmxy
jymxy
jzmxy
axmxy
aymxy
azmxy
bx2mxy
by2mxy
bz2mxy
bxbymxy
bxbzmxy
bybzmxy
poynxmxy
poynymxy
poynzmxy
jbmxy
abmxy
examxy1

hBx iz
hBy iz
hBz iz
hJx iz
hJy iz
hJz iz
hAx iz
hAy iz
hAz iz
hBx2 iz
By2 z
hBz2 iz
hBx By iz
hBx Bz iz
hBy Bz iz
hE × Bix
hE × Biy
hE × Biz
hJ · Biz
hA · Biz
hE × Aiz |x

231

232

T HE P ENCIL C ODE

examxy2
examxy3
StokesImxy
StokesQmxy
StokesUmxy
StokesQ1mxy
StokesU1mxy
beta1mxy

hE × Aiz |y
hE × Aiz |z
hǫB⊥ iz |z
− hǫB⊥ cos 2χiz |z
− hǫB⊥ sin 2χiz |z
+ hF ǫB⊥ sin 2χiz |z
− hF ǫB⊥ cos 2χiz |z
B 2 /(2µ0 p) z |z

Module ‘temperature_idealgas.f90’

TTmxy
Emzmxy

hT iz
hEmz iz

TTmxy

Module ‘thermal_energy.f90’

hT iz

fviscmxy
fviscsmmxy
fviscymxy

J.11

Emission in z-direction

Module ‘viscosity.f90’

h2ν̺ui Six iz (x-xomponent of viscous flux)
h2νSmag ̺ui Six iz (x-xomponent of viscous flux)
h2ν̺ui Siy iz (y-xomponent of viscous flux)

Boundary conditions

The following tables list all possible boundary condition labels that are documented.
J.11.1 Boundary condition bcx
Variable

Meaning
Module ‘boundcond.f90’

0
p
s
sf
ss
s0d
a
af
a2
a2v
a2r
cpc
cpp
cpz
spr
v

zero value in ghost zones, free value on boundary
periodic
symmetry, fN +i = fN −i ; implies f ′ (xN ) = f ′′′ (x0 ) = 0
symmetry with respect to interface
symmetry, plus function value given
symmetry, function value such that df/dx=0
antisymmetry, fN +i = −fN −i ; implies f (xN ) = f ′′ (x0 ) = 0
antisymmetry with respect to interface
antisymmetry relative to boundary value, fN +i = 2fN − fN −i ; implies
f ′′ (x0 ) = 0
set boundary value and antisymmetry relative to it fN +i = 2fN − fN −i ;
implies f ′′ (x0 ) = 0
sets d2 f /dr2 + 2df /dr − 2f /r2 = 0 This is the replacement of zero second
derivative in spherical coordinates, in radial direction.
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
spherical perfect conductor implies f ′′ + 2f ′ /R = 0 and f (xN ) = 0
vanishing third derivative

J.11 Boundary conditions

cop
1s
d1s
n1s
1so
cT
c1
Fgs
Fct
Fcm
sT
asT
f
fg
1
set
der
slo
slp
shx
shy
shz
dr0
ovr
out
e1o
ant
e1
e2
e3
hat
jet
spd
sfr
sr1
nfr
nr1
sa2

233

copy value of last physical point to all ghost cells
onesided
onesided for 1st/2nd derivative in two first inner points, Dirichlet in
boundary point
onesided for 1st/2nd derivative in two first inner points, Neumann in
boundary point
onesided
constant temperature (implemented as condition for entropy s or temperature T )
constant temperature (or maybe rather constant conductive flux??)
Fconv = - chi t*rho*T*grad(s)
Fbot = - K*grad(T) - chi t*rho*T*grad(s)
F bot = −K ∗ grad(T ) −chit ∗ rho ∗ T ∗ grad(s)
symmetric temperature, TN −i = TN +i ; implies T ′ (xN ) = T ′′′ (x0 ) = 0
select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
“freeze” value, i.e. maintain initial
“freeze” value, i.e. maintain initial
f = 1 (for debugging)
set boundary value to fbcx
set derivative on boundary to fbcx
set slope at the boundary = fbcx
set slope at the boundary and in ghost cells = fbcx
set shearing boundary proportional to x with slope=fbcx and abscissa=fbcx2
set shearing boundary proportional to y with slope=fbcx and abscissa=fbcx2
set shearing boundary proportional to z with slope=fbcx and abscissa=fbcx2
set boundary value [really??]
overshoot boundary condition ie (d/dx − 1/dist)f = 0.
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
allow outflow, but no inflow uses the e1 extrapolation scheme
stops and prompts for adding documentation
extrapolation [describe]
extrapolation [describe]
extrapolation in log [maintain a power law]
top hat jet profile in spherical coordinate.
top hat jet profile in cartezian coordinate.
sets d(rAα )/dr = fbcx(j)
stress-free boundary condition for spherical coordinate system.
Stress-free bc for spherical coordinate system. Implementation with
one-sided derivative.
Normal-field bc for spherical coordinate system. Some people call this
the “(angry) hedgehog bc”.
Normal-field bc for spherical coordinate system. Some people call this
the “(angry) hedgehog bc”. Implementation with one-sided derivative.
(d/dr)(rBφ ) = 0 imposes boundary condition on 2nd derivative of rAφ .
Same applies to θ component.

234

pfc
fix
fil
cfb
g
nil
ioc
s

T HE P ENCIL C ODE
perfect-conductor in spherical coordinate: d/dr(Ar ) + 2/r = 0.
set boundary value [really??]
set boundary value from a file
radial centrifugal balance
set to given value(s) or function
do nothing; assume that everything is set
inlet/outlet on western/eastern hemisphere in cylindrical coordinates
do nothing; assume that everything is set
implies f ′ (yN ) = f ′′′ (y0 ) = 0
Module ‘boundcond_alt.f90’

0
p
s
ss
s0d
a
a2
a2r
cpc
cpp
cpz
spr
v
cop
1s
1so
cT
c1
Fgs
Fct
Fcm
sT
asT
f
fg
1
set
der
slo
dr0
ovr
out
e1o
ant

zero value in ghost zones, free value on boundary
periodic
symmetry, fN +i = fN −i ; implies f ′ (xN ) = f ′′′ (x0 ) = 0
symmetry, plus function value given
symmetry, function value such that df/dx=0
antisymmetry, fN +i = −fN −i ; implies f (xN ) = f ′′ (x0 ) = 0
antisymmetry relative to boundary value, fN +i = 2fN − fN −i ; implies
f ′′ (x0 ) = 0
sets d2 f /dr2 + 2df /dr − 2f /r2 = 0 This is the replacement of zero second
derivative in spherical coordinates, in radial direction.
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
cylindrical perfect conductor implies f ′′ + f ′ /R = 0
spherical perfect conductor implies f ′′ + 2f ′ /R = 0 and f (xN ) = 0
vanishing third derivative
copy value of last physical point to all ghost cells
onesided
onesided
constant temperature (implemented as condition for entropy s or temperature T )
constant temperature (or maybe rather constant conductive flux??)
Fconv = - chi t*rho*T*grad(s)
Fbot = - K*grad(T) - chi t*rho*T*grad(s)
F bot = −K ∗ grad(T ) −chit ∗ rho ∗ T ∗ grad(s)
symmetric temperature, TN −i = TN +i ; implies T ′ (xN ) = T ′′′ (x0 ) = 0
select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
“freeze” value, i.e. maintain initial
“freeze” value, i.e. maintain initial
f = 1 (for debugging)
set boundary value to fbcx12
set derivative on boundary to fbcx12
set slope at the boundary = fbcx12
set boundary value [really??]
overshoot boundary condition ie (d/dx − 1/dist)f = 0.
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
allow outflow, but no inflow uses the e1 extrapolation scheme
stops and prompts for adding documentation

J.11 Boundary conditions

e1
e2
e3
hat
jet
spd
sfr
nfr
sa2
pfc
fix
fil
g
nil
ioc
s

235

extrapolation [describe]
extrapolation [describe]
extrapolation in log [maintain a power law]
top hat jet profile in spherical coordinate.
top hat jet profile in cartezian coordinate.
sets d(rAα )/dr = fbcx12(j)
stress-free boundary condition for spherical coordinate system.
Normal-field bc for spherical coordinate system. Some people call this
the “(angry) hedgehog bc”.
(d/dr)(rBφ ) = 0 imposes boundary condition on 2nd derivative of rAφ .
Same applies to θ component.
perfect-conductor in spherical coordinate: d/dr(Ar ) + 2/r = 0.
set boundary value [really??]
set boundary value from a file
set to given value(s) or function
do nothing; assume that everything is set
inlet/outlet on western/eastern hemisphere in cylindrical coordinates
do nothing; assume that everything is set
implies f ′ (yN ) = f ′′′ (y0 ) = 0

J.11.2 Boundary condition bcy
Variable

Meaning
Module ‘boundcond.f90’

sds
0
p
pp
yy
ap
s
ss
sds
cds
s0d
a
a2
v
v3
out
1s
d1s
n1s
cT
sT

symmetric-derivative-set
zero value in ghost zones, free value on boundary
periodic
periodic across the pole
Yin-Yang grid
anti-periodic across the pole
symmetry symmetry, fN +i = fN −i ;
symmetry, plus function value given
symmetric-derivative-set
complex symmetric-derivative-set
symmetry, function value such that df/dy=0
antisymmetry
antisymmetry relative to boundary value
vanishing third derivative
vanishing third derivative
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
onesided
onesided for 1st and 2nd derivative in two first inner points, Dirichlet
in boundary point
onesided for 1st and 2nd derivative in two first inner points, Neumann
in boundary point
constant temp.
symmetric temp.

236

asT
f
s+f
fg
fBs
fB
1
set
sse
sep
e1
e2
e3
der
cop
c+k
sfr
nfr
spt
pfc
nil’,’
sep
crk

T HE P ENCIL C ODE
select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
freeze value
freeze value
“freeze” value, i.e. maintain initial
frozen-in B-field (s)
frozen-in B-field (a2)
f=1 (for debugging)
set boundary value
symmetry, set boundary value
set boundary value
extrapolation
extrapolation
extrapolation in log [maintain a power law]
set derivative on the boundary
outflow: copy value of last physical point to all ghost cells
no-inflow: copy value of last physical point to all ghost cells, but suppressing any inflow
stress-free boundary condition for spherical coordinate system.
Normal-field bc for spherical coordinate system. Some people call this
the “(angry) hedgehog bc”.
spherical perfect conducting boundary condition along θ boundary f ′′ +
cot θf ′ = 0 and f (xN ) = 0
perfect conducting boundary condition along θ boundary
do nothing; assume that everything is set
set boundary value
no-inflow: copy value of last physical point to all ghost cells, but suppressing any inflow
Module ‘boundcond_alt.f90’

sds
0
p
pp
ap
s
ss
sds
cds
s0d
a
a2
v
v3
out
1s
cT
sT

symmetric-derivative-set
zero value in ghost zones, free value on boundary
periodic
periodic across the pole
anti-periodic across the pole
symmetry symmetry, fN +i = fN −i ;
symmetry, plus function value given
symmetric-derivative-set
complex symmetric-derivative-set
symmetry, function value such that df/dy=0
antisymmetry
antisymmetry relative to boundary value
vanishing third derivative
vanishing third derivative
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
onesided
constant temp.
symmetric temp.

J.11 Boundary conditions

asT
f
s+f
fg
1
set
sse
sep
e1
e2
e3
der
cop
c+k
sfr
nfr
spt
pfc
nil’,’
sep

237

select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
freeze value
freeze value
“freeze” value, i.e. maintain initial
f=1 (for debugging)
set boundary value
symmetry, set boundary value
set boundary value
extrapolation
extrapolation
extrapolation in log [maintain a power law]
set derivative on the boundary
outflow: copy value of last physical point to all ghost cells
no-inflow: copy value of last physical point to all ghost cells, but suppressing any inflow
stress-free boundary condition for spherical coordinate system.
Normal-field bc for spherical coordinate system. Some people call this
the “(angry) hedgehog bc”.
spherical perfect conducting boundary condition along θ boundary f ′′ +
cot θf ′ = 0 and f (xN ) = 0
perfect conducting boundary condition along θ boundary
do nothing; assume that everything is set
set boundary value

J.11.3 Boundary condition bcz
Variable

Meaning
Module ‘boundcond.f90’

0
p
yy
s
sf
s0d
0ds
a
a2
a2v
af
a0d
v
v3
1s
d1s
n1s

zero value in ghost zones, free value on boundary
periodic
Yin-Yang grid
symmetry
symmetry with respect to interface
symmetry, function value such that df/dz=0
symmetry, function value such that df/dz=0
antisymmetry
antisymmetry relative to boundary value
set boundary value and antisymmetry relative to it
antisymmetry with respect to interface
antisymmetry with zero derivative
vanishing third derivative
vanishing third derivative
one-sided
onesided for 1st and 2nd derivative in two first inner points, Dirichlet
in boundary point
onesided for 1st and 2nd derivative in two first inner points, Neumann
in boundary point

238

a1s

fg
c1
c1s
Fgs
Fct
c3
pfe
p1D
pot
pwd
hds
cT
cT1
cT2
cT3
hs
hse
cp
sT
ctz
cdz
ism
asT
c2
db
ce
e1
e2
ex
exf
exd
exm
b1
b2
b3
f’,’fa
fs
fBs
fB
g
1
StS
set
der
div

T HE P ENCIL C ODE
special for perfect conductor with const alpha and etaT when A considered as B; one-sided for 1st and 2nd derivative in two first inner
points
“freeze” value, i.e. maintain initial
complex
complex
Fconv = - chi t*rho*T*grad(s)
Fbot = - K*grad(T) - chi t*rho*T*grad(s)
constant flux at the bottom with a variable hcond
potential field extrapolation
potential field extrapolation in 1D
potential magnetic field
a variant of ’pot’ for nprocx=1
hydrostatic equilibrium with a high-frequency filter
constant temp.
constant temp.
constant temp. (keep lnrho)
constant temp. (keep lnrho)
hydrostatic equilibrium
hydrostatic extrapolation rho or lnrho is extrapolated linearily and the
temperature is calculated in hydrostatic equilibrium.
constant pressure
symmetric temp.
for interstellar runs copy T
for interstellar runs limit rho
for interstellar runs limit rho
select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
complex
complex
complex
extrapolation
extrapolation
simple linear extrapolation in first order
simple linear extrapolation in first order
simple linear extrapolation in first order
simple linear extrapolation in first order
extrapolation with zero value (improved ’a’)
extrapolation with zero value (improved ’a’)
extrapolation with zero value (improved ’a’)
freeze value + antisymmetry
freeze value + symmetry
frozen-in B-field (s)
frozen-in B-field (a2)
set to given value(s) or function
f=1 (for debugging)
solar surface boundary conditions
set boundary value
set derivative on the boundary
set the divergence of u to a given value use bc = ’div’ for iuz

J.11 Boundary conditions

ovr
inf
ouf
in
out
in0
ou0
ind
oud
ubs
win
cop
nil

set boundary value
allow inflow, but no outflow
allow outflow, but no inflow
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards relaxes to vanishing 1st derivative at boundary
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards relaxes to vanishing 1st derivative at boundary
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards creates inwards pointing or zero 1st derivative at boundary
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards creates outwards pointing or zero 1st derivative at boundary
copy boundary outflow,
forces massflux given as Σρi (ui + u0 ) = fbcz1/2(ρ)
copy value of last physical point to all ghost cells
do nothing; assume that everything is set
Module ‘boundcond_alt.f90’

cfb
fBs
fB
0
p
s
sf
s0d
0ds
a
a2
af
a0d
v
v3
1s
fg
c1
Fgs
Fct
c3
pfe
p1D
pot
pwd
hds
cT
cT2
cT3

239

radial centrifugal balance
frozen-in B-field (s)
frozen-in B-field (a2)
zero value in ghost zones, free value on boundary
periodic
symmetry
symmetry with respect to interface
symmetry, function value such that df/dz=0
symmetry, function value such that df/dz=0
antisymmetry
antisymmetry relative to boundary value
antisymmetry with respect to interface
antisymmetry with zero derivative
vanishing third derivative
vanishing third derivative
one-sided
“freeze” value, i.e. maintain initial
complex
Fconv = - chi t*rho*T*grad(s)
Fbot = - K*grad(T) - chi t*rho*T*grad(s)
constant flux at the bottom with a variable hcond
potential field extrapolation
potential field extrapolation in 1D
potential magnetic field
a variant of ’pot’ for nprocx=1
hydrostatic equilibrium with a high-frequency filter
constant temp.
constant temp. (keep lnrho)
constant temp. (keep lnrho)

240

hs
hse
cp
sT
ctz
cdz
asT
c2
db
ce
e1
e2
ex
exf
exd
exm
b1
b2
b3
f’,’fa
fs
fBs
fB
g
1
StS
set
der
div
ovr
inf
ouf
in
out
in0
ou0
ind
oud
ubs
win
cop
nil

T HE P ENCIL C ODE
hydrostatic equilibrium
hydrostatic extrapolation rho or lnrho is extrapolated linearily and the
temperature is calculated in hydrostatic equilibrium.
constant pressure
symmetric temp.
for interstellar runs copy T
for interstellar runs limit rho
select entropy for uniform ghost temperature matching fluctuating
boundary value, TN −i = TN =; implies T ′ (xN ) = T ′ (x0 ) = 0
complex
complex
complex
extrapolation
extrapolation
simple linear extrapolation in first order
simple linear extrapolation in first order
simple linear extrapolation in first order
simple linear extrapolation in first order
extrapolation with zero value (improved ’a’)
extrapolation with zero value (improved ’a’)
extrapolation with zero value (improved ’a’)
freeze value + antisymmetry
freeze value + symmetry
frozen-in B-field (s)
frozen-in B-field (a2)
set to given value(s) or function
f=1 (for debugging)
solar surface boundary conditions
set boundary value
set derivative on the boundary
set the divergence of u to a given value use bc = ’div’ for iuz
set boundary value
allow inflow, but no outflow
allow outflow, but no inflow
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards relaxes to vanishing 1st derivative at boundary
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards relaxes to vanishing 1st derivative at boundary
allow inflow, but no outflow forces ghost cells and boundary to not point
outwards creates inwards pointing or zero 1st derivative at boundary
allow outflow, but no inflow forces ghost cells and boundary to not point
inwards creates outwards pointing or zero 1st derivative at boundary
copy boundary outflow,
forces massflux given as Σρi (ui + u0 ) = fbcz1/2(ρ)
copy value of last physical point to all ghost cells
do nothing; assume that everything is set

J.12 Initial condition parameter dependence

J.12

241

Initial condition parameter dependence

zero
gaussian-noise
gaussian-noise-x
xjump
Beltrami-x
Beltrami-y
Beltrami-z
trilinear-x
trilinear-y
trilinear-z
cos-cos-sin-uz
tor pert
trilinear-x
sound-wave
shock-tube
bullets
Alfven-circ-x
const-ux
const-uy
tang-discont-z
Fourier-trunc
up-down

•
•
•

⋄

•
•
•
•
•
•
•
•
•
•
•

⋄
⋄

•

•

•

⋄

•
•
⋄

•

•

⋄

•

⋄

•

•

•

•

kz uu

•

ky uu

uy right

•

kx uu

uy left

•

uu lower

uu right

•

uu upper

uu left

urand

widthuu

inituu

ampluu

The following tables list which parameters from each Namelist are required (•), optional
(⋄) or irrelevant (blank). The distinction is made between required and optional where by
a parameter requires a setting if the default value would give an invalid or degenerate
case for the initial condition.

•

•

•

•

•

•
•

•

radius ss

242

initss

zero

const ss

blob

isothermal

Ferrière

xjump

hor-fluxtube

hor-tube

sedov

sedov-dual

isobaric

isentropic

linprof

piecew-poly

polytropic

ampl ss

•

widthss

•

•

epsilon ss
grads0

•

ss left

•

ss right
•

ss const
mpoly0
mpoly1
mpoly2
isothtop
khor ss

•

center1 y

•

center1 z

•

center2 x

•

center2 y

•

center2 z

•

thermal background

•

•

center1 x

T HE P ENCIL C ODE

pertss

•
•
•

REFERENCES

243

References
[1] Abramowitz, A., Stegun, I. A., Pocketbook of Mathematical Functions, Harri
Deutsch, Frankfurt (1984)
[2] Brandenburg, A., Astrophys. J. 550, 824–840 (2001) “The inverse cascade and nonlinear alpha-effect in simulations of isotropic helical hydromagnetic turbulence”
[3] Brandenburg, A., in Advances in non-linear dynamos, ed. A. Ferriz-Mas &
M. Núñez Jiménez, (The Fluid Mechanics of Astrophysics and Geophysics,
Vol. 9) Taylor & Francis, London and New York, pp. 269–344 (2003);
http://arXiv.org/abs/astro-ph/0109497
[4] Brandenburg, A., Dobler, W., Astron. Astrophys. 369, 329–338 (2001) “Large scale
dynamos with helicity loss through boundaries”
[5] Brandenburg, A., & Hazlehurst, J., Astron. Astrophys. 370, 1092–1102 (2001) “Evolution of highly buoyant thermals in a stratified layer”
[6] Brandenburg, A., & Sarson, G. R., Phys. Rev. Lett. 88, 055003 (2002) “The effect of
hyperdiffusivity on turbulent dynamos with helicity”
[7] Brandenburg, A., Dobler, W., & Subramanian, K., Astron. Nachr. 323, 99–122
(2002) “Magnetic helicity in stellar dynamos: new numerical experiments”
[8] Brandenburg, A., Enqvist, K., & Olesen, P., Phys. Rev. D 54, 1291–1300 (1996)
“Large-scale magnetic fields from hydromagnetic turbulence in the very early universe”
[9] Brandenburg, A., Jennings, R. L., Nordlund, Å., Rieutord, M., Stein, R. F., & Tuominen, I., J. Fluid Mech. 306, 325–352 (1996) “Magnetic structures in a dynamo simulation”
[10] A. Brandenburg, T. Kahniashvili, S. Mandal, A. Roper Pol, A. G. Tevzadze, and T.
Vachaspati, Phys. Rev. D 96, 123528 (2017) “Evolution of hydromagnetic turbulence
from the electroweak phase transition”
[11] Brandenburg, A., Moss, D., & Shukurov, A., MNRAS 276, 651–662 (1995) “Galactic
fountains as magnetic pumps”
[12] Brandenburg, A., Nordlund, Å., Stein, R. F., & Torkelsson, U., Astrophys. J. 446,
741–754 (1995) “Dynamo-generated turbulence and large scale magnetic fields in a
Keplerian shear flow”
[13] Collatz, L., The numerical treatment of differential equations, Springer-Verlag, New
York, p. 164 (1966)
[14] Dobler, W., Stix, M., & Brandenburg, A.: 2006, “Convection and magnetic field generation in fully convective spheres,” Astrophys. J. 638, 336-347
[15] Durrer, R., “The Cosmic Microwave Background,” Cambridge University Press
(2008)
[16] Gammie, C. F., Astrophys. J. 553, 174–183 (2001) “Nonlinear outcome of gravitational instability in cooling, gaseous disks”
[17] Goodman, J., Narayan, R. & Goldreich, P., Month. Not. Roy. Soc. 225, 695–711
(1987) “The stability of accretion tori – II. Nonlinear evolution to discrete planets”

244

T HE P ENCIL C ODE

[18] Haugen, N. E. L., & Brandenburg, A. Phys. Rev. E 70, 026405 (2004) “Inertial range
scaling in numerical turbulence with hyperviscosity”
[19] Hockney, R. W., & Eastwood, J. W., Computer Simulation Using Particles, McGrawHill, New York (1981)
[20] Hurlburt, N. E., Toomre, J., & Massaguer, J. M., Astrophys. J. 282, 557–573 (1984)
“Two-dimensional compressible convection extending over multiple scale heights”
[21] Kim, J., Moin, P. & Moser, R J. of Fluid Mech. 177, 133 (1987) “Turbulence statistics
in fully developed channel flow at low Reynolds number”
[22] Kippenhahn, R. & Weigert, A. Stellar structure and evolution, Springer: Berlin
(1990)
[23] Krause, F., Rädler, K.-H., Mean-Field Magnetohydrodynamics and Dynamo Theory,
Akademie-Verlag, Berlin; also Pergamon Press, Oxford (1980)
[24] Lele, S. K., J. Comp. Phys. 103, 16–42 (1992) “Compact finite difference schemes
with spectral-like resolution”
[25] Nordlund, Å., & Galsgaard, K., A 3D MHD code for Parallel Computers,
http://www.astro.ku.dk/~aake/NumericalAstro/papers/kg/mhd.ps.gz (1995)
[26] Nordlund, Å., Stein, R. F., Comput. Phys. Commun. 59, 119 (1990) “3-D simulations
of solar and stellar convection and magnetoconvection”
[27] Press, W., Teukolsky, S., Vetterling, W., & Flannery, B., Numerical Recipes in Fortran 90, 2nd ed., Cambridge (1996)
[28] Stanescu, D., Habashi, W. G., J. Comp. Phys. 143, 674 (1988) “2N -storage low dissipation and dispersion Runge–Kutta schemes for computational acoustics”
[29] Williamson, J. H., J. Comp. Phys. 35, 48 (1980) “Low-storage Runge–Kutta
schemes”

245

Part IV

Indexes

File Index

cparam.inc . . 14, 32, 78
2d-tests/spherical_*.c . . . . . . . . . . . . . . 12
viscous_ring 105
*.f90 . . . . . . . . . . . . 12
cparam.local . . . 6, 12,
*.in . . . . . . . . . 32, 171
14, 22, 41, 49, 54,
aa[xyz].{xz,yz,xy,xy2}
*.local . . . . . . . . . . 32
100, 145
215
../32c . . . . . . . . . . 145
cparam_pencils.inc 102
ab[xyz].{xz,yz,xy,xy2}
./config . . . . . . . . . . 18
ctimeavg.local . 12, 30,
215
./host-ID . . . . . . . . . 18
31
adapt-mkfile
.
.
.
.
.
.
22
.adapt-mkfile.inc . . 77
advective_gauge.f90186
data/7, 14, 27, 33, 50, 86
.bashrc . . . . . . . . . . . 5
alive.info
.
.
.
.
.
.
.
173
data/ . . . . . . . . . . . 15
.conf . . . . . . . . . . . . 19
anelastic.f90
187,
216
data/dim.dat . . . . . 165
.cshrc . . . . . . . . . . . . 5
auto-test
.
.
.
.
.
.
.
.
2,
9
data/param.nml 86, 146,
.emacs . . . . . . . . . . 125
165
.svn/ . . . . . . . . . . . . 12
b2.{xz,yz,xy,xy2} . 215
data/procN
/ . . . . . . 25
/scratch/ . . . . . . . . . . 7
bb.net . . . . . . . . . . . 42
data/proc*/ . . . . 27, 28
 . . . . . . . . . . . . . 17
bb[xyz].{xz,yz,xy,xy2}
data/proc*/alive.info
$HOME/.pencil/config-local
215
26
18
beta1.{xz,yz,xy,xy2}
data/proc*/slice* . . 27
$HOME/.pencil/host-ID
215
data/proc*/var.dat . 40
18
bfield.f90 187, 221, 226
datadir.in . . . 7, 14, 15
$PENCIL_HOME . . . . . . 87
bin/ 6, 9, 12, 14, 31, 33,
debug_c.c . . . . . . . . 109
$PENCIL_HOME/config 18,
78
density.f90 . . . . 31, 66,
19
boundcond.f90 232, 235,
95, 181, 215, 216,
$PENCIL_HOME/config-local
237
218, 223, 225,
18
boundcond_alt.f90 234,
227, 230
$PENCIL_HOME/config/compilers/*.conf
236, 239
density_stratified.f90
17
bugs/ . . . . . . . . . . . . 12
189, 221, 224,
$PENCIL_HOME/config/hosts
226, 228, 231
17
cdata.f90 . 89, 108, 178,
$PENCIL_HOME/config/hosts/.conf216
detonate.f90 . . . . . 189
17
dim.dat . . . . 15, 16, 32
chemistry.f90 . . . . 188
$PENCIL_HOME/config/hosts/IWF/host-andromeda-GNU_divu.{xz,yz,xy,xy2}215
chemistry_simple.f90
Linux-Linux.conf
188
doc/ . . . . . . . . . . . . . 12
17
chiral_fluids.f90 . 188
dustdensity.f90 . . . 189
$PENCIL_HOME/config/hosts/pencil.org/workhorse.pencil.org.conf
chiral_fluids_dx/ . . . . . . . . . . . . . . 12
17
gradtheta.f90
dx/basic . . . . . . . . . . 41
$PENCIL_HOME/host-ID18
188
dx/macros/ . . . . . . . . 32
$PENCIL_HOME/python 46
chiral_mhd.f90 . . . 188
ec.{xz,yz,xy,xy2} . 215
${PENCIL_HOME}/config-local/hosts/
config/ . . . . . . . . . . 18
entropy.f9031, 109, 181,
106
ConfigFinder . . . 17, 18
215, 216, 219,
~/.idl_history . . . . 44
configure . . . . . . . . . 88
224, 225, 227, 230
~/pencil-auto-test 106
conv-slab/ 6, 9, 49, 148,
entropy_anelastic.f90
1D_loop.f90 . . . . . . 186
149
189, 216
1d-test/ambipolar-diffusion conv-slab/src/ . . . . 11
eos_ionization.f90 215
66
coronae.f90 . . . . . . 189
Equ . . . . . . . . . . . . . . 95
2d-tests/baroclinic105
cosmicray.f90 . . . . 215
equ.f90 . . . . . . . . . . 98
cosmicray_current.f90
2d-tests/battery_term
examples/pro/ . . . . . 42
189
85
246

FILE INDEX
experimental . . . . . 126
forced/ . . . . . . . . . .
forced/idl/ . . . . . . .
fourier_fftpack.f90
fourier_transform_y

12
55
53
53

generate_kvectors.pro
148
getconf.csh 9, 12, 14, 33
gravitational_waves.f90 . . 190
gravitational_waves_hij6.f90 . . . 190
gravity_simple.f90 190,
221, 224, 226, 231
gravz/ . . . . . . . . . . . 12
grid.dat . . . . . . . 16, 23
heatflux.f90 . . . . . 191
helical-MHDturb 35, 148
host_ID.conf . . . . . . 17
hosts/ . . . . . . . . . . . 18
hsections.pro . . . . 149
hydro.f90 . . . 31, 66, 76,
95, 100, 178, 214,
216, 217, 223,
225, 227, 229
hyperresi_strict_2nd
137
hypervisc_strict_2nd
137
idl/ . . . . . . . . . . 12, 43
index.pro . . . . . . 15, 32
Intel.conf . . . . . . . . 17
Intel_MPI.conf . . . . 17
interlocked-fluxrings
105
interstellar.dat . . . 15
Isurf.xz . . . . . . . . . 215
j2.{xz,yz,xy,xy2} . 215
jb.{xz,yz,xy,xy2} . 215
jj[xyz].{xz,yz,xy,xy2}
215
k.dat . . . . . . . . 32, 148
K_VECTORS/ . . . . . . . 148
kinematic/ . . . . . . . . 12
legend.dat . . . . . . . . 15
lncc.{xz,yz,xy,xy2}215

lnrho.{xz,yz,xy,xy2}
215
lnrho.net . . . . . . . . . 42
lnTT.{xz,yz,xy,xy2}215
lorenz_gauge.f90 . . 191
mach.{xz,yz,xy,xy2}215
magnetic.f90 viii, 31, 96,
98, 99, 109, 182,
215, 216, 219,
224, 225, 227, 230
magnetic_shearboxJ.f90
191, 217, 221,
224, 226, 228, 231
Makefile . 13, 21, 22, 31,
32, 78, 79, 122
Makefile.local . viii, 6,
12–14, 32, 49, 52,
54, 62, 64, 77, 78,
88
Makefile.src . 6, 12, 14,
21, 78
meanfield.f90 195, 222,
229
meanfield_demfdt.f90
195
mkcparam . . . 14, 78, 102
mpicomm.f90 . . . . . . . 79
neutraldensity.f90 . 66
neutralvelocity.f90 66
NEWDIR . . . . . . . . 34, 83
nochiral . . . . . . . . . . 78
noentropy.f90 . . . . 196
NOERASE . . . . . . . . . . 86
noinitial_condition.f90
104
noionization.f90 . . 174
nomagnetic.f90 viii, 98,
99
nompicomm.f90 . 79, 111
nospecial.f90 . . . . 103
o2.{xz,yz,xy,xy2} . 215
oo[xyz].{xz,yz,xy,xy2}
215
os/Unix.conf . . . . . . 19
param.nml . . 15, 32, 145
param2.nml . . . . . 15, 32
params.log . . 15, 33, 34

247

particles_chemistry.f90
196
particles_dust.f90 196
particles_dust_brdeplete.f90
196
particles_lagrangian.f90
196
particles_mass_swarm.f90 . . 196
particles_surfspec.f90
196
pc_read_phiavg.pro . 30
pc_read_video . . . . . 28
pc_read_xyaver . . . . 29
pc_setupsrc . . . . . . . . 6
pencil-code/ 11, 12, 87
pencil-code/doc/manual.tex
90, 104
pencil-code/idl/read 8
pencil-code/license/developers.
3, 91
pencil-code/utils . . 33
pencil-runs/ . . . . . . 12
Pencil::ConfigFinder17
Pencil::ConfigParser17
pencil_check.f90 . . . 82
phiaver.in . . . . 29, 216
phiaverages.dat . . . . 29
PHIAVGN . . . . . . . . . . 30
phiavg.pro . . . . . . . . 30
polymer.f90 . . . . . . 197
power . . . . . . . . . . . . 53
power.pro . . . . . . . . . 54
power_kin.dat . . . . . 52
power_krms.dat . . . . 53
powerbx_x.dat . . . . . 52
powerhel_mag.dat . . . 52
powerux_x.dat . . 52, 53
Poynting[xyz].{xz,yz,xy,xy2}
215
pp.{xz,yz,xy,xy2} . 215
print.in . . 8, 14, 25, 98,
99, 146, 178
procN . . . . . . 15, 32, 33
proc0 . . . . . . . . . . . . 15
proc1 . . . . . . . . . . . . 15
procN/ . . . . . . . . . . 108
pscalar.f90 . . 186, 215
Qrad.{xz,yz,xy,xy2}215

248
r.pro . . . . . . 44, 45, 50
radiation_ray.f90 . 215
rall.pro . . . . 44, 45, 50
README . . . 32, 107, 148
reference.out . . . 8, 14
RELOAD . . . . . 15, 33, 34
remesh.csh . . . . . . . 164
RERUN . . . . . . . . . 34, 83
rho.{xz,yz,xy,xy2} 215
rings/ . . . . . . . . . . . 12
run.csh 9, 12, 14, 31, 33,
34, 83
run.in . . . . . . . . viii, 8,
14, 26–28, 30–32,
34, 37–40, 52, 54,
82, 100, 145, 151,
171
run.pro . . . . . . . . . . 32
run.x . 8, 15, 31, 39, 107
runA_32a . . . . . . . . . . 22
runs/ . . . . . . . . . . . . 12
rvid_box . . . . . . . . . . 27
rvid_box.pro . . . . . . 27
rvid_line.pro . . . . . 27
rvid_plane.pro . . . . 27
samples/ . . 2, 3, 5, 9, 14,
119
samples/1d-tests/H2_flamespeed . . 38
samples/dust-vortex 69
samples/parameter_scan
32
samples/README . . . . 12
samples/sedimentation/
56
SCRATCH_DIR . . . . . . . 83
seed.dat . . . . . . . 16, 40
sfb-1.dat . . . . . . . . . 54
sfu-1.dat . . . . . . . . . 54
sfz1-1.dat . . . . . . . . 54
shear.f90 . . . . . . . . 197
shock.{xz,yz,xy,xy2}
215
shock.f90 . . . . 197, 215
shock_highorder.f90
197, 222, 224, 227
slice . . . . . . . . . . . . 86
slice_uu1.yz . . . . . . 27
solar_corona.f90 . . 197

T HE P ENCIL C ODE

solid_cells_CGEO.f90
197
solid_cells_reactive.f90
197
sound-spherical-noequi
24
sourceme . . . . . . . . 5, 75
sourceme.csh . 5, 12, 75,
86
sourceme.sh 5, 12, 75, 86
SPEED . . . . . . . . . . . . 33
spher/ . . . . . . . . . . . 12
src/ viii, 3, 6, 11, 12, 14,
33, 78
src/ . . . . . . . . . . . . 14
src/*.local . . . . . . . 32
src/cparam.local . . 145
src/Makefile.inc . . . 52
src/Makefile.local 38,
104, 145, 150
src/read_all_videofiles.x 27
src/read_videofiles.x
27, 28
ss.{xz,yz,xy,xy2} . 215
ss.net . . . . . . . . . . . 42
start.csh . 9, 12, 14, 31,
33, 39, 66, 82
start.in . . . . . . . viii, 8,
14, 24, 28, 32, 34,
35, 37, 39, 41, 59,
82, 100, 104, 146,
151, 163, 164, 171
start.pro . . . 32, 44, 86
start.x 8, 15, 39, 41, 77,
82, 107, 175
STOP . . . . . . . . . . 33, 34
structure.pro . . . . . 55
sub.f90 . . . . . . . . . . 96
TAVGN . . . . . . . . . . . . 31
temperature_idealgas.f90
197, 215, 222,
224, 227, 229, 232
temperature_ionization.f90
198, 223
testfield_axisym.f90
198
testfield_axisym2.f90
199

testfield_axisym4.f90
199
testfield_compress_z.f90 . . . . . 199
testfield_meri.f90 202
testfield_nonlin_z.f90
203
testfield_x.f90 . . . 205
testfield_xz.f90 . . 207
testfield_z.f90 . . . 207
testflow_z.f90 . . . 209
testperturb.f90 . . . 209
testscalar.f90 . . . 210
testscalar_axisym.f90
211
testscalar_simple.f90
212
thermal_energy.f90 214,
223, 224, 227,
229, 232
time.dat . . . . . . . 16, 40
time_series.dat . 8, 14,
15, 25, 32, 33, 41,
44
timeavg.dat . . . . . . . 31
top.log . . . . . . . . . . 33
ts.pro . . . . 42, 44, 149
tsnap.dat . . . . . 15, 172
TT.{xz,yz,xy,xy2} . 215
tvid.dat . . . . . . . . . . 15
u2.{xz,yz,xy,xy2} . 214
u[xyz].{xz,yz,xy,xy2}
214
uu.net . . . . . . . . . . . 42
VAR . . . . . . . 40, 86, 173
VARN 15, 16, 26, 32, 33,
36, 172
var.dat . . . . . . . . . . 8,
15, 16, 25, 31, 32,
34, 36, 40, 42, 45,
86, 144, 164, 172,
173
var.general . . . . . . . 41
VAR0 . . . . . . . . . . . . 164
VAR1 . . . . . . . . . . . . 144
VAR# . . . . . . . . . . . . . 34
video.in 26–28, 146, 214
visc_smagorinsky.f90
214

FILE INDEX
viscosity.f90 214, 223,
227, 232
vsections.pro . . . . 150
vsections2.pro . . . 150
X.xy . . . . . . . . . . . . . 16
X.xz . . . . . . . . . . . . . 16

X.yz . . . . . . . . . . . . . 16
xyaver.in . 29, 146, 217
xyaverages.dat . . . . 29
xzaver.in . . . . . 29, 223
xzaverages.dat . . . . 29
yaver.in . . . . . . 29, 227

249
yaverages.dat . . . . . 29
yH.{xz,yz,xy,xy2} . 215
yzaver.in . . . . . 29, 224
yzaverages.dat . . . . 29
zaver.in . . . . . . 29, 229
zaverages.dat . . . . . 29

Variable Index

.true. .
0ds . .
1s . . .
1so . .

.
.
.
.

.
.
.
.

.
.
.
.

. . . . . . . . . 29
. . . . 237, 239
233–237, 239
. . . . 233, 234

0 . . . . 232, 234–237, 239
1 233, 234, 236–238, 240
a . . . . 232, 234–237, 239
a0d . . . . . . . . . 237, 239
a11xy . . . . . . . . . . . 202
a12xy . . . . . . . . . . . 202
a13xy . . . . . . . . . . . 202
a1s . . . . . . . . . . . . . 238
a2 . . . 232, 234–237, 239
a21xy . . . . . . . . . . . 202
a22xy . . . . . . . . . . . 202
a23xy . . . . . . . . . . . 202
a2m . . . . . . . . 184, 193
a2r . . . . . . . . . 232, 234
a2v . . . . . . . . . 232, 237
a31xy . . . . . . . . . . . 202
a32xy . . . . . . . . . . . 202
a33xy . . . . . . . . . . . 202
aa . . . . . . . . . . . . . . 215
ab . . . . . . . . . . . . . . 215
ab int . . . . . . . 182, 191
ABC A . . . . . . . . . . 176
ABC B . . . . . . . . . . 176
ABC C . . . . . . . . . . 176
abm . . . . . . . . 182, 191
abmh . . . . . . . 182, 191
abmn . . . . . . . 182, 191
abms . . . . . . . 182, 191
abmxy . . . . . . . . . . 231
abmz . . . . . . . 220, 222
abrms . . . . . . . 182, 191
abumx . . . . . . 182, 191
abumy . . . . . . 182, 191
abumz . . . . . . 182, 191
abuxmz . . . . . . 219, 221
abuymz . . . . . . 219, 221
abuzmz . . . . . . 219, 221
accpowzdownmz . . . 218
accpowzmz . . . . . . . 218
accpowzupmz . . . . . 218
af . . . . . . 232, 237, 239

ajm . . . . . . . . . 182, 191
aklam . . . . . . . . . . . 209
aklamQ . . . . . . . . . 209
alp11 200, 203, 205, 207,
209
alp11 x . . . . . . . . . . 205
alp11 x2 . . . . . . . . . 206
alp11cc . . 200, 203, 205,
207
alp11x . . . . . . . . . . 206
alp11z . . . . . . . . . . 208
alp12 200, 203, 205, 207,
209
alp12 x . . . . . . . . . . 206
alp12 x2 . . . . . . . . . 206
alp12cs . . 200, 203, 205,
207
alp12x . . . . . . . . . . 206
alp12z . . . . . . . . . . 208
alp13 . . . . . . . . . . . 207
alp13z . . . . . . . . . . 208
alp21 200, 203, 205, 207,
209
alp21 x . . . . . . . . . . 205
alp21 x2 . . . . . . . . . 206
alp21sc . . 200, 203, 205,
207
alp21x . . . . . . . . . . 206
alp21z . . . . . . . . . . 208
alp22 200, 203, 205, 207,
209
alp22 x . . . . . . . . . . 206
alp22 x2 . . . . . . . . . 206
alp22ss . . 200, 203, 205,
207
alp22x . . . . . . . . . . 206
alp22z . . . . . . . . . . 208
alp23 . . . . . . . . . . . 207
alp23z . . . . . . . . . . 208
alp31 200, 203, 205, 207,
209
alp32 200, 203, 205, 207,
209
alpK . . . . . . . . 200, 203
alpM . . . . . . . 200, 203
alpm . . . . . . . . . . . . 195
alpMK . . . . . . 200, 203
250

alpmxz . . . . . . . . . . 229
alpPARA . . . . 198, 199
alpPARAz . . . . 198, 199
alpPERP . . . . 198, 199
alpPERPz . . . . 198, 199
amax . . . . . . . 184, 193
ambmz . . . . . . 185, 194
ambmzh . . . . . 185, 194
ambmzn . . . . . 185, 194
ambmzs . . . . . 185, 194
ampl ff . . . . . . . . . . 177
ampl forc . . . . . . . . 174
ampl ss . . . . . . . . . 168
amplaa . . . . . . . . . . 170
amplaa2 . . . . . . . . . 170
ampllncc . . . . . . . . 170
ampllncc2 . . . . . . . . 170
ampllnrho . . . . . . . 167
ampluu . . . . . . 128, 166
ampluu=1e-1 . . . . . 128
ant . . . . . . . . . 233, 234
ap . . . . . . . . . . 235, 236
apbrms . . . . . . . . . . 187
arms . . . . . . . . 184, 193
asT . 233, 234, 236–238,
240
axmxy . . . . . . . 230, 231
axmxz . . . . . . . 227, 228
axmz . . . . . . . 219, 221
axp2 . . . . . . . . 183, 193
axpt . . . . . . . . 183, 192
aymxy . . . . . . . 230, 231
aymxz . . . . . . . 227, 228
aymz . . . . . . . 219, 221
ayp2 . . . . . . . . 183, 193
aypt . . . . . . . . 183, 192
azmxy . . . . . . . 230, 231
azmxz . . . . . . . 227, 228
azmz . . . . . . . . 219, 221
azp2 . . . . . . . . 183, 193
azpt . . . . . . . . 183, 192
b0rms 201, 204, 206, 208
b1 . . . . . . . . . . 238, 240
b111xy . . . . . . . . . . 202
b112xy . . . . . . . . . . 203

VARIABLE INDEX
b11rms . . . . . . 201, 204,
206–208
b121xy . . . . . . . . . . 202
b122xy . . . . . . . . . . 203
b12rms . . 201, 204, 206,
208
b131xy . . . . . . . . . . 203
b132xy . . . . . . . . . . 203
b1b23m . . . . . 185, 194
b1b32m . . . . . 185, 194
b1m . . . . . . . . 182, 191
b1rms . . . . . . . 198, 199
b2 . . . . . . 215, 238, 240
b211xy . . . . . . . . . . 203
b212xy . . . . . . . . . . 203
b21rms . . . . . . 201, 204,
206–208
b221xy . . . . . . . . . . 203
b222xy . . . . . . . . . . 203
b22rms . . 201, 204, 206,
208
b231xy . . . . . . . . . . 203
b232xy . . . . . . . . . . 203
b2b13m . . . . . 185, 194
b2b31m . . . . . 185, 194
b2divum . . . . . 186, 195
b2m . . . . 182, 187, 191
b2mphi . . . . . . 216, 217
b2mx . . . . . . . 225, 226
b2mxz . . . . . . . 227, 228
b2mz . . . . . . . 220–222
b2rms . . . . . . . 198, 199
b2ruzm . . . . . . 182, 191
b2tm . . . . . . . . 182, 191
b2uzm . . . . . . 182, 191
b3 . . . . . . . . . . 238, 240
b311xy . . . . . . . . . . 203
b312xy . . . . . . . . . . 203
b321xy . . . . . . . . . . 203
b322xy . . . . . . . . . . 203
b331xy . . . . . . . . . . 203
b332xy . . . . . . . . . . 203
b3b12m . . . . . 185, 194
b3b21m . . . . . 185, 194
b3rms . . . . . . . 198, 199
b4m . . . . . . . . . . . . 182
B ext . . . . . . . . . . . . 176
bamp . . . 201, 205, 208
bb . . . . . . . . . . 109, 215
bbmphi . . . . . . 216, 217

bbsphmphi . . . 216, 217
bbxmax . . . . . . 184, 193
bbxmz . . . . . . . 219, 221
bbymax . . . . . . 184, 193
bbymz . . . . . . . 219, 221
bbzmax . . . . . . 184, 193
bbzmz . . . . . . . 220, 221
bc{x,y,z} . . . . . . . . . . 39
bcosphz . . . . . . 185, 194
bcx 39, 40, 83, 165, 173,
232
bcy . . . 39, 165, 173, 235
bcz 39, 40, 83, 165, 173,
237
beta1 . . . . . . . . . . . 215
beta1m . . . . . . 184, 193
beta1max . . . . 184, 193
beta1mxy . . . . 231, 232
beta1mz . . . . . 220, 222
beta2mx . . . . . . . . . 226
beta2mz . . . . . 220–222
betam . . . 184, 188, 193
betamax . 184, 188, 193
betamin . 184, 188, 193
betamx . . . . . . . . . . 226
betamz . . . . . . 220–222
betPARA . . . . . 198, 199
betPARAz . . . . 198, 199
betPERP . . . . . 198, 199
betPERP2 . . . . . . . . 199
betPERPz . . . . 198, 199
bf2m . . . . . . . . . . . . 184
bf2mz . . . . . . . 220, 222
bf4m . . . . . . . . . . . . 184
bfrms . . . . . . . 184, 193
bgmu5rms . . . 188, 189
bgtheta5rms . . . . . . 188
bjtm . . . . . . . . 182, 191
bm . . . . . . . . . . . . . 187
bm2 . . . . . . . . 182, 191
bmax . . . 184, 187, 193
bmin . . . . . . . . . . . . 187
bmx . . 29, 184, 193, 226
bmxy rms . . . . 186, 195
bmy . . . . . 29, 184, 193
bmz . . 29, 184, 193, 221
bmzA2 . . . . . . 184, 194
bmzph . . . . . . 185, 194
bmzphe . . . . . . 185, 194
bmzS2 . . . . . . 184, 194

251
bpmphi . . . . . . 216, 217
brmphi . . . . . . 216, 217
brms . . . . 183, 187, 193
brms=3.871E-01 . . . 128
brmsx . . . . . . . 186, 195
brmsz . . . . . . . 186, 195
brsphmphi . . . 216, 217
bsinphz . . . . . . 185, 194
bthmphi . . . . . 216, 217
bx0mz 202, 205, 206, 209
bx0pt . . . 200, 204, 208
bx11pt . . 200, 204, 208
bx12pt . . 200, 204, 208
bx1mxz . . . . . . 227, 228
bx1pt . . . . . . . 198, 199
bx21pt . . 200, 204, 208
bx22pt . . 200, 204, 208
bx2m . . . 185, 187, 194
bx2mx . . . . . . . . . . 226
bx2mxy . . . . . . 230, 231
bx2mxz . . . . . . . . . . 228
bx2my . . . . . . . . . . 224
bx2mz . . . . . . . 220–222
bx2pt . . . . . . . 198, 199
bx2rmz . . . . . . 220, 222
bx3pt . . . . . . . 198, 199
bxbym . . 184, 187, 193
bxbymx . . . . . . . . . . 226
bxbymxy . . . . . . . . . 231
bxbymxz . . . . . . . . . 228
bxbymy . . . . . . . . . . 224
bxbymz . . . . . . 220–222
bxbzm . . . . . . . . . . . 187
bxbzmx . . . . . . . . . . 226
bxbzmxy . . . . . . . . . 231
bxbzmxz . . . . . . . . . 228
bxbzmy . . . . . . . . . . 224
bxbzmz . . . . . . 220–222
bxm . . . . 184, 187, 193
bxmax . . 184, 187, 193
bxmin . . . . . . . 184, 193
bxmx . . . . . . . . . . . 226
bxmxy . . . . . . . 230, 231
bxmxz . . . . . . . . . . . 228
bxmy . . . . . . . . . . . 224
bxmz . . . . . . . 220, 221
bxp2 . . . . . . . . 183, 192
bxpt . . . . . . . . 183, 192
by0mz 202, 205, 206, 209
by0pt . . . 201, 204, 208

252
by11pt . . 200, 204, 208
by12pt . . 200, 204, 208
by1mxz . . . . . . . . . . 228
by21pt . . 200, 204, 208
by22pt . . 201, 204, 208
by2m . . . 185, 187, 194
by2mx . . . . . . . . . . 226
by2mxy . . . . . . . . . . 231
by2mxz . . . . . . . . . . 228
by2my . . . . . . . . . . 224
by2mz . . . . . . . 220–222
by2rmz . . . . . . 220, 222
bybzm . . . . . . . . . . . 187
bybzmx . . . . . . . . . . 226
bybzmxy . . . . . . . . . 231
bybzmxz . . . . . . . . . 228
bybzmy . . . . . . . . . . 224
bybzmz . . . . . . 220–222
bym . . . . 184, 187, 193
bymax . . 184, 187, 193
bymin . . . . . . . 184, 193
bymx . . . . . . . . . . . 226
bymxy . . . . . . . 230, 231
bymxz . . . . . . . . . . . 228
bymy . . . . . . . . . . . 224
bymz . . . . . . . 220, 221
byp2 . . . . . . . . 183, 192
bypt . . . . . . . . 183, 192
bz0mz 202, 205, 206, 209
bz1mxz . . . . . . . . . . 228
bz2m . . . 185, 187, 194
bz2mx . . . . . . . . . . . 226
bz2mxy . . . . . . . . . . 231
bz2mxz . . . . . . . . . . 228
bz2my . . . . . . . . . . . 224
bz2mz . . . . . . . 220–222
bz2rmz . . . . . . 220, 222
bzm . . . . 184, 187, 193
bzmax . . . 184, 187, 193
bzmin . . . . . . . 184, 193
bzmphi . . . . . . 216, 217
bzmx . . . . . . . . . . . 226
bzmxy . . . . . . . 230, 231
bzmxz . . . . . . . . . . . 228
bzmy . . . . . . . . . . . 224
bzmz . . . . . . . . 220, 221
bzp2 . . . . . . . . 183, 192
bzpt . . . . . . . . 183, 192
c+k . . . . . . . . . 236, 237
c1 . . . 233, 234, 238, 239

T HE P ENCIL C ODE
c1pt . . . . . . . . 211–213
c1rms . . . 210, 212, 213
c1s . . . . . . . . . . . . . 238
c2 . . . . . . . . . . 238, 240
c2pt . . . . . . . . 211–213
c2rms . . . 210, 212, 213
c3 . . . . . . . . . . 238, 239
c3pt . . . . . . . . 211–213
c3rms . . . 210, 212, 213
c4pt . . . . . . . . 211–213
c4rms . . . 210, 212, 213
c5pt . . . . . . . . 211–213
c5rms . . . 210, 212, 213
c6pt . . . . . . . . 211–213
c6rms . . . . . . . 211–213
ccglnrm . . . . . . . . . 186
ccmax . . . . . . . . . . . 186
cdiffrho . . . . . . . . . 174
cds . . . . . . . . . 235, 236
cdt . . . . . . . 38, 41, 172
cdts . . . . . . . . . . . . . 38
cdtv . . . . . . . . . 38, 172
cdz . . . . . . . . . 238, 240
ce . . . . . . . . . . 238, 240
cfb . . . . . . . . . 234, 239
chi . . . . . . . . . . . . . 175
chi t . . . . . . . . . . . . 175
coeff grid . . . . . . 23, 25
cool . . . . . . . . . . . . . 175
cooltype . . . . . . . . . 175
cop . 233, 234, 236, 237,
239, 240
cosjbm . . . . . . 186, 195
cosubm . . . . . . 183, 192
cp . . . . . . . . . . 238, 240
cpc . . . . . . . . . 232, 234
cpp . . . . . . . . . 232, 234
cpz . . . . . . . . . 232, 234
crk . . . . . . . . . . . . . 236
cs0 . . . . . . . . . 167, 174
cs2 . . . . . . . . . . . . . 109
cs2bot . . . . . . . 167, 174
cs2cool . . . . . . . . . . 175
cs2mphi . . . . . 216, 217
cs2top . . . . . . . 167, 174
csm . . . . . 181, 189, 198
cT . . . 233–236, 238, 239
cT1 . . . . . . . . . . . . . 238
cT2 . . . . . . . . . 238, 239
cT3 . . . . . . . . . 238, 239

ctz . . . . . .
curlru2mz .
cutoff=0 . .
cvsid . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

238, 240
. . . . 217
. . . . 128
164, 171

d1s . . . . . 233, 235, 237
d2davg . . . . . . . 30, 173
d2Lambrms . . . . . . 187
d2Lamrms . . . . . . . 187
d6abmz . . . . . 220, 222
d6amz1 . . . . . 220, 222
d6amz2 . . . . . 220, 222
d6amz3 . . . . . 220, 222
damp . . . . . . . 173, 174
dampu . . . . . . . . . . 173
dampuext . . . . . . . . 173
dampuint . . . . . . . . 173
datadir . . . . . . . . . . . 45
db . . . . . . . . . . 238, 240
dbx2m . . . . . . . . . . 187
dbxm . . . . . . . . . . . 187
dbxmax . . . . . . . . . 187
dby2m . . . . . . . . . . 187
dbym . . . . . . . . . . . 187
dbymax . . . . . . . . . 187
dbz2m . . . . . . . . . . 187
dbzm . . . . . . . . . . . 187
dbzmax . . . . . . . . . . 187
dcoolx . . . . . . . . . . . 225
dcoolxy . . . . . . . . . . 230
dcoolz . . . . . . . . . . . 219
del . . . . . . . . . 198, 199
del2 . . . . . . . . . . . . 199
deltay . . . . . . . . . . . 197
delz . . . . . . . . 198, 199
der . 233, 234, 236–238,
240
detn . . . . . . . . . . . . 189
dettot . . . . . . . . . . . 189
dexbmx . . . . . . 186, 195
dexbmy . . . . . . 186, 195
dexbmz . . . . . . 186, 195
dheat buffer1 . . . . . 175
div . . . . . . . . . 238, 240
divabrms . . . . . . . . 187
divapbrms . . . . . . . 187
divarms . . . . . 184, 193
divbmax . . . . . . . . . 188
divbrms . . . . . . . . . 188
divrhoum . . . . . . . . 187
divrhoumax . . 180, 187

VARIABLE INDEX
divrhourms . . 180, 187
divru2mz . . . . . . . . 217
divu . . . . . . . . . . . . 215
divu2m . . . . . . . . . . 179
divu2mz . . . . . . . . . 217
divuHrms . . . . . . . . 181
divum . . . . . . . . . . . 179
divumz . . . . . . . . . . 217
dobrms . . . . . . 182, 192
dr0 . . . . . . . . . 233, 234
drho2m . . . . . 181, 189
drho2mx . . . . . . . . . 226
drho2mxy . . . . . . . . 231
drho2mxz . . . . . . . . 228
drho2my . . . . . . . . . 224
drho2mz . . . . . . . . . 221
drhom . . . . . . 181, 189
drhomax . . . . . . . . . 189
drhomx . . . . . . . . . . 226
drhomxy . . . . . . . . . 231
drhomxz . . . . . . . . . 228
drhomy . . . . . . . . . . 224
drhomz . . . . . . . . . . 221
drhorms . . . . . . . . . 189
dsnap . . . . . 26, 36, 172
dstalk . . . . . . . . . . . 171
dt . . . 8, 38, 41, 172, 178
dt bb 1 . . . . . . . . . . 189
dt chiral . . . . . . . . . 189
dt mu5 1 . . . . . . . . 189
dt mu5 2 . . . . . . . . 189
dt mu5 3 . . . . . . . . 189
dtb . . . . . . . . . 184, 193
dtc 8, 181, 189, 196, 198
dtchem . . . . . . . . . . 188
dtchi . . . 8, 181, 190, 198
dtchi2 . . . 186, 189, 197
dteta . . . . . . . . 184, 193
dtF . . . . . . . . . . . . . 181
dtH . . . . . . . . . . . . 181
dtmin . . . . . . . . . . . 172
dtnewt . . . . . . . . . . 197
dtnu . . . . . . . . . . 8, 214
dtpchem . . . . . . . . . 196
dtq . . . . . . . . . . . . . 191
dtq2 . . . . . . . . . . . . 191
dtrad . . . . . . . 186, 189
dtradloss . . . . . . . . 197
dtshear . . . . . . . . . . 197
dtspitzer . 186, 189, 191,

197
dtu . . . . . . . . . . . 8, 180
dtvel . . . . . . . . . . . . 197
dubrms . . . . . . 182, 192
dudx . . . . . . . . . . . . 180
durms . . . . . . . . . . . 178
dvid . . . . 26, 27, 36, 172
dz 1 . . . . . . . . . . . . . 23
dz tilde . . . . . . . . . . . 23
E0rms201, 204, 206, 208
E0Um . . . 202, 205, 209
E0Wm . . 202, 205, 209
e1 . . . 233, 235–238, 240
E10z . 201, 205, 206, 209
E111z . . . 201, 205–208
E112z 201, 205, 206, 209
E11rms . 201, 204, 206,
208
E11xy . . . . . . . . . . . 202
E121z . . . 201, 205–208
E122z 201, 205, 206, 209
E12rms . 201, 204, 206,
208
E12xy . . . . . . . . . . . 202
E13xy . . . . . . . . . . . 202
e1o . . . . . . . . . 233, 234
e2 . . . 233, 235–238, 240
E20z . 201, 205, 206, 209
E211z . . . 201, 205–208
E212z 201, 205, 206, 209
E21rms . 201, 204, 206,
208
E21xy . . . . . . . . . . . 202
E221z . . . 201, 205–208
E222z 201, 205, 206, 209
E22rms . 201, 204, 206,
208
E22xy . . . . . . . . . . . 202
E23xy . . . . . . . . . . . 202
e3 . . . . . . 233, 235–237
E30z . 202, 205, 206, 209
E311z . . . 201, 205–208
E312z 201, 205, 206, 209
E31xy . . . . . . . . . . . 202
E321z . . . 201, 205–208
E322z 201, 205, 206, 209
E32xy . . . . . . . . . . . 202
E33xy . . . . . . . . . . . 202
e3xamz1 . . . . . 220, 222
e3xamz2 . . . . . 220, 222

253
e3xamz3 . . . . . 220, 222
E41xy . . . . . . . . . . . 202
E42xy . . . . . . . . . . . 202
E43xy . . . . . . . . . . . 202
E51xy . . . . . . . . . . . 202
E52xy . . . . . . . . . . . 202
E53xy . . . . . . . . . . . 202
E61xy . . . . . . . . . . . 202
E62xy . . . . . . . . . . . 202
E63xy . . . . . . . . . . . 202
E71xy . . . . . . . . . . . 202
E72xy . . . . . . . . . . . 202
E73xy . . . . . . . . . . . 202
E81xy . . . . . . . . . . . 202
E82xy . . . . . . . . . . . 202
E83xy . . . . . . . . . . . 202
E91xy . . . . . . . . . . . 202
E92xy . . . . . . . . . . . 202
E93xy . . . . . . . . . . . 202
EBpq 202, 205, 206, 209
ecr . . . . . . . . . . . . . 215
eem . . 181, 189, 198, 214
eemz . . . . . . . . . . . . 223
ekin . . . . . . . . . . . . 180
ekincr . . . . . . . . . . . 189
ekinmx . . . . . . . . . . 225
ekinmz . . . . . . . . . . 218
ekinp . . . . . . . . . . . 196
ekintot . . . . . . . . . . 180
emag . . . . . . . 183, 193
embmz . . . . . . 185, 194
EMFdotB int . . . . . 195
EMFdotBm . . . . . . 195
EMFmax . . . . . . . . 195
EMFmin . . . . . . . . . 195
EMFmz1 . . . . . . . . 195
EMFmz2 . . . . . . . . 195
EMFmz3 . . . . . . . . 195
EMFrms . . . . . . . . . 195
emxamz3 . . . . 185, 194
Emymxz . . . . . . . . . 229
Emzmask . . . . . . . . 198
Emzmxy . . . . . . . . . 232
eos merger . . . . . . . . 89
epot . . . . . . . . . . . . 190
epotmx . . . . . . . . . . 226
epotmxy . . . . . . . . . 231
epotmy . . . . . . . . . . 224
epotmz . . . . . . . . . . 221
epottot . . . . . . . . . . 190

254
epotuxmx . . . . . . . . 226
epotuxmxy . . . . . . . 231
epotuzmz . . . . . . . . 221
epsAD . . . . . . . 183, 192
epsilonaa . . . . . . . . 170
epsK . . . . . . . . . . . . 214
epsKmz . . . . . . . . . . 223
epsM . . . . . . . 183, 192
epsMmz . . . . . 221, 222
eruzmz . . . . . . . . . . 223
eta . . . . . . . . . . . . . 176
eta11 200, 203, 205, 207,
209
eta11 x . . . . . . . . . . 206
eta11 x2 . . . . . . . . . 206
eta11cc . . 200, 203, 205,
207
eta11x . . . . . . . . . . . 206
eta11z . . . . . . . . . . . 208
eta12 200, 203, 205, 207,
209
eta12 x . . . . . . . . . . 206
eta12 x2 . . . . . . . . . 206
eta12cs . . 200, 203, 205,
207
eta12x . . . . . . . . . . . 207
eta12z . . . . . . . . . . . 208
eta21 200, 203, 205, 207,
209
eta21 x . . . . . . . . . . 206
eta21 x2 . . . . . . . . . 206
eta21sc . . 200, 203, 205,
207
eta21x . . . . . . . . . . . 207
eta21z . . . . . . . . . . . 208
eta22 200, 203, 205, 207,
209
eta22 x . . . . . . . . . . 206
eta22 x2 . . . . . . . . . 206
eta22ss . . 200, 204, 205,
207
eta22x . . . . . . . . . . . 207
eta22z . . . . . . . . . . . 208
eta31 . . . . . . . 207, 209
eta32 . . . . . . . 207, 209
eta ext . . . . . . . . . . 176
eta int . . . . . . . . . . 176
eta out . . . . . . . . . . 176
eta tdep . . . . . . . . . 182
etaj2max . . . . 186, 195

T HE P ENCIL C ODE
etajmax . . . . . 186, 195
etajrhomax . . . 186, 195
etasmagm . . . . 186, 195
etasmagmax . . 186, 195
etasmagmin . . 186, 195
etatm . . . . . . . . . . . 195
etatotalmx . . . . . . . 226
etatotalmz . . . 220, 222
etavamax . . . . 186, 195
ethm 181, 189, 196, 198,
214
ethmax . . . . . . . . . . 214
ethmcr . . . . . . . . . . 189
ethmin . . . . . . . . . . 214
ethmz . . . . . . . 219, 223
ethtot 181, 190, 198, 214
etot . . . . . . . . . . . . . 214
ex . . . . . . . . . . 238, 240
Ex0pt . . . 201, 204, 208
Ex11pt . . 201, 204, 208
Ex12pt . . 201, 204, 208
Ex21pt . . 201, 204, 208
Ex22pt . . 201, 204, 208
exabot . . . . . . . 183, 193
examx . . . . . . . 185, 194
examxy1 . . . . . . . . . 231
examxy2 . . . . . 231, 232
examxy3 . . . . . 231, 232
examy . . . . . . . 185, 194
examz . . . . . . . 185, 194
examz1 . . . . . . 220, 222
examz2 . . . . . . 220, 222
examz3 . . . . . . 220, 222
exatop . . . . . . . 183, 193
exd . . . . . . . . . 238, 240
exf . . . . . . . . . 238, 240
exjmx . . . . . . . 185, 195
exjmy . . . . . . . 185, 195
exjmz . . . . . . . 185, 195
exm . . . . . . . . 238, 240
Exmxy . . . . . . 186, 195
Exmxz . . . . . . . . . . 228
Exmz . . . . . . . 220, 221
Exp2 . . . . . . . . 183, 192
Expt . . . . . . . . 183, 192
Ey0pt . . . 201, 205, 208
Ey11pt . . 201, 204, 208
Ey12pt . . 201, 205, 208
Ey21pt . . 201, 204, 208
Ey22pt . . 201, 205, 208

Eymxy
Eymxz
Eymz .
Eyp2 . .
Eypt . .
Ezmxy
Ezmxz
Ezmz .
Ezp2 . .
Ezpt . .

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

186, 195
228, 229
220, 221
183, 192
183, 192
186, 195
228, 229
220, 221
183, 193
183, 192

f . . . . 233, 234, 236, 237
f’,’fa . . . . . . . . 238, 240
F11z . . . . . . . . 211–213
F12z . . . . . . . . 211–213
F21z . . . . . . . . 211–213
F22z . . . . . . . . 211–213
F31z . . . . . . . . 211–213
F32z . . . . . . . . 211–213
fB . . . . . . 236, 238–240
fbcx . . . . . . . . . . . . 233
fbcx12 . . . . . . . . . . . 234
fbcx2 . . . . . . . . . . . . 233
fbm . . . . . . . . . 183, 192
Fbot . . . . . . . . . . . . 175
fBs . . . . . 236, 238–240
Fcm . . . . . . . . 233, 234
fconvm . . . . . . . . . . 182
fconvmz . . . . . . . . . 223
fconvxmx . . . . . . . . 225
fconvxy . . . . . . . . . . 230
fconvyxy . . . . . . . . . 230
fconvz . . . . . . . . . . . 219
fconvzxy . . . . . . . . . 230
Fct . . 233, 234, 238, 239
ffakez . . . . . . . . . . . 223
FFLAGS DOUBLE . . 52
fg . . . 233, 234, 236–239
Fgs . . 233, 234, 238, 239
fil . . . . . . . . . . 234, 235
fix . . . . . . . . . . 234, 235
fkinxmx . . . . . . . . . 218
fkinxmxy . . . . . . . . 230
fkinymxy . . . . . . . . 230
fkinzm . . . . . . . . . . 178
fkinzmz . . . . . . . . . 217
fmasszmz . . . . . . . . 217
fmax . . . . . . . . . . . . . 98
force . . . . . . . . . . . . 177
fountain . . . . . . . . . 177
fpresxmz . . . . . . . . . 223

VARIABLE INDEX
fpresymz . . . . . . . . . 223
fpreszmz . . . . . . . . . 223
fradbot . . 181, 190, 198
fradmx . . . . . . . . . . 225
fradmz . . . . . . . . . . 223
fradtop . . 181, 190, 197
fradx kramers . . . . 225
fradxy Kprof . . . . . 230
fradxy kramers . . . . 230
fradymxy Kprof . . . 230
fradz . . . . . . . . . . . 219
fradz constchi . . . . . 219
fradz Kprof . . . . . . 219
fradz kramers . . . . . 219
fring1,fring2 . . . . . . 170
frmax . . . . . . . . . . . 197
fs . . . . . . . . . . 238, 240
fsum . . . . . . . . . . . . . 98
fturbfz . . . . . . . . . . 219
fturbmx . . . . . . . . . 225
fturbmz . . . . . . . . . 219
fturbrxy . . . . . . . . . 230
fturbthxy . . . . . . . . 230
fturbtz . . . . . . . . . . 219
fturbxy . . . . . . . . . . 230
fturbymxy . . . . . . . . 230
fturbz . . . . . . . . . . . 219
fum . . . . . . . . . . . . 180
fviscm . . . . . . . . . . . 214
fviscmax . . . . . . . . . 214
fviscmin . . . . . . . . . 214
fviscmx . . . . . . . . . . 227
fviscmxy . . . . . . . . . 232
fviscmz . . . . . . . . . . 223
fviscrmsx . . . . . . . . 214
fviscsmmxy . . . . . . . 232
fviscsmmz . . . . . . . . 223
fviscymxy . . . . . . . . 232
fxbxm . . . . . . . 183, 192
g . . . . 234, 235, 238, 240
g11pt . . . . . . . . . . . 190
g12pt . . . . . . . . . . . 190
g22pt . . . . . . . . . . . 190
g23pt . . . . . . . . . . . 190
g31pt . . . . . . . . . . . 190
g33pt . . . . . . . . . . . 190
gal . . . . . . . . . . . . . 209
gam . . . . . . . . 198, 199
gam11 . . . . . . 210–212
gam11z . . 210, 211, 213

gam12 . . . . . . 210–212
gam12z . . 210, 211, 213
gam13 . . . . . . 210–212
gam13z . . 210, 211, 213
gam21 . . . . . . 210–212
gam21z . . 210, 211, 213
gam22 . . . . . . 210–212
gam22z . . 210, 211, 213
gam23 . . . . . . 210–212
gam23z . . 210, 212, 213
gam31 . . . . . . 210–212
gam31z . . 210, 212, 213
gam32 . . . . . . 210–212
gam32z . . 210, 212, 213
gam33 . . . . . . 210–212
gam33z . . 210, 212, 213
gam3z . . . . . . . . . . 212
gamc . . . . . . . . . . . 211
gamcz . . . . . . . . . . . 211
gamma . . 167, 174, 209
gammaQ . . . . . . . . 209
gamz . . . . . . . 198, 199
gdivu2m . . . . . . . . . 179
gg2m . . . . . . . . . . . 190
ggT2m . . . . . . . . . . 190
ggTm . . . . . . . . . . . 190
ggTp2 . . . . . . . . . . . 190
ggTpt . . . . . . . . . . . 190
ggTXm . . . . . . . . . . 190
ggX2m . . . . . . . . . . 190
ggXm . . . . . . . . . . . 190
ggXp2 . . . . . . . . . . . 190
ggXpt . . . . . . . . . . . 190
gLambm . . . . . . . . . 186
gLamrms . . . . . . . . 187
gmu5mx . . . . . . . . . 188
gmu5my . . . . . . . . . 189
gmu5mz . . . . . . . . . 189
gmu5rms . . . . . . . . 188
gpu astaroth.f90 . . . 163
grads0 . . . . . . . . . . 168
grav amp . . . . . . . . 168
grav profile . . . 129, 130,
167, 177
grav tilt . . . . . . . . . 168
gravz . . . 129, 167, 177
grhomax . . . . . . . . . 181
grid func . . . . . . . . . 23
gshockmax . . . . . . . 197
gsrms . . . . . . . . . . . 182

255
gsxmxy . . . .
gsymxy . . . .
gszmxy . . . .
gT2m . . . . .
gtheta5mx .
gtheta5my .
gtheta5mz . .
gtheta5rms .
gTmax . . . .
gTrms . . . . .
gTxgsrms . .
gTxgsx2mxy
gTxgsx2mz .
gTxgsxmxy .
gTxgsxmz . .
gTxgsy2mxy
gTxgsy2mz .
gTxgsymxy .
gTxgsymz . .
gTxgsz2mxy
gTxgsz2mz .
gTxgszmxy .
gTxgszmz . .
gTxmxy . . .
gTymxy . . . .
gTzmxy . . . .
guxgTm . . .
guygTm . . .
guzgTm . . .
gzlnrhomz .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . 230
. . . . 230
. . . . 230
. . . . 197
. . . . 188
. . . . 188
. . . . 188
. . . . 188
182, 197
. . . . 182
. . . . 182
. . . . 230
. . . . 219
. . . . 230
. . . . 219
. . . . 230
. . . . 219
. . . . 230
. . . . 219
. . . . 230
. . . . 219
. . . . 230
. . . . 219
. . . . 230
. . . . 230
. . . . 230
. . . . 197
. . . . 197
. . . . 197
. . . . 219

h22rms . . .
h23rms . . .
h33rms . . .
hat . . . . . .
hcond0 . . .
hcond1 . . .
hcond2 . . .
hds . . . . . .
headt . . . .
headtt . . . .
heatmz . . .
height eta .
height ff . .
hhT2m . . .
hhThhXm .
hhTp2 . . .
hhTpt . . . .
hhTXm . . .
hhX2m . . .
hhXp2 . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . 190
. . . . 190
. . . . 190
233, 235
174, 175
. . . . 174
. . . . 174
238, 239
. . . . 108
. . . . 108
. . . . 219
. . . . 176
. . . . 177
. . . . 190
. . . . 190
. . . . 190
. . . . 190
. . . . 190
. . . . 190
. . . . 190

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

256
hhXpt . . . . .
hjparallelm
hjperpm . . .
hjrms . . . . .
Hmax . . . . .
hrms . . . . . .
hs . . . . . . . .
hse . . . . . . .
hydro.f90 . .

T HE P ENCIL C ODE
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

. . . . 190
186, 195
186, 195
184, 193
. . . . 181
. . . . 190
238, 240
238, 240
. . . . 163

ialive . . . . . . . . 25, 173
ialive=0 . . . . . . . . . . 26
idiag jbm . . . . . . . . . 98
IDL PATH . . . . . . 5, 55
idx tavg . . . 30, 31, 173
iforce . . . . . . . . . . . 176
iforce2 . . . . . . . . . . 176
iheatcond . . . . 174, 176
imax . . . . . . . . . . . . . 54
in . . . . . . . . . . 239, 240
in0 . . . . . . . . . 239, 240
ind . . . . . . . . . 239, 240
inf . . . . . . . . . 239, 240
init ads mol frac . . 171
init surf mol frac . . 171
initaa . . . . . . . . . . . 169
initaa2 . . . . . . . . . . 170
initlncc . . . . . . . . . . 170
initlncc2 . . . . . . . . . 170
initlnrho . . . . . . . . . 167
initpower . . . . . . . . 128
initpower2 . . . . . . . 128
initpower2=-5/3 . . . 128
initpower=2 . . . . . . 128
initpower=4. . . . . . . 128
initpower aa=2. . . . 128
initss . . . . . . . . . . . 168
inituu . . . . . . . . . . . 166
ioc . . . . . . . . . 234, 235
ip . . . . . . . 35, 164, 171
Iring1,Iring2 . . . . . 170
isav . . . . . . . . . . . . . 25
isave . . . . . . . . . . . . 172
ism . . . . . . . . . . . . . 238
isothtop . . . . . 169, 175
Isurf . . . . . . . . . . . . 215
it . . . . . . 8, 33, 34, 178
it1 . . . . . . . 25, 29, 172
it1d . . . . . . . . . 29, 172
itorder . . . . . . . 36, 172
iuut . . . . . . . . . . . . 100

ivar
ivisc
iwig
ix . .
iy . .
iz . .
iz2 .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

. . . 108
. . . 177
. . . 172
26, 172
26, 172
26, 172
26, 172

j11rms . . . . . . 201, 204
j2 . . . . . . . . . . . . . . 215
j2m . . . . . 182, 188, 191
j2mz . . . . . . . . 221, 222
jb . . . . . . . . . . . 98, 215
jb0m . . . . 201, 204, 208
jb int . . . . . . . 182, 191
jbm . . . . . . 98, 182, 191
jbmh . . . . . . . . 182, 191
jbmn . . . . . . . . 182, 192
jbmphi . . . . . . 216, 217
jbms . . . . . . . . 182, 192
jbmxy . . . . . . . . . . . 231
jbmz . . . . . . . . 220, 222
jbrms . . . . . . . 182, 191
jbtm . . . . . . . . 182, 191
jdel2am . . . . . . . . . 186
jet . . . . . . . . . . 233, 235
jj . . . . . . . . . . . . . . 215
jm . . . . . . . . . . . . . 187
jm2 . . . . . . . . . 182, 191
jmax . . . . 184, 187, 193
jmbmz . . . . . . 185, 194
jmin . . . . . . . . . . . . 187
jmx . . . . . . . . . 184, 194
jmy . . . . . . . . . 185, 194
jmz . . . . . . . . . 185, 194
jparallelm . . . 186, 195
jperpm . . . . . . 186, 195
jrms . . . . 184, 187, 193
jx2m . . . . . . . . . . . . 188
jxaprms . . . . . . . . . 187
jxarms . . . . . . . . . . 187
jxbm . . . . . . . . 185, 194
jxbmx . . . . . . . 185, 194
jxbmy . . . . . . . 185, 194
jxbmz . . . . . . . 185, 194
jxbr2m . . . . . . 186, 195
jxbrmax . . . . . 186, 195
jxbxm . . . . . . . 183, 192
jxbym . . . . . . . 183, 192
jxbzm . . . . . . . 183, 192
jxgLamrms . . . . . . . 187

jxm . .
jxmax
jxmxy
jxmxz
jxmz .
jxp2 .
jxpt . .
jy2m .
jybxm
jybym
jybzm
jym . .
jymax
jymxy
jymxz
jymz .
jyp2 .
jypt . .
jz2m .
jzbxm
jzbym
jzbzm
jzm . .
jzmax
jzmxy
jzmxz
jzmz .
jzp2 .
jzpt . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . . . . . 188
184, 188, 193
. . . . 230, 231
. . . . . . . . 228
. . . . 220, 221
. . . . 183, 192
. . . . 183, 192
. . . . . . . . 188
. . . . 183, 192
. . . . 183, 192
. . . . 183, 192
. . . . . . . . 188
184, 188, 193
. . . . 230, 231
. . . . . . . . 228
. . . . 220, 221
. . . . 183, 192
. . . . 183, 192
. . . . . . . . 188
. . . . 183, 192
. . . . 183, 192
. . . . 183, 192
. . . . . . . . 188
184, 188, 193
. . . . 230, 231
. . . . . . . . 228
. . . . 220, 221
. . . . 183, 192
. . . . 183, 192

k forc . . . . . . . . . . . 174
kap11 . . . . . . . 210–212
kap11z . . 210, 212, 213
kap12 . . . 210, 211, 213
kap12z . . 210, 212, 213
kap13 . . . 210, 211, 213
kap13z . . 210, 212, 213
kap21 . . . 210, 211, 213
kap21z . . 210, 212, 213
kap22 . . . 210, 211, 213
kap22z . . 210, 212, 213
kap23 . . . 210, 211, 213
kap23z . . 210, 212, 213
kap31 . . . 210, 211, 213
kap31z . . 210, 212, 213
kap32 . . . 210, 211, 213
kap32z . . 210, 212, 213
kap33 . . . 210, 211, 213
kap33z . . 210, 212, 213
kapcPARA . . . . . . . 211
kapcPARAz . . . . . . . 211

VARIABLE INDEX
kapcPERP1 . . . . . . 211
kapcPERP2 . . . . . . 211
kapcPERPz . . . . . . 211
kapPARA . . . . 198, 199
kapPARAz . . . 198, 199
kapPERP . . . . 198, 199
kapPERP2 . . . . . . . 199
kapPERPz . . . 198, 199
kfountain . . . . . . . . 177
khor ss . . . . . . . . . . 169
kinflow . . . . . . . . . . 176
KK2m . . . . . . . . . . . 189
KKm . . . . . . . . . . . . 189
Kkramersm . . . . . . 182
Kkramersmx . . . . . . 225
Kkramersmz . . . . . . 219
kmz . . . . . . . . 185, 194
kpeak . . . . . . . . . . . 128
kpeak=3. . . . . . . . . . 128
kx . . . . . . . . . . . . . . 176
kx aa . . . 170, 185, 194
kx lncc . . . . . . . . . . 170
ky . . . . . . . . . . . . . . 176
ky aa . . . . . . . . . . . 170
ky lncc . . . . . . . . . . 170
kz . . . . . . . . . . . . . . 176
kz aa . . . . . . . . . . . 170
kz lncc . . . . . . . . . . 170
l1 . . . . . . . . . . . . . . 108
l2 . . . . . . . . . . . . . . 108
Lambzm . . . . . . . . . 186
Lambzmz . . . . . . . . 186
Lamm . . . . . . . . . . 186
Lamp2 . . . . . . . . . . 186
Lampt . . . . . . . . . . 186
Lamrms . . . . . . . . . 186
lb nxgrid . . . . . . . . . 54
lbubble . . . . . . . . . . 178
lcalc heatcond constchi
174
ldamp fade . . . . . . . 173
ldensity var . . . . . . . 89
ldragforce dust par . 177
ldragforce gas par . 177
ldraglaw steadystate 177
lequidist . . . . . . . . . . 23
lfirst . . . . . . . . . . . . 108
lfirstpoint . . . . . . . . 108
lhcond global . . . . . 176
lignore Bext in b2 . . 176

lncc . . . . . . . . . . . . 215
lnowrite . . . . . . . . . 164
lnrho . . . . . . . . . . . 215
lnrhomphi . . . . . . . 216
lnTT . . . . . . . . . . . . 215
lout . . . . . . . . . . . . . 108
lperi . . . . . . . . . . . . 164
lpress equil . . . . . . . 170
lprocz slowest . . 50, 164
lpscalar sink . . . . . 177
lread oldsnap . . . . . 164
lread oldsnap nomag
165
lread oldsnap nopscalar
165
lroot . . . . . . . . . . . . 108
lshift origin . . . . . . 165
lspecies transfer [T] 178
lstalk ap . . . . . . . . . 171
lstalk bb . . . . . . . . . 171
lstalk grho . . . . . . . 171
lstalk guu . . . . . . . . 171
lstalk relvel . . . . . . 171
lstalk rho . . . . . . . . 171
lstalk uu . . . . . . . . . 171
lstalk vv . . . . . . . . . 171
lstalk xx . . . . . . . . . 171
lthiele [T] . . . . . . . . 178
luminosity . . . . . . . 175
lupw lnrho . . . . . . . 174
lupw ss . . . . . . . . . . 175
luse Bext in b2 . . . . 176
lwrite 2d . . . . . . . . 164
lwrite aux . . . . 164, 173
lwrite ic . . . . . . . . . 164
lwrite phiaverages . . 29
lwrite yaverages . . . . 29
lwrite zaverages . . . . 29
Lxyz . . . . 23, 25, 35, 164
m ....
m1 . . .
M11 . .
M11cc .
M11ss .
M11z .
M12cs .
m2 . . .
M22 . .
M22cc .
M22ss .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

. . . . . . . . 108
. . . . . . . . 108
200, 204, 207
200, 204, 207
200, 204, 207
202, 205, 209
200, 204, 207
. . . . . . . . 108
200, 204, 207
200, 204, 207
200, 204, 207

257
M22z . . . 202, 205, 209
M33 . . . . 200, 204, 207
M33z . . . 202, 205, 209
mach . . . . . . . . . . . 215
mag flux . . . . . . . . . 197
magfricmax . . 185, 194
MAGNETIC INIT PARS . . . . . 165
Mamax . . . . . . . . . . 180
Marms . . . . . . . . . . 180
mass . . . . 181, 187, 189
maux . . . . . . . . 32, 100
maxadvec . . . . 141, 178
meshRemax . . . . . . 214
mgam33 . 210, 212, 213
mkap33 . 210, 212, 213
MMxm . . . . . . . . . . 189
MMym . . . . . . . . . . 189
MMzm . . . . . . . . . . 189
mpm . . . . . . . . . . . . 196
mpmax . . . . . . . . . . 196
mpmin . . . . . . . . . . 196
mpoly0 . . . . . . 168, 169
mpoly1 . . . . . . 168, 169
mpoly2 . . . . . . . . . . 169
mu . . . . . . . . . 198, 199
mu2 . . . . . . . . . . . . 199
mu5m . . . . . . . . . . . 188
mu5rms . . . . . . . . . 188
muc1 . . . . . . . . . . . 211
muc2 . . . . . . . . . . . 211
mucz . . . . . . . . . . . 211
mumz . . . . . . . . . . . 223
muz . . . . . . . . 198, 199
mvar . . . . . 32, 78, 165
mx . . . . . . 26, 108, 109
my . . . . . . 26, 108, 109
mz . . . . . . 26, 108, 109
n . . . . . . . . . . . . . . 108
n1 . . . . . . . . . . . . . . 108
n1s . . . . . 233, 235, 237
n2 . . . . . . . . . . . . . . 108
nfr . . . . . 233, 235–237
ngam33 . 210, 212, 213
nil . . 234, 235, 239, 240
nil’,’ . . . . . . . . 236, 237
nkap33 . . 210, 212, 213
noentropy.f90 . . . . . 163
nogpu.f90 . . . . . . . . 163
nohydro.f90 . . . . . . 163

258

T HE P ENCIL C ODE

nopower spectrum.f90
163
noyinyang.f90 . . . . . 163
noyinyang mpi.f90 . 163
npm . . . . . . . . . . . . 196
nprocy . . . . . . . 50, 164
nprocz . . . . . . . . . . . 50
nr1 . . . . . . . . . . . . . 233
nr directions . . . . . . . 54
nt . . . . . . . . . 8, 36, 171
nu . . . . . . . . . 177, 209
nu epicycle . . . 168, 177
nu hyper2 . . . . . . . . 177
nu hyper3 . . . . . . . . 177
nu LES . . . . . . . . . 214
nu tdep . . . . . . . . . . 214
num . . . . . . . . . . . . 214
numx . . . . . . . . . . . 227
nuQ . . . . . . . . . . . . 209
nusmagm . . . . . . . . 214
nusmagmax . . . . . . 214
nusmagmin . . . . . . 214
nv . . . . . . . . . . . . . . 108
nvar . . . . . . . . . . 26, 83
nx . 28, 29, 101, 108, 109
nxgrid . . . . 53, 54, 108
ny . . . . . 28, 29, 108, 144
nygrid . . . . . . . . . . . 41
nz . . . . . 28, 29, 108, 144
nzgrid . . . . . . . . . . . . 41
o2 . . . . .
o2m . . .
o2mz . .
odel2um
ogux2mz
oguxmz .
oguy2mz
oguymz .
oguz2mz
oguzmz .
omax . .
Omega .
omega ff
omumz .
oo . . . . .
orms . . .
ou0 . . . .
ou int . .
oud . . . .
ouf . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . 215
. . . . 180
. . . . 217
. . . . 180
. . . . 218
. . . . 218
. . . . 218
. . . . 218
. . . . 218
. . . . 218
. . . . 180
. . . . 173
. . . . 177
. . . . 179
. . . . 215
. . . . 180
239, 240
. . . . 180
239, 240
239, 240

oum . . . . . . . . . 25, 180
oumphi . . . . . . . . . . 180
oumx . . . . . . . . . . . 225
oumxy . . . . . . . . . . . 229
oumxz . . . . . . . . . . . 227
oumy . . . . . . . . . . . 223
oumz . . . . . . . . . . . 218
out . . 233–236, 239, 240
out1 . . . . . . . . . . . . 172
out2 . . . . . . . . . . . . 172
outm . . . . . . . . . . . . 178
ovr . . 233, 234, 239, 240
ox2m . . . . . . . . . . . 180
ox2mx . . . . . . . . . . . 225
ox2mz . . . . . . . . . . . 217
oxdivu2mz . . . . . . . 218
oxdivumz . . . . . . . . 218
oxmxy . . . . . . . . . . . 229
oxmz . . . . . . . . . . . . 217
oxoym . . . . . . . . . . . 180
oxozm . . . . . . . . . . . 180
oxuxxmz . . . . . . . . . 218
oxuyxmz . . . . . . . . . 218
oxuzxm . . . . . . . . . . 180
oxuzxmz . . . . . . . . . 218
oy2m . . . . . . . . . . . 180
oy2mx . . . . . . . . . . . 225
oy2mz . . . . . . . . . . . 217
oydivu2mz . . . . . . . 218
oydivumz . . . . . . . . 218
oymxy . . . . . . . . . . . 229
oymz . . . . . . . . . . . . 217
oyozm . . . . . . . . . . . 180
oyuxymz . . . . . . . . . 218
oyuyymz . . . . . . . . . 218
oyuzym . . . . . . . . . . 180
oyuzymz . . . . . . . . . 218
oz2m . . . . . . . . . . . 180
oz2mx . . . . . . . . . . . 225
oz2mz . . . . . . . . . . . 217
ozdivu2mz . . . . . . . 218
ozdivumz . . . . . . . . 218
ozmxy . . . . . . . . . . . 229
ozmz . . . . . . . . . . . . 217
p . . . . 232, 234–237, 239
p1D . . . . . . . . 238, 239
PATH . . . . . . . . 5, 6, 12
pdivum . . 181, 189, 196
peffmxz . . . . . . . . . . 229
PENCIL HOME 5, 6, 75

PENCIL HOST ID . . 18
pertss . . . . . . . . . . . 168
pfc . . . . . . . . . 234–237
pfe . . . . . . . . . 238, 239
phi11 . . . . . . . 200, 203
phi12 . . . . . . . 200, 203
phi21 . . . . . . . 200, 203
phi22 . . . . . . . 200, 203
phi32 . . . . . . . 200, 203
phibmx . . . . . . 186, 195
phibmy . . . . . . 186, 195
phibmz . . . . . . 186, 195
phibzm . . . . . . . . . . 191
phibzmz . . . . . . . . . 191
phiK . . . . . . . . 200, 203
phiM . . . . . . . 200, 203
phim . . . . . . . . . . . 191
phiMK . . . . . . 200, 203
phimphi . . . . . . . . . 216
phip2 . . . . . . . . . . . 191
phipt . . . . . . . . . . . 191
placeholder . . . . . . . 178
polytrm . . . . . . . . . . 197
pot . . . . . . . . . 238, 239
power spectrum.f90 163
Poynting . . . . . . . . . 215
poynxmxy . . . . . . . . 231
poynymxy . . . . . . . . 231
poynzmxy . . . . . . . . 231
poynzmz . . . . . 221, 222
pp . . . . . . 215, 235, 236
ppm . 181, 189, 198, 214
ppmx . . . . . . . 225, 227
ppmy . . . . . . . . . . . 224
ppmz . . . 219, 222, 223
pr1mz . . . . . . . . . . . 223
pretend lnTT . . . . . 165
pscalar diff . . . . . . 176
PSCALAR INIT PARS
165
pscalar sink rate . . 177
psi11 . . . . . . . 200, 203
psi12 . . . . . . . 200, 203
psi21 . . . . . . . 200, 203
psi22 . . . . . . . 200, 203
puzmz . . . . . . . . . . . 223
pvzm . . . . . . . . . . . 180
pvzmxy . . . . . . . . . . 229
pwd . . . . . . . . 238, 239
q2m . . . . . . . . . . . . 180

VARIABLE INDEX
qam . . . . . . . . . . . . 195
qem . . . . . . . . . . . . 195
qfm . . . . . . . . . . . . . 180
qfviscm . . . . . . . . . . 214
qmax . 54, 180, 186, 191
qom . . . . . . . . . . . . 180
qpm . . . . . . . . . . . . 195
qpmz . . . . . . . . . . . 222
Qrad . . . . . . . . . . . 215
qrms . . . . 180, 186, 191
qsatmin . . . . . . . . . 191
qsatrms . . . . . . . . . 191
qshear . . . . . . 171, 177
qsm . . . . . . . . . . . . 195
quxom . . . . . . . . . . 180
qxmax . . . . . . . . . . 191
qxmin . . . . . . . . . . . 191
qymax . . . . . . . . . . 191
qymin . . . . . . . . . . . 191
qzmax . . . . . . . . . . . 191
qzmin . . . . . . . . . . . 191
r ext . . . . . . . . . . . . 176
r ff . . . . . . . . . . . . . 177
r int . . . . . . . . . . . . 176
radius . . . . . . . . . . 170
radius ss . . . . . . . . 168
random gen . . 165, 173
rcool . . . . . . . . . . . . 175
rcylmphi . . . . . . . . . 216
rdamp . . . . . . . . . . 173
rdampext . . . . . . . . 173
rdampint . . . . . . . . 173
rdivum . . . . . . . . . . 179
REAL PRECISION 52,
144
reinitialize lncc . . . . 176
relhel . . . . . . . 148, 177
relhel uu=0 . . . . . . . 128
relhel uu=1 . . . . . . . 128
Remz . . . . . . . . . . . 218
Reshock . . . . . . . . . 214
rho . . . . . . . . . . . . . 215
rho0 . . . . 167, 174, 177
rho2mx . . . . . . . . . . 219
rho2mz . . . . . . . . . . 218
rho left . . . . . . . . . . 167
rho right . . . . . . . . . 167
rhoccm . . . . . . . . . . 186
rhom . . . 8, 25, 181, 187
rhomax . . . . . . 181, 189

rhomin . . . . . .
rhomphi . . . . .
rhomx . . . . . . .
rhomxmask . .
rhomxy . . . . . .
rhomxz . . . . . .
rhomy . . . . . . .
rhomz . . . . . . .
rhomzmask . . .
rlx2m . . . . . . .
rlxm . . . . . . . .
rly2m . . . . . . .
rlym . . . . . . . .
rlz2m . . . . . . .
rlzm . . . . . . . .
Rmesh . . . . . .
Rmesh3 . . . . .
Rmmz . . . . . . .
rmphi . . . . . . .
Rring1,Rring2
rumax . . . . . .
rux2m . . . . . . .
rux2mx . . . . . .
rux2mxy . . . . .
rux2mz . . . . . .
ruxm . . . . . . .
ruxmx . . . . . . .
ruxmxy . . . . . .
ruxmz . . . . . . .
ruxtot . . . . . . .
ruxuy2mz . . . .
ruxuym . . . . . .
ruxuymx . . . . .
ruxuymxy . . . .
ruxuymz . . . . .
ruxuz2mz . . . .
ruxuzm . . . . . .
ruxuzmx . . . . .
ruxuzmxy . . . .
ruxuzmz . . . . .
ruy2m . . . . . . .
ruy2mx . . . . . .
ruy2mxy . . . . .
ruy2mz . . . . . .
ruym . . . . . . .
ruymx . . . . . . .
ruymxy . . . . . .
ruymz . . . . . . .
ruyuz2mz . . . .
ruyuzm . . . . . .

181, 189
. . . . 216
. . . . 225
. . . . 181
. . . . 230
. . . . 227
. . . . 223
. . . . 218
. . . . 181
. . . . 180
. . . . 180
. . . . 180
. . . . 180
. . . . 180
. . . . 180
. . . . 178
. . . . 178
. . . . 185
. . . . 216
. . . . 170
. . . . 179
. . . . 179
. . . . 225
. . . . 229
. . . . 218
. . . . 179
. . . . 225
. . . . 229
. . . . 218
. . . . 179
. . . . 218
. . . . 180
. . . . 225
. . . . 229
. . . . 218
. . . . 218
. . . . 180
. . . . 225
. . . . 229
. . . . 218
. . . . 179
. . . . 225
. . . . 229
. . . . 218
. . . . 179
. . . . 225
. . . . 229
. . . . 218
. . . . 218
. . . . 180

259
ruyuzmx . . .
ruyuzmxy . .
ruyuzmz . . .
ruz2m . . . . .
ruz2mx . . . .
ruz2mxy . . .
ruz2mz . . . .
ruzdownmz .
ruzm . . . . .
ruzmx . . . . .
ruzmxy . . . .
ruzmz . . . . .
ruzupmz . . .

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

225
230
218
179
225
229
218
217
179
225
229
218
217

s . . . . 232, 234–237, 239
s+f . . . . . . . . . 236, 237
s0d . . 232, 234–237, 239
s2kzDFm 200, 204, 207
sa2 . . . . . . . . . 233, 235
sds . . . . . . . . . 235, 236
sep . . . . . . . . . 236, 237
set . 233, 234, 236–238,
240
sf . . . . . . 232, 237, 239
sfr . . . . . 233, 235–237
Shchm . . . . . . . . . . 196
shock . . . . . . . . . . . 215
shockmax . . . . . . . . 197
shx . . . . . . . . . . . . . 233
shy . . . . . . . . . . . . . 233
shz . . . . . . . . . . . . . 233
Sij2m . . . . . . . . . . . 214
slice position 26, 27, 172
slo . . . . . . . . . 233, 234
slp . . . . . . . . . . . . . 233
spd . . . . . . . . . 233, 235
spr . . . . . . . . . 232, 234
spt . . . . . . . . . 236, 237
sr1 . . . . . . . . . . . . . 233
ss . . . 215, 232, 234–236
ss2m . . . . . . . . 181, 189
ss2mx . . . . . . . . . . . 225
ss2mz . . . . . . . . . . . 219
sse . . . . . . . . . 236, 237
ssm . . . . 8, 181, 189, 198
ssmax . . . . . . . . . . . 182
ssmin . . . . . . . . . . . 182
ssmphi . . . . . . . . . . 216
ssmx . . . . . . . . . . . . 225
ssmxy . . . . . . . 190, 230
ssmxz . . . . . . . 190, 227

260
ssmy . . . . . . . . . . . . 224
ssmz . . . . . . . . 219, 223
ssruzm . . . . . . . . . . 181
ssuzm . . . . . . . . . . . 181
sT . . . 233–236, 238, 240
StokesImxy . . . 231, 232
StokesQ1mxy . 231, 232
StokesQmxy . . 231, 232
StokesU1mxy . 231, 232
StokesUmxy . . 231, 232
strTpt . . . . . . . . . . . 190
strXpt . . . . . . . . . . . 190
StS . . . . . . . . . 238, 240
t . . . . . . . 8, 25, 108, 178
tauheat buffer . . . . . 175
tauhmin . . . . . . . . . 181
tavg . . . . . . 30, 31, 173
tdamp . . . . . . . . . . . 173
Tdxpm . . . . . . . . . . 197
Tdypm . . . . . . . . . . 197
Tdzpm . . . . . . . . . . 197
tensor pscalar diff . 176
thcool . . . . . . . . . . . 198
theta . . . . . . . . . . . . 173
theta5m . . . . . . . . . 188
theta5rms . . . . . . . . 188
timestep.f90 . . . . . . 163
timestep strang.f90 . 163
timestep subcycle.f90 163
tmax . . . . . . . . . . . . 172
tot ang mom . . . . . . 180
total carbon sites . . 171
totmass . . . . . . . . . . 181
Trms . . . . . . . . . . . 197
ts . . . . . . . . . . . . . . . 44
TT . . . . . . . . . . . . . 215
TT2m . . . . . . . . . . . 197
TT2mx . . . . . . . . . . 225
TT2mz . . . . . . 219, 223
TTheat buffer . . . . . 175
TTm . 181, 197, 198, 214
TTmax . . 182, 197, 198,
214
TTmin182, 197, 198, 214
TTmx . . . . . . . 225, 227
TTmxy . . . . . . 230, 232
TTmxz . . . . . . 227, 229
TTmy . . . . . . . . . . . 224
TTmz . . . . . . . 219, 223
ttransient . . . . . . . . 173

T HE P ENCIL C ODE
TTtop . . . . . . . 181, 190
TTzmask . . . . . . . . 197
TugTm . . . . . . . . . . 197
Tugux uxugTm . . . . 197
Tuguy uyugTm . . . . 197
Tuguz uzugTm . . . . 197
u0rms . . . .
u11rms . . .
u12rms . . .
u1u23m . .
u1u32m . .
u2 . . . . . . .
u21rms . . .
u22rms . . .
u2m . . . . .
u2mphi . . .
u2mz . . . .
u2tm . . . .
u2u13m . .
u2u31m . .
u3u12m . .
u3u21m . .
uabxmz . . .
uabymz . . .
uabzmz . . .
uam . . . . .
uamz . . . .
ubbzm . . .
ubm . . . . .
ubmz . . . .
ubs . . . . . .
udpxxm . .
ufpresm . .
ufviscm . . .
uglnrhom .
uglnrhomz
ugm . . . . .
ugrhom . . .
ugrhomz . .
ugu2m . . .
ugurmsx . .
uguxmxy . .
uguymxy . .
uguzmxy . .
ujm . . . . .
ujxbm . . . .
umamz . . .
umax . . . .
umbmz . . .
umin . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

201, 204
201, 204
201, 204
. . . . 179
. . . . 179
. . . . 214
201, 204
201, 204
. . . . 178
. . . . 216
. . . . 217
. . . . 178
. . . . 179
. . . . 179
. . . . 179
. . . . 179
219, 221
219, 221
219, 221
183, 192
220, 222
182, 191
182, 192
220, 222
239, 240
. . . . 181
. . . . 182
. . . . 214
. . . . 181
. . . . 219
. . . . 190
181, 187
. . . . 219
. . . . 180
. . . . 180
. . . . 229
. . . . 229
. . . . 229
183, 192
186, 195
. . . . 179
. . 8, 178
. . . . 179
. . . . 178

umx . . . . . . . . . 29, 179
umxbmz . . . . . . . . . 179
umy . . . . . . . . . 29, 179
umz . . . . . . . . . 29, 179
unit density . . . 37, 165
unit length . . . . 37, 165
unit system . . . . 37, 165
unit temperature 36, 37,
165
unit velocity . . . 37, 165
uotm . . . . . . . . . . . . 178
upmphi . . . . . . . . . . 216
urand . . . . . . . . . . . 166
urmphi . . . . . . . . . . 216
urms . . . . 8, 25, 128, 178
urms=3.981E-01 . . . 128
urms=5.560E-01 . . . 128
urmsx . . . . . . . . . . . 178
urmsz . . . . . . . . . . . 178
ursphmphi . . . . . . . 216
uthmphi . . . . . . . . . 216
uu . . . . . . . . . . . . . 214
uu left . . . . . . . . . . 166
uu right . . . . . . . . . 166
uumphi . . . . . . . . . 216
uusphmphi . . . . . . . 216
uut . . . . . . . . . . . . . 100
ux0m . . . . . . . 201, 204
ux0mz . . . . . . . . . . 209
ux11m . . . . . . 201, 204
ux2ccm . . . . . . . . . . 179
ux2m . . . . . . . . . . . 179
ux2mx . . . . . . . . . . 225
ux2mxy . . . . . . . . . . 229
ux2mxz . . . . . . . . . . 227
ux2mz . . . . . . . . . . 217
ux2ssm . . . . . . . . . . 179
uxbm . . . . . . . 185, 194
uxbmx . . . . . . 185, 194
uxbmy . . . . . . 185, 194
uxbmz . . . . . . 185, 194
uxbxm . . . . . . 182, 192
uxbxmz . . . . . . 220, 222
uxbym . . . . . . 183, 192
uxbymz . . . . . . 220, 222
uxbzm . . . . . . 183, 192
uxbzmz . . . . . . 220, 222
uxglnrym . . . . . . . . 180
uxm . . . . . . . . . . . . 179
uxmax . . . . . . . . . . 179

VARIABLE INDEX
uxmin . . . . . .
uxmx . . . . . .
uxmxy . . . . . .
uxmxz . . . . . .
uxmy . . . . . .
uxmz . . . . . .
uxp2 . . . . . . .
uxpt . . . . . . .
uxrms . . . . . .
uxTm . . . . . .
uxTmz . . . . .
uxTTmx . . . .
uxTTmxy . . .
uxTTmz . . . .
uxuycsm . . . .
uxuydivum . .
uxuym . . . . .
uxuymx . . . .
uxuymxy . . . .
uxuymxz . . . .
uxuymz . . . . .
uxuzm . . . . .
uxuzmx . . . . .
uxuzmxy . . . .
uxuzmxz . . . .
uxuzmz . . . . .
uxxrms . . . . .
uxzrms . . . . .
uy0m . . . . . .
uy0mz . . . . .
uy11m . . . . .
uy2ccm . . . . .
uy2m . . . . . .
uy2mx . . . . .
uy2mxy . . . . .
uy2mxz . . . . .
uy2mz . . . . .
uy2ssm . . . . .
uybxm . . . . .
uybxmxz . . . .
uybxmz . . . . .
uybym . . . . .
uybymz . . . . .
uybzm . . . . .
uybzmxz . . . .
uybzmz . . . . .
uyglnrxm . . .
uygzlnrhomz .
uym . . . . . . .
uymax . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 223
. . . . 217
. . . . 178
. . . . 178
. . . . 178
. . . . 197
. . . . 223
. . . . 225
. . . . 230
. . . . 219
. . . . 179
. . . . 180
. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 218
. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 218
. . . . 181
. . . . 181
201, 204
. . . . 209
201, 204
. . . . 179
. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 217
. . . . 179
182, 192
. . . . 228
220, 222
183, 192
220, 222
183, 192
. . . . 228
220, 222
. . . . 180
. . . . 219
. . . . 179
. . . . 179

uymin . . . . . .
uymx . . . . . .
uymxy . . . . . .
uymxz . . . . . .
uymy . . . . . .
uymz . . . . . .
uyp2 . . . . . . .
uypt . . . . . . .
uyrms . . . . . .
uyTm . . . . . .
uyTmz . . . . .
uyTTmx . . . .
uyTTmxy . . .
uyTTmz . . . .
uyuzm . . . . .
uyuzmx . . . . .
uyuzmxy . . . .
uyuzmxz . . . .
uyuzmz . . . . .
uyxuzxmz . . .
uyyrms . . . . .
uyyuzymz . . .
uyzrms . . . . .
uyzuzzmz . . .
uz0mz . . . . . .
uz2m . . . . . .
uz2mx . . . . .
uz2mxy . . . . .
uz2mxz . . . . .
uz2mz . . . . . .
uzbxm . . . . .
uzbxmz . . . . .
uzbym . . . . .
uzbymz . . . . .
uzbzm . . . . . .
uzbzmz . . . . .
uzdivum . . . .
uzdivumz . . .
uzdownmz . .
uzgylnrhomz .
uzm . . . . . . .
uzmax . . . . .
uzmin . . . . . .
uzmphi . . . . .
uzmx . . . . . .
uzmxy . . . . . .
uzmxz . . . . . .
uzmy . . . . . .
uzmz . . . . . .
uzp2 . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 223
. . . . 217
. . . . 178
. . . . 178
. . . . 178
. . . . 197
. . . . 223
. . . . 225
. . . . 230
. . . . 219
. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 218
. . . . 218
. . . . 181
. . . . 218
. . . . 181
. . . . 218
. . . . 209
. . . . 179
. . . . 225
. . . . 229
. . . . 227
. . . . 217
182, 192
220, 222
183, 192
220, 222
183, 192
220, 222
. . . . 180
. . . . 217
. . . . 217
. . . . 219
. . . . 179
. . . . 179
. . . . 179
. . . . 216
. . . . 225
. . . . 229
. . . . 227
. . . . 223
. . . . 217
. . . . 178

261
uzpt . . . .
uzrms . . .
uzTm . . .
uzTmz . .
uzTTmx .
uzTTmxy
uzTTmz .
uzupmz .
uzyrms . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

178
179
197
223
225
230
219
217
181

v . . . . 232, 234–237, 239
v3 . . . . . . 235–237, 239
vAm . . . . . . . . . . . . 188
vAmax . . 184, 188, 193
vAmin . . . . . . . . . . 188
vAmxz . . . . . . 228, 229
vArms . . . . . . . 184, 193
visc heatm . . . . . . . 214
vol . . . . . . . . . . . . . 181
vpx2m . . . . . . . . . . 196
vpxm . . . . . . . . . . . 196
vpxmax . . . . . . . . . . 196
vpxmin . . . . . . . . . . 196
vrelpabsm . . . . . . . . 196
w forc . . . . . . . . . . . 174
walltime . . . . . . . . . 178
wcool . . . . . . . . . . . 175
wdamp . . . . . . 173, 174
wheat . . . . . . . . . . . 175
width ff . . . . . . . . . 177
widthaa . . . . . . . . . 170
widthlnrho . . . . . . . 167
widthss . . 168, 174, 175
widthuu . . . . . . . . . 166
win . . . . . . . . . 239, 240
wr1,wr2 . . . . . . . . . 170
write slices . . . . . . . . 27
wsnaps.f90 . . . . . . . 172
xi . . . . .
xiQ . . . .
xp2m . .
xpm . . .
xpmax .
xpmin . .
xy . . . . .
xy2 . . . .
xyz0 . . .
xyz star

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

. . . . . 209
. . . . . 209
. . . . . 196
. . . . . 196
. . . . . 196
. . . . . 196
. . . . . 172
. . . . . 172
23, 25, 164
. . . . . . . 25

yH . . . . . . . . . . . . . 215

262
yHm . . . . . . . . 181, 198
yHmax . . . . . . 181, 198
yHmin . . . . . . . . . . 198
yinyang.f90 . . . . . . . 163
yinyang mpi.f90 . . . 163
yy . . . . . . . . . . 235, 237

T HE P ENCIL C ODE
z0aa . . . .
z1 . . . . . .
z2 . . . . . .
zbot slice
zeta . . . .
zetaQ . . .

.
.
.
.
.
.

.
.
.
.
.
.

. . . . . . 170
. . . . . . 167
. . . . . . 168
26, 28, 172
. . 177, 209
. . . . . . 209

zheat buffer
zmphi . . . . .
zref . . . . . .
ztop slice . .

. . . . . . 175
. . . . . . 216
. . 167, 177
26, 28, 172

Index
This index contains options, names, definitions and commands. Files and variables have
their own indexes.
’VAR10’ . . . . . . . . . . 45
.r . . . . . . . . . . . . . . . 45
.run . . . . . . . . . . . . . 45
.svn . . . . . . . . . . . . . 12
/trimall . . . . . . . . . . 45
$PENCIL HOME/idl/files
46
2N-scheme . . . . . . . 156
6th-order derivatives153
pc mkproctree 16 . . 145
adapt-mkfile . . . . . . . 77
adapt-mkfile . . . . 22, 88
anelastic . . . . . . . . . . 67
autoconf . . . . . . . . . . 88
Autoconf . . . . . . . . . . 88
Autoconf/automake . . 88
Averages . . . . . . 29, 30
Azimuthal averages . 29
bandwidth . . . . . . . . 50
Bash . . . . . . . . . . . 5, 75
bash . . . . . . . . . . . . . 44
bc . . . . . . . . . . . . . . . 44
Beowulf clusters . . . . 50
Bidiagonal scheme . 155
bin/pc run . . . . . . . . 82
Boundary conditions . 39
Bourne shell . . . . . . . . 5
C . . . . . . . . . viii, 1, 109
Cdata . . . . . . . . . . . 108
cgs units . . . . . . 36, 165
Changes . . . . . . . . . 126
Check-in details . . . . 91
CHIRAL=nochiral . . 78
Coding standards97, 122
Comments . . . . . . . 123
copy-proc-to-proc
seed.dat
../hydro256e . . . . 31
Coriolis force . . . . . 173
Cosmic rays . . . . . . . 68
Courant number . . . . 36
Cron . . . . . . . . . . . . 106

crontab -e . . . . . . . . 106
CSH . . . . . . . . . . . . viii
Csh . . . . . . . 1, 5, 12, 75
csh . . . . . . . . . . . . . . 75
CVS . . . . . . . . . . . . . . 2
CVS . . . . . . . . . . . . . vii
Cvs . . . . . . . . . . . . . . 11
cvs-add-rundir . . . . . 32
Daainit . . . . . . . . . . 146
Data directory . . . . . 14
Data explorer . . . . 1, 41
datafiles . . . . . . . . . . 25
Density init pars . . 166
Density run pars . . 174
diffrho hyper3 mesh=2
141
double precision 51, 144
Download . . . . 2, 44, 75
Download forbidden . 75
DX . viii, 1, 5, 12, 32, 41,
42
Emacs settings . . . . 125
Entropy . . . . . . . . . . 61
Entropy . . . . 15, 40, 61
Entropy.f90 . . . . . . . . 27
Entropy init pars . . 168
Entropy run pars . . 174
Equation of state . . . 63
Error, diffusive . . . . 157
Etatest . . . . . . . . . . 146
f-array . . .
f90 . . . . . .
F95 . . . . . .
f95 . . . . . .
FAQ . . . . .
FBCX1 . . .
FBCX2 2 .
FC=mpif90
ff . . . . . . .
ff.varname
FFT . . . . .
Fftpack . . .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

263

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

. . 100
21, 22
viii, 1
. . . 21
. . . 75
. . 146
. . 146
. . . 81
. . . 45
. . . 46
. . . 13
. . . 52

Filters . . . . . . . . . . 137
Flag . . . . . . . . 163, 171
Flux rings . . . . . . . . 128
Forcing run pars . . 36,
147, 148, 176
Fortran record . . 26, 30
Fortran record . . . . . 26
fp-array . . . . . . . . . 101
Frequently Asked Questions . . . . . . . 75
ftp . . . . . . . . . . . . . . 44
Fully
qualified
host
name . . . . . . 18
G77 . . . . . . . . . . . . . 77
G95 . . . . . . . . . . 52, 76
GDL . . . . . . . . . . . . . 42
Gfortran . . . . . . viii, 52
Ghost points . . . . . . . 39
Ghost zones . . . . 39, 50
Git . . . . . . . . . . . . . . . 2
Git . . . . . . . . . . . . . 2, 3
git . . . . . . . . . . . . . . . 3
Glibc . . . . . . . . . . . . 76
Gnu Data Language . 42
GNU gcc . . . . . . . . . viii
Gnuplot . . . . . . . 15, 41
gnuplot . . . . . . . . . . . 44
Grav init pars . . . . 167
Grav r . . . . . . . . . . . 13
Grav run pars 146, 177
Gravitational Waves . 71
Gravity . . . . . . . . . . . 13
Gravity simple . . . . . 13
grep . . . . . . . viii, 95, 96
grid, nonuniform . . . 23
h-index . . . . . . . . . . . 91
Hydro.f90 . . . . . . . . . 27
Hydro init pars . . . 165
Hydro run pars . . . 173
hyperdiffusivity . . . 137
Hyperviscosity 137, 139–
141, 153, 155
Icc . . . . . . . . . . . . . . 76

264

T HE P ENCIL C ODE

idiff=’hyper3 mesh’ . 141
IDL . . viii, 1, 5, 8, 9, 12,
15, 27, 28, 30, 32,
42–44, 53, 55
IDL . . . . . . . . . . . . 148
idl . . . . . . . . . . . . . . 44
Ifc . . . . . . . . . . . . . . . 76
Ifort . . . . . . . . . . 76, 77
ifort . . . . . . . . . . . . . 52
incompressible . . . . . 67
Init pars . . . 34, 35, 164
Initial conditions . . 127
InitialCondition module
104
Interlocked flux rings128
Interstellar . . . . . . . . 15
IO . . . . . . . . . . 108, 109
Io mpiodist.f90 . . . . . 16
Ionization . . . . . 64, 158
Ionization.f90 . . . 36, 37
itorder=5 . . . . . . . . . 38
Janus . . . . . . . . . 21, 22
LANG=POSIX
ldensity . . . . .
Linux . . . . . . .
locate mpif.h . .
lwrite aux=T .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

. . 83
. . 89
1, 76
. . 81
. . 66

Magnetic . . . . . . . . . 98
Magnetic . . . . . . . . . 98
Magnetic helicity . . . vii
Magnetic.f90 . . . 27, 28
Magnetic init pars . 169
Magnetic run pars 146,
176
Make . . . . 1, 13, 22, 78
make . . . . 6, 13, 19, 20
make clean . . . . . . . . 76
Makefile . . 6, 14, 21, 80
Makefile . . . . . . . . . . 20
Manual . . . . . iv, 90, 104
mesh, nonuniform . . 23
Message passing interface . . . . . . . 49
Module . . . . . . . . . . viii
Module.h . . . . . . . . . 76
Modules . . . . . . . . . . 13
Modules . . . . . . viii, 13
MPEG . . . . . . . . . . . 27

Mpeg encode . . . . . . 27
MPI vii, 1, 9, 13–15, 21,
49, 77, 88
mpif90 -show . . . . . . 81
mpif90 -showme . . . . 81
mpirun . . . . . . . . . . . 14
Namelist . . . . . . . . . 32
Namelists . . . . . . . . . 34
Networks . . . . . . . . . 41
NEWDIR file . . . . . . 34
Newphysics . . . . . . 104
Noentropy . . 15, 40, 61
NOERASE file . . . . . 86
Nogravity . . . . . . . . . 13
Noionization.f90 . 36, 37
Nomodule.f90 . . . . . . 76
Nonewphysics . . . . 104
nonuniform grid . . . . 23
Nospecial.f90 . 103, 104
octave . . . . . . . . . . . . 44
Onsager . . . . . . . . . . 22
OpenDX . . . . . . . . . . . 1
Option ‘–host-id’ . . . . 18
Option ‘–use-pc’ . . . 107
Option ‘–use-pc auto-test’
21
Option ‘-b’ . . . . . . . . . 21
Option ‘-D ¡dir¿’ . . . 107
Option ‘-f’ . . . . . . . . 106
Option ‘-fast’ . . . . . . . 22
Option
‘-fno-secondunderscore’ . . 76,
77
Option ‘-H’ . . . . 18, 107
Option ‘-l’ . . . . . . . . 106
Option ‘-lmpi’ . . . . . . 21
Option ‘-m ¡email-list¿’
107
Option ‘-N 15’ . . . . . 107
Option ‘-nothreads’ . . 77
Option ‘-O2 -u’ . . . . . 21
Option ‘-O3’ . . . . 21, 79
Option ‘-qextname’ . . 76
Option ‘-t 15m’ . . . . 107
Option ‘-T ¡file¿’ . . . 107
Option ‘-Uc’ . . . . . . 107
Option ‘-Wa,–max-level’
107

Option

‘-Wa,–maxlevel=2’ . . . . 107
Option ‘/png’ . . . . . . 27
Option ‘¿ $HOME/public html/pencilcode/tests/nightlytests.html’ . . 107
Option ‘a’ . . . . . . . . . 39
Option ‘a2’ . . . . . . . . 40
Option ‘c1’ 40, 132, 175
Option ‘c2’ . . . . 40, 174
Option ‘ce’ . . . . . . . . . 40
Option ‘cT’ . . . . . . . . 40
Option ‘db’ . . . . . . . . 40
Option ‘g’ . . . . . . . . . 40
Option ‘hs’ . . . . . . . . 40
Option ‘nohydro’ . . . 176
Option ‘p’ . . . . . . . . . 39
Option ‘pot’ . . . . . . . 132
Option ‘pwd’ . . . . . . 132
Option ‘s’ . . . . . . . . . 40
Option ‘she’ . . . . . . . . 40
Particles . . . . . . . . . . 69
Particles ads init pars
171
Particles ads run pars
178
Particles chem init pars
171
Particles chem run pars
178
Particles run pars . 177
Particles stalker init pars . . . . . . 171
Particles surf init pars
171
Particles surf run pars
178
pc jobtransfer . . . . . . 34
pc auto-test . . 20, 21, 91,
106
pc auto-test –help21, 106
pc build . . . . . . . 17, 20
pc build –cleanall . . 107
pc build –help . . . . . . 20
pc get quantity . . 44, 45
pc jobtransfer . . . . . . 34
pc mkdatadir . . . . . . . 7
pc newrun . . . . . . . . . 7
pc read const, obj=cst 46

INDEX

pc read param, obj=par
46
pc read param, obj=par2,
/param2 . . . 46
pc read pvar, obj=fp . 46
pc read ts, obj=ts . . . 46
pc read var . . . . . 44, 45
pc read var, obj=ff, /trimall . . . . . . . 46
pc read var raw . . . . 44
pc read var raw,
obj=var, tags=tags
46
pc read xyaver, obj=xya
46
pc read xzaver, obj=xza
46
pc read yzaver, obj=yza
46
pc run . . 17, 19, 20, 82
pc run –help . . . . . . . 20
pc setupsrc . . 12, 32, 52,
75, 76
pc svnup . . . . . . . . . . . 3
pc svnup -val . . . . . . . 3
pc tsnap . . . . . . . . . . 26
Pencil case . . . . . . . 101
Pencil check . . . . . . 102
Pencil Code . . . . . . . 88
Pencil consistency check
102
Pencil design . . . . . . 12
pencil-test . 21, 106, 107
pencil-test –help 21, 106
pencil check small=F 66
Pencils . . . . . . . vii, 101
Perl . . . . . . . . viii, 1, 12
perldoc Pencil::ConfigFinder
17
perldoc PENCIL::ConfigParser
17
Planet solution . . . . 134
PNG . . . . . . . . . . . . . 27
Polytropic atmosphere
130
Potential-field boundary
condition . . 132

power . . . . . . . . . 53, 54
pretend lnTT . . 62, 165
Programming style . 97,
122
Pscalar . . . . . . . . . . 104
Pscalar init pars . . 170
Pscalar run pars . . 176
Python . . . . . . . . . . viii
Python . . . . . . . . . . . 46
Radiative transfer . 66,
159
Readline . . . . . . . . . . 44
Regridding . . . . . . . 144
Remeshing . . . . . . . 144
RERUN file . . . . . . . 34
restart-new-dir
../32c
145
Restarting . . . . 40, 145
rlwrap . . . . . . . . . . . 44
Run directory . . . . . . . 6
run.x . . . . . . . . . . . . 30
Run pars 26–28, 34, 145,
171, 173
Runge-Kutta . . . . . 156
Runge-Kutta time step
38
Runge-Kutta-Fehlberg
time step . . . 38
Scripts . . . . . . . . . . . 31
Setup . . . . . . . 5, 42, 75
Shear . . . . . . . . . . . . 40
Shear init pars . . . . 170
Shear run pars 146, 177
Shock viscosity . . 37, 63
SI units . . . . . . 36, 165
Sixth-order derivatives
153
slice files . . . . . . . . . 28
Slice files . . . . . . 26, 27
Special module . . . . 103
start.csh . . . . . . . . . . 40
Stdout . . . . . . . . . . . 25
Stratification . . . . . 129
structure . . . . . . . . . . 55
Style . . . . . . . 3, 97, 122
Sub . . . . . . . . . . . . . 109

265
summarize-history . . 34
svn . . . . . . 2–4, 14, 117
Svn . 2–4, 11, 14, 31, 32,
79, 117, 164, 171
svn . . . . . . . . . . . . . . . 3
svn annotate src/*.f90 v
svn mv file.f90 experimental/file.f90
126
svn up sourceme.csh . 87
svn up sourceme.sh . . 87
svn update . . . . . . . 107
svn update -r ##### . . 4
Svn/git . . . . . . . . . . 2, 3
Syscalls . . . . . . . . . . 76

tab . . . . . . . . . . . . . . 97
Tab characters . . . . 122
tag names . . . . . . . . . 45
Tcsh . . . . . . . . . . . . . . 5
teach/PencilCode/material/Burg
142
Testfield method 71, 145
Time averages . . . . . 30
Time step . . . . . 38, 156
Toroidal averages . . . 29
touch NEWDIR . . . . . 34
touch NOERASE . . 164
touch RELOAD . . . . 171
uname . . . . . . . . . . . 22
Underscore problem . 77
Units . . . . . . . . 36, 165
Unix . . . . . . . . . . . . . . 1
Upwinding . 37, 95, 154
use . . . . . . . . . . . . . 108
Vector potential . . . . 61
Video files . . . . . . . . . 26
Viscosity . . . . . . . 37, 63
Viscosity run pars 146,
177
Weyl gauge . . . . . . . . 61
Whitespace . . . . . . . 122
Xlf . . . . . . . . . . . . . . 76
Xmgrace . . . . . . . . . . 15

266

Id

T HE P ENCIL C ODE



---

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.4
Linearized                      : No
Page Count                      : 282
Page Mode                       : UseOutlines
XMP Toolkit                     : XMP toolkit 2.9.1-13, framework 1.6
About                           : uuid:0a5fd210-8ca2-11f3-0000-53a524296f03
Producer                        : dvips + GPL Ghostscript 9.05
Keywords                        : ()
Modify Date                     : 2018:05:10 21:08:15+02:00
Create Date                     : 2018:05:10 21:08:15+02:00
Creator Tool                    : LaTeX with hyperref package
Document ID                     : uuid:0a5fd210-8ca2-11f3-0000-53a524296f03
Format                          : application/pdf
Title                           : ()
Creator                         : ()
Description                     : ()
Subject                         : 
Author                          : 

EXIF Metadata provided by EXIF.tools