Introduction

This tool is intended to provide a convenient way to develop plugin libraries for Pythia 8 that can be accessed through the use repository of PYTHIA-CONTRIB. For more details on how the plugin system for Pythia 8 works, specifically how it can be configured by the user, please see consult the HTML manual. The basic usage of this is tool is to create a PYTHIA-CONTRIB package template using the syntax,

./generate PACKAGE BASE/NAME ...

which will create a ready-to-compile package template with name PACKAGE and classes of base-class type BASE with corresponding names NAME, given by the list of BASE/NAME pairs following the package name. By default, the BASE arguments should be taken from the list of available base classes which can be passed to Pythia as a plugin using the Init:plugins setting.

Init:plugins = {libPACKAGE.so::NAME}

This list of base-class templates can be printed with,

./generate --list

which is summarized here with further documentation provided for each base class. Of note, the UserHooks class provides an omnibus interface to most parts of the Pythia event generation sequence, and should be considered when a more specific base class is not available.

For the eager developer, a very abbreviated set of commands is given below that will generate, compile, and run an example PYTHIA-CONTRIB package, assuming that Pythia is already installed and pythia8-config is available as a command.

# Generate the template for two user classes, MyUserHooks and MyPDF.
./generate MyPackage UserHooks/MyUserHooks PDF/MyPDF
# Build and configure the package.
cd MyPackage
make -j
# Set the PYTHIA8CONTRIB path.
export PYTHIA8CONTRIB=$PWD/share
# Compile and run the test example for MyUserHooks.
cd share/MyPackage/examples/
make testPlugins
./testPlugins MyUserHooks.cmnd

Further details for each step are given in the remainder of this documentation.

Package Generation

The output of generate will produce a folder with name PACKAGE, with the following structure. Each item of this list is described in further detail in its own dedicated section below.

Example

As an initial example, the following command will produce a PYTHIA-CONTRIB package with the name MyPackage consisting of: a MyUserHooks class of base-class type UserHooks and a MyPDF class of base-class type PDF.

./generate MyPackage UserHooks/MyUserHooks PDF/MyPDF

The package structure for this example is as follows.

MyPackage/
├── configure
├── include
│   └── MyPackage
│       ├── MyPDF.h
│       └── MyUserHooks.h
├── Makefile
├── README.md
├── share
│   └── MyPackage
│       ├── examples
│       │   ├── Makefile
│       │   ├── MyPDF.cmnd
│       │   ├── MyUserHooks.cmnd
│       │   └── testPlugins.cc
│       └── xmldoc
│           ├── Formatting.xml
│           ├── Index.xml
│           └── Settings.xml
└── src
    ├── MyPDF.cc
    ├── MyUserHooks.cc
    └── Package.cc

By default only base classes from the list given by ./generate --list can be provided as BASE arguments to the BASE/NAME list. If a base-class is not available, generate will skip that BASE/NAME pair and issue the following warning.

./generate PACKAGE BASE/NAME
WARNING: Unknown template class 'BASE', skipping 'BASE/NAME'.

If the option --nocheck is specified, then the BASE/NAME pair will not be skipped and a minimal template will be produced for that BASE/NAME pair. For more details on working with classes that cannot be loaded via the Init:plugins setting, please see the additional classes section.

Template Code

For each base-class type, the default behavior of generate is to produce template code where all inherited base-class methods are provided. All methods that need to be defined, i.e., pure virtual methods of the base class, are implemented with an ansatz that should be changed by the package developer. Methods that are implemented in the base class but may be overridden are provided as comments. In the example above, the following method of MyPDF must be implemented,

  // Update parton densities (defined in src/MyPDF.cc). Must be defined.
  void xfUpdate(int id, double x, double Q2) override;

while the method,

  // // Allow for new scaling factor for VMD PDFs.
  // void setVMDscale(double = 1.) override {}

is optional. The constructor and destructor for every package class must also be defined,

  // Constructor (defined in src/MyPDF.cc).
  MyPDF(Pythia*, Settings*, Logger*);

  // Destructor.
  ~MyPDF() = default;

where a template signature is provided in the header file and a template definition in the source file. The templates produced with the default generate behavior compile out-of-the-box.

Minimal Template

In some cases, the developer might wish to begin with a minimal template, where only the constructor and destructor are already defined. This is possible by passing the --minimal option to generate. The template code generated here will not compile out-of-the-box, as the pure virtual methods of the base class will need to be implemented by the developer. When using the --nocheck option above, full template code will be produced if the base class is known, and by necessity, minimal template code will be produced if the base class is not known.

Modifying a Package

Because the generate command could potentially overwrite already written code, the default behavior of generate is to fail when asked to produce the template for a package directory that already exists. For example, running the command

./generate PACKAGE BASE/NAME

twice will result in the following error message.

ERROR: The 'PACKAGE' package already exists, use '--update' to update.

Here, it is possible to update an already generated package with additional classes using the --update command.

./generate --update PACKAGE BASE/NAME

Any class of type NAME being requested that already exists in the package will be skipped with the following warning.

WARNING: Skipping existing class name 'NAME'.

Testing a Package

To compile and test a package, a working installation of Pythia is required. The following steps are intended to quickly set up such an environment, and should give a basic outline of the process. For more details on installing Pythia, please consult the README included as part of Pythia source.

Build Pythia

Pythia can be installed by first checking out the Pythia releases repository.

git clone git@gitlab.com:Pythia8/releases.git
cd releases
# Optionally check out a specific version.
# git checkout pythia8317

It is possible to check out a specific version of Pythia, but only version 8.310 and higher is compatible with the plugin structure of PYTHIA-CONTRIB. Next, configure, build, and install Pythia.

./configure  # Include optional configuration flags here. 
make -j      # Build with an optional number of cores.
make install # Use --prefix options with configure to change installation.
cd ../

By default, the Pythia installation will be placed in the directory in which it was built when make install is called, unless --prefix options are passed to --configure.

Build the Package

Once Pythia is installed, make sure that the pythia8-config script can be found by the configure script of the PYTHIA-CONTRIB package being developed. This can either be done by adjusting the system PATH so that pythia8-config is along it, or by directly passing the location of the pythia8-config script to configure. Assuming the installation direction of the previous section, the system path can be updated as follows.

PATH=$PATH:releases/bin

Alternatively, the package configure script can be called with the location specified.

cd MyPackage
./configure --with-pythia8-config=../releases/bin/pythia8-config

Once the path to pythia8-config is specified using either method, the package can be built with make called from the generated MyPackage directory.

cd MyPackage # If not already there from calling ./configure.
make -j      # Build with an optional number of cores.

If the package was built successfully, then the library lib/libPACKAGE.so should exist. In this case, the command above builds the library lib/libMyPackage.so.

Test the Package

Minimal tests for the package are included in the template. First, set the XML path for PYTHIA-CONTRIB, and then change to the examples directory of the package.

export PYTHIA8CONTRIB=$PWD/share
cd share/MyPackage/examples

There a is a simple test which can be used to check the loading of each plugin class and reads in settings, either directly specified by the plugin library, or via XML documentation provided by the package. The test takes as arguments command files to pass to Pythia with the readFile command. For each plugin class provided by the package there is a corresponding command file which instructs Pythia to load the plugin. In this example, there are two such configurations, MyPDF.cmnd and MyUserHooks.cmnd. First we test the PDF class. 

make testPlugins
./testPlugins MyPDF.cmnd

The output of this test should just be the Pythia welcome banner, as well as the settings that were introduced by the package, both directly through the plugin library as well as the XML files. The setting output should look as follows.

 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: library setting MyPackage:registered is 1
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:flag is 0
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:modeopen is 0
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:modepick is 0
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:parm is 0.000000
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:word is value
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:fvec is {1,0}
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:mvec is {0,1}
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:pvec is {0.000000,1.000000}
 PYTHIA Info from Pythia8Contrib::MyPackage::MyPDF::MyPDF: XML setting MyPackage:wvec is {value1,value2}

If any issues are encountered, there will be additional output describing the problem. If the error cannot open shared object file is returned, please see the section on library paths. Next we can test the UserHooks class, and check that the output is similar.

./testPlugins MyUserHooks.cmnd

Documentation

Documentation for the package should be provided in a top-level README.md as a markdown file. Additional documentation can be provided by the authors using the mechanism of their choice, but at a minimum, README.md is required. The contents of this file are parsed when building the index of all PYTHIA-CONTRIB packages, and are then displayed on the PYTHIA-CONTRIB summary pages.

The following documentation is required to be provided, and should adhere to the same syntax as given here, namely the opening and closing tags for each requested type of documentation. Note, the tags do not need to be included on their own line; this is done below for clarity. The general form of the tags is documentation. However, it is also possible to hide the documentation with the form <!--keyword documentation --> such that it is not present in the markdown rendered README.md. Throughout the documentation, markdown syntax may be used.

Package Description

Everything included within the text bracketed by the tags and will be interpreted as the description of the package. This should provide a brief summary of the package and its use. More details can be given elsewhere in the README.md. The following provides an example.


This is the example description for the template building tool for PYTHIA-CONTRIB. The tool can be used to build template packages that can be used as part of the Pythia plugin system.

Package Authors

The authors for the package should be enclosed within the and tags. General markdown may be used in this description, for example to specify email addresses, etc. The author list is not required to follow a specific format, but it is suggested to use the format last name, first name <email address>; .... An example is given below.


MLhad team; Ilten, Philip <a href="mailto:philten@cern.ch">philten@cern.ch</a>; Szewc, Manuel <a href="mailto:szewcml@ucmail.uc.edu">szewcml@ucmail.uc.edu</a>

Package Documentation

Any relevant documentation should be provided in README.md within the tags and. By default, a Doxygen page will be generated for every package in PYTHIA-CONTRIB, as well as the HTML pages, if XML pages have been provided. This can, for example, include papers or websites. An example is given below.


The MLhad project is documented via a number of websites, codebases, and papers, listed below.

* [MLhad website](https://uchep.gitlab.io/mlhad-docs/): provides a comprehensive description of the MLhad project.
* [Towards a data-driven model of hadronization using normalizing flows](https://inspirehep.net/literature/2723190): paper describing a normalizing flow hadronization model, and methods to train this model with data.

Package Dependencies

Any dependencies for the package should be listed in README.md, enclosed in the and tags. This is not a technical listing, i.e., it is not used when configuring the package, but rather is intended for users to easily see what the dependencies are for the package.


ROOT ([root.cern](https://root.cern/), > 6.28/10); FASTJET ([fastjet.fr](https://fastjet.fr/), > 3.4.2)

Additional Documentation

In Pythia, documentation and data are provided through the share/Pythia8 directory which consists of a number of sub-directories.

In the context of PYTHIA-CONTRIB, it is expected that while developers might want to follow a structure similar to Pythia, they may also have their own preferences on how to provide documentation for their package. Consequently, other than the required documentation described above, documentation of the package is left to the developers. However, in the template package, explicit outlines are given for xmldoc and examples as both of these are considered to be sufficiently important: XML documentation because it can be used to define settings, and example programs as these can be used to help test the package as well as provide concrete scenarios for users.

XML Settings

Settings for a package may either be included directly in code through the PYTHIA8_PLUGIN_SETTINGS macro (see package development), or via XML settings. For the XML settings of a package to be enabled, the starting XML file must be specified by the line

PYTHIA8_PLUGIN_XML("PACKAGE/xmldoc/Index.xml")

which should be included in the package source (typically src/Package.cc). This starting index file is then searched for, in order, along the following paths.

  1. Relative to the PYTHIA8CONTRIB environment variable, i.e., $PYTHIA8CONTRIB/PACKAGE/xmldoc/Index.xml.
  2. Relative to the PYTHIA8DATA environment variable, i.e., $PYTHIA8DATA/../../PACKAGE/xmldoc/Index.xml.
  3. The path itself (whether relative or absolute), i.e., PACKAGE/xmldoc/Index.xml.

If the macro PYTHIA_PLUGIN_XML is used in the package source but the specified XML index cannot be found, the following error will be issued when loading the plugin.

PYTHIA Error in Settings::init: settings file PACKAGE/xmldoc/Index.xml not found

When the index is successfully found, all files specified with the <aidx href="FILE">TITLE</aidx> tags within the index will be loaded by the init method of the Pythia Settings object. In the template, the following files are included.

<aidx href="Formatting">Formatting</aidx><br/>
<aidx href="Settings">Settings</aidx><br/>

Here, the files PACKAGE/xmldoc/Formatting.xml and PACKAGE/xmldoc/Settings.xml will be read via the Settings initialization, and any settings defined in these XML files will then be registered.

A number of settings types are available through the Pythia XML settings scheme. It is recommended that at all settings defined here take the prefix PACKAGE, where this is the package name. This ensures that there will be no settings collisions with other PYTHIA-CONTRIB packages, or with Pythia itself. For each setting type, a number of arguments can be specified. At a minimum, the name of the setting must be specified with the name argument and the default value with the default argument. For settings which are numerical, optional min and max arguments may be provided to limit the values the user may specify. If either of these arguments is missing, then the value will not be limited from below (min) or above (max).

<flag name="PACKAGE:flag" default="off">
This is an example boolean setting which can take the values of either "on" or "off".
</flag>

<modeopen name="PACKAGE:modeopen" default="0" min="0" max="10">
This is an example integer setting that can take any integer value between the limits.
</modeopen>

<modepick name="PACKAGE:modepick" default="0" min="0" max="10">
This is also an example integer setting, the same as "modeopen", but now  specific values can be defined with the "option" tags below. Note, these options are not technically enforced, i.e. a user can still choose an integer value not specified below, as long as it remains between the limits.
<option value="0">Example defined option.</option>
</modepick>

<parm name="PACKAGE:parm" default="0.0" min="0.0" max="10.0">
This is an example double-precision setting which can take real-number values between the limits.
</parm>

<word name="PACKAGE:word" default="value">
This is an example character-string setting which can take any string value without blanks.
</word>

Variable length lists (or vectors) for each of the setting types of above, corresponding to boolean (flag), integer (mode), double-precision (parm), and string (word) values, can also be specified using the vec variations below. The general format for these settings is {value1, value2, ...}.

<fvec name="PACKAGE:fvec" default="{on,off}">
This is an example variable length list of boolean (flag) settings.
</fvec>

<mvec name="PACKAGE:mvec" default="{0,1}" min="0" max="10">
This is an example variable length list of integer (mode) settings.
</mvec>

<pvec name="PACKAGE:pvec" default="{0.0,1.0}" min="0.0" max="10.0">
This is an example variable length list of double-precision (parm) settings.
</pvec>

<pvec name="PACKAGE:pvec" default="{0.0,1.0}" min="0.0" max="10.0">
This is an example variable length list of double-precision (parm) settings.
</pvec>

<wvec name="PACKAGE:wvec" default="{value1,value2}">
This is an example variable length list of character-string (word) settings. Unlike for "word" these string values can contain spaces, but cannot contain commas, ",".
</wvec>

Examples

One of the best ways to demonstrate how to use code is through examples; for context, Pythia has over 100 examples demonstrating various aspects of the program. A very minimal example is provided via the template, testPlugins.cc located in share/PACKAGE/examples. The example creates a Pythia object, reads any user provided command files, and initializes Pythia.

  Pythia pythia;
  pythia.readString("ProcessLevel:all = off");
  for (int iArg = 1; iArg < argc; ++iArg)
    pythia.readFile(argv[iArg]);
  pythia.init();

For each class with name NAME created by the template, a command file NAME.cmnd is provided.

Init:plugins = {libPACKAGE.so::NAME}

In the MyPackage example of the introduction, the test can be run for the PDF class with the following.

./testPlugins MyPDF.cmnd

This will then load the MyPDF class through the plugin system.

A minimal build system is provided in the examples directory of the template. By default it requires the plugin library to be built, and provides rules to build any main* programs as well as the test* programs. External packages can be linked, for more details see build system. The Makefile first reads theMakefile.inc configuration in the same directory, if it exists. It then tries to find the path of the package path (PACKAGE_PATH), first looking along the local path ../../../ and then the Pythia path. Finally, it will build any examples with the following compiler flags.

Note that the flags for the package headers path (-IPACKAGE_INCLUDE_PATH), the link-time path to the package library (-LPACKAGE_LIB_PATH), and the package library (-lPACKAGE) are not passed to the compiler, as it is assumed that the examples use the package as a plugin through Pythia, and not as a standalone class. The run-time path to the package library (-Wl,-rpath,PACKAGE_LIB_PATH) must still be specified to ensure that the plugin library can be found at run time.

Package Development

The PYTHIA-CONTRIB system utilizes the Pythia plugin system. In short, users request the use of a plugin by defining the Init:plugins setting.

Init:plugins = {libPACKAGE1.so::NAME1,libPACKAGE2.so::NAME2,...}

Details on this setting syntax, as well as the Pythia plugin system, are documented in the Pythia HTML manual. At their simplest, PYTHIA-CONTRIB package plugins consist of two parts: developer defined classes from the list of the introduction, and macros which allow the plugin to be loaded by Pythia. These macros are summarized here, and are explained in further detail below in package definition. These macros should not be used in header files.

If any of the three single-use macros are called more than once in a package, the package will fail to compile.

Namespace

Because the PYTHIA-CONTRIB system is intended for decentralized development, it is very possible that there are conflicts in class names between different packages. In general, this should not be an issue, since the run-time loading mechanism of the Pythia plugin system protects against this. However, if users wish to link against the plugin at build time, then this can become an issue. Consequently, it is requested that developers use the nested namespace Pythia8::Pythia8Contrib::PACKAGE for package PACKAGE. In the template header and source, this convention is set with the following structure.

namespace Pythia8 {
namespace Pythia8Contrib {
namespace PACKAGE {
// PACKAGE code here.
} // end namespace PACKAGE
} // end namespace Pythia8Contrib
} // end namespace Pythia8

Without using or namespace statements, this means that accessing objects can be somewhat cumbersome,

Pythia8::Pythia8Contrib::PACKAGE::NAME

where NAME is the object being accessed. Consequently, it is suggested that the above namespace code is always used to enclose the body of header and source files. Alternatively, using <NAMESPACE> statements can be used instead, although this is not recommended; these statements should only be used in source files (*.cc) and never be included in header files (*.h).

Headers

The headers produced by the template generator are all located in include/PACKAGE. One header, include/PACKAGE/NAME.h, is generated for each BASE/NAME pair passed as an argument to generate. This header is intended to define the structure of the class, without including the implementation, which is intended for the corresponding source file, src/NAME.h. However, in some cases it may be necessary to provide implementations in the header, e.g., for the case of templated classes or methods. In general, the macros summarized at the start of this section should not be used in the headers.

When a package is installed with the PYTHIA-CONTRIB system, the headers will be placed in the location include/Pythia8Contrib/PACKAGE, i.e., with an additional level of nesting. However, this should not cause issues, as an additional include path can always be passed to the compiler, -Iinclude/Pythia8Contrib/PACKAGE. In this way the package headers should always be included with statements such as,

#include PACKAGE/NAME.h

without the need for prefixing the include statement with Pythia8Contrib/. This is what is done in the Makefile for the examples located in share/PACKAGE/exampes, where the location of the package is automatically determined to either be local or part of a Pythia installation.

The constructor for the class NAME defined by the header NAME.h must take the following form.

NAME(Pythia* pythiaPtr, Settings* settingsPtr, Logger* loggerPtr)

There are three pointers which may be passed to the constructor.

When a class is constructed from a Pythia object as a plugin, i.e., from the Init:plugins setting, all three pointers are valid. However, it is also possible for plugins to be created directly in the C++ code, where none of these pointers is required to be valid; further details are given in the following section.

When using the full template produced with generate, all required methods for class will be included in the corresponding header. For example, the UserHooks base class does not have any pure virtual methods, and so other than the constructor and destructor, no additional methods need be implemented. However, there are a large number of methods that can be overridden from the base class. In the template, each optionally implemented method will be included as a commented out block of code.

  // // Initialisation after beams have been set by Pythia::init().
  // bool initAfterBeams() override;
  // {return true;}

Here, code is given which will override the initAfterBeams method of the base UserHooks class and return the value of true. The general form of the comment will contain the following three sections: a commented description of the method, the call signature of the method, and optionally the implementation of the method if sufficiently succinct for a header. Note that these methods will always end with the override keyword to indicate that this method should be overriding the base-class implementation.

An example of a required method can be found in a PDF derived class where the protected method xfUpdate must be implemented.

  // Update parton densities (defined in src/MyPDF.cc). Must be defined.
  void xfUpdate(int id, double x, double Q2) override;

Note that here no implementation is given, which is instead relegated to the source file.

Source

For each developer defined class with name NAME, there is a corresponding src/NAME.cc source file. This file is intended to include any detailed implementations that are not suitable for the corresponding include/PACKAGE/NAME.h header. In the generated template, an implementation for the constructor is always provided, which demonstrates how to use methods of the passed Pythia pointer, access settings from the passed Settings pointer, and write messages via the passed Logger pointer. The following provides an abbreviated template for derived-class NAME of base-class BASE.

//--------------------------------------------------------------------------

// Constructor.

NAME::NAME(
  Pythia* pythiaPtrIn, Settings* settingsPtrIn,
  Logger* loggerPtrIn) : BASE() {

  // If a pointer is not needed its name (but not type) can be
  // commented out above, e.g. change pythiaPtrIn to /*pythiaPtrIn*/,
  // to prevent compiler warnings. If a pointer is not needed ensure
  // to specify is not required in the corresponding call to the macro
  // PYTHIA8_PLUGIN_CLASS.

  // Include whatever code is needed here to construct the plugin
  // class. Below, we give some examples of how the pointer arguments
  // can be used.

  // Access and print the settings as defined by the XML files.
  loggerPtrIn->INFO_MSG("XML setting PACKAGE:flag is " + to_string(
      settingsPtrIn->flag("MyPackage:flag")));
  // ...

}

As discussed in the headers section, the passed pointers may not be required to be valid. This is specified by the macro PYTHIA8_PLUGIN_CLASS, which registers the class so that it can be accessed from the plugin library. The usage of PYTHIA8_PLUGIN_CLASS is as follows.

PYTHIA8_PLUGIN_CLASS(BASE, NAME,
  true, // Require the Pythia pointer to be valid.
  true, // Require the Settings pointer to be valid.
  true) // Require the Logger pointer to be valid.

Here, BASE is the base class, e.g., UserHooks, while NAME is the derived class to be registered, e.g., MyUserHooks. If an argument is true than that type of pointer must be valid pointer when using the Pythia plugin system to load the plugin. If false, then that pointer may be passed as a null pointer. If a pointer is specified by the developer as not being required, then it is important that the validity of that pointer is checked in the implemented constructor for the derived class.

In the Pythia plugin system the plugins may be loaded four different ways.

  1. Via user defined settings via the format Init:plugins = {libPACKAGE.so::NAME}. Here, all three pointers passed will always be valid, and correspond to the Pythia object loading the plugin.

  2. Via the following make_plugin method called in C++, where no pointers are passed.

    shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME)

    Here, the PACKAGE string would be given as libPACKAGE.so and NAME is the name of the plugin class to be loaded. All three pointers loaded with this method are null. If any of the PYTHIA8_PLUGIN_CLASS pointer arguments are set as true, then this method will fail and return a null pointer. 3. Via the following make_plugin method called in C++, where now a Pythia pointer is passed.

    shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME, Pythia* pythiaPtr)

    If the Pythia pointer is valid, then all three pointers to the plugin constructor will be valid and taken from the Pythia pointer. If the Pythia pointer is null and any of the pointers are required, then this method will fail and return a null pointer. 4. Via the following make_plugin method called in C++, where all three pointers are passed individually.

    shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME, Pythia* pythiaPtr, Settings* settingsPtr, Logger* loggerPtr)

    If pythiaPtr is valid, and either settingsPtr or loggerPtr is null, then the corresponding pointer from the pythiaPtr will be passed to the plugin class constructor instead. If any of the pointers are required by the PYTHIA8_PLUGIN_CLASS call, and that corresponding pointer is null, then this method will fail and return a null pointer.

In addition to the required implementation of the constructor, a number of methods may be required, dependent upon the base class. The generated template source file for each developer defined class will contain an outline of how to implement the required methods. In the example above for the PDF derived class, an implementation of the required xfUpdate method is given.

//--------------------------------------------------------------------------

// Calculates the PDF distributions for the specified parton id at (x,
// Q^2) and stores the resulting values in the corresponding
// fields. If id == 9, the values should be calculated for all
// relevant partons. The user may choose to always calculate the value
// for all partons, in which case they should set idSav = 9. The PDF
// class flexibly handles antiparticle and isospin symmetries, so
// xfUpdate should always give the values that correspond to the
// particle with positive id and isospin.

void NAME::xfUpdate(int /*id*/, double /*x*/, double /*Q2*/) {

  // // Update the values (currently 0 is used as a dummy value).
  // xg     = 0;
  // ...

  // // idSav = 9 to indicate that all flavours reset, otherwise use id.
  // idSav = 9;

}

Optional methods with more lengthy implementations are also provided in the template source file. For a UserHooks derived class, the following commented out code implements the optional multiplySigmaBy method.

//--------------------------------------------------------------------------

// multiplySigmaBy allows the user to introduce a multiplicative factor
// that modifies the cross section of a hard process. Since it is called
// from before the event record is generated in full, the normal analysis
// does not work. The code here provides a rather extensive summary of
// which methods actually do work. It is a convenient starting point for
// writing your own derived routine.

// double NAME::multiplySigmaBy(const SigmaProcess* sigmaProcessPtr,
//   const PhaseSpace* phaseSpacePtr, bool inEvent) {
// 
//   // Process code, necessary when some to be treated differently.
//   int code       = sigmaProcessPtr->code();
// 
//   // Final multiplicity, i.e. whether 2 -> 1 or 2 -> 2.
//   int nFinal     = sigmaProcessPtr->nFinal();
//
//   // Incoming x1 and x2 to the hard collision, and factorization scale.
//   double x1      = phaseSpacePtr->x1();
//   double x2      = phaseSpacePtr->x2();
//   double Q2Fac   = sigmaProcessPtr->Q2Fac();
//   // ...
//
//   // Dummy statement.
//   return 1.;

This commented code provides a brief outline of how the method could be implemented, including access of relevant data members from the class.

Package Definition

The remaining three macros given at the introduction of this package development section can only be called once per plugin library, unlike PYTHIA8_PLUGIN_CLASS which must be called per class definition. If called more than once in the package code, the package will fail to compile. These macros define the package and can be called anywhere in the source files. However, it is preferred that they are all called in a single source file, which in the generated template is done in src/Package.cc. The first macro,

PYTHIA8_PLUGIN_VERSIONS(8310, 8311, ...)

defines the versions of Pythia that the plugin library is compatible with. Any number of versions can be passed. When a plugin library is loaded by Pythia, a check is made to see if the version of Pythia loading the plugin library is compatible. If it is not, a warning is issued, and a null pointer is returned for any class being constructed from that plugin library.

The second macro is optional, but if called, requires XML files for the plugin library to be present.

PYTHIA8_PLUGIN_XML("PACKAGE/xmldoc/Index.xml")

Any starting index file can be passed, but in the template the file PACKAGE/xmldoc/Index.xml is used as the convention. More details on how settings can be defined via XML, as well as the search path for the starting XML file is given in the XML settings section of this documentation.

Finally, settings can be registered directly in the C++ code, rather than via XML files, through interactions with a Settings object with the following macro.

PYTHIA8_PLUGIN_SETTINGS(PACKAGESettings)

The object PACKAGESettings can take any name, must be a defined method which has the following call signature.

void PACKAGESettings(Settings* settingsPtr)

When this method is called, settingsPtr is guaranteed to be valid and does not need to be checked.

The following add methods for the Settings class allow settings to be added to the settings database. Possible setting types are boolean (flag), integer (mode), double (parm), string (word), vector of booleans (fvec), vector of integers (mvec), vector of doubles (pvec), and vector of strings (wvec). It is recommended that all settings begin with the PACKAGE prefix, to prevent collisions with settings from other PYTHIA-CONTRIB packages, as well as Pythia itself. The arguments to these methods are as follows:

The methods themselves are:

// Add a boolean setting (flag).
settingsPtr->addFlag(string keyIn, bool defaultIn);

// Add an integer setting (mode).
settingsPtr->addMode(string keyIn, int defaultIn, bool hasMinIn,
  bool hasMaxIn, int minIn, int maxIn, bool optOnlyIn = false);

// Add a double setting (parm).
settingsPtr->addParm(string keyIn, double defaultIn, bool hasMinIn,
  bool hasMaxIn, double minIn, double maxIn);

// Add a string setting (word).
settingsPtr->addWord(string keyIn, string defaultIn);

// Add a vector of booleans setting (fvec).
settingsPtr->addFVec(string keyIn, vector<bool> defaultIn);

// Add a vector of integers setting (mvec).
settingsPtr->addMVec(string keyIn, vector<int> defaultIn, bool hasMinIn,
  bool hasMaxIn, int minIn, int maxIn);

// Add a vector of doubles setting (pvec).
settingsPtr->addPVec(string keyIn, vector<double> defaultIn, bool hasMinIn,
  bool hasMaxIn, double minIn, double maxIn);

// Add a vector of strings setting (wvec).
settingsPtr->addWVec(string keyIn, vector<string> defaultIn);

Formatting

No formatting requirements are defined for PYTHIA-CONTRIB packages, although it is recommended to follow the Pythia formatting style. The generated templates follow the Pythia coding style, which is provided in the somewhat lengthy CODINGSTYLE file distributed with the Pythia source. For convenience a few points are summarized here.

Build System

There are any number of build system tools available, and in many cases developers have strong preferences for certain systems over others. Consequently, there are no restrictions on what build systems developers use for their package. However, in many cases, developers may want a build system out-of-the-box, and so we provide such a build system based on the Pythia build system. This build system has relatively few dependencies, other than standard POSIX tools. First we describe the interaction that the build system (whatever that system is) must have with PYTHIA-CONTRIB, then we describe how to customize the provided build system in more detail.

PYTHIA-CONTRIB Interaction

The top-level build system for PYTHIA-CONTRIB consists of a configure script and a Makefile. Each PYTHIA-CONTRIB package is then expected to consist of a folder in the top-level of PYTHIA-CONTRIB. The general build procedure for PYTHIA-CONTRIB is as follows.

./configure [OPTIONS]
make
make install

This will then configure and build all packages in the top level of PYTHIA-CONTRIB. It is expected that the package-provided build system optionally has a configure script, and must have a Makefile. Further specifications for each of these components are given below.

Configure Requirements

The top-level PYTHIA-CONTRIB configure script is designed so that users can configure all the PYTHIA-CONTRIB packages at the top level without needing to configure each package individually. To this end, the top-level configure script takes options of the form:

--with-DEP[=DIR]       : Use the external dependency DEP  with
                         its top directory optionally set by DIR. If the
                         enabled DEP requires other external packages,
                         these packages will also be enabled.
--with-DEP-bin=DIR     : Set the path to the DEP binaries.
--with-DEP-lib=DIR     : Set the path to the DEP libraries.
--with-DEP-include=DIR : Set the path to the DEP headers.
--with-DEP-config=BIN  : Set the package configuration script BIN.

and passes these on to the configure script for each package. Note that here, DEP should be passed in all lower case.

Additionally, the option --nowarn is passed, which tells the package-level configure script to not issue warnings for --with options that are not available. In this way, if package A requires the dependency fastjet3, while package B does not, package B will not warn that it is being passed an invalid option --with-fastjet3. The top-level configure will skip calling the package-level configure script if it is missing, and will issue a warning instead that the package must be configured manually, following documentation that should be provided in the package README.md.

It is expected that if a package is configured, then a package-level Makefile.inc will be present. This file may be empty if a custom build system is being used, but is intended to be included by the package-level Makefile. The PYTHIA-CONTRIB configure script will skip running the package-level configure if the package-level Makefile.inc is present, unless the --force option is passed.

To summarize, the package-level configure script is optional. If it does exist, it should adhere to the following specifications.

Make Requirements

The top-level PYTHIA-CONTRIB build system uses a Makefile to build all the packages. It does this by calling

make -C PACKAGE contrib

for each package with name PACKAGE. The package-level Makefile should then perform the following actions when called with target contrib.

If the library lib/libPACKAGE.so is not produced, then it is assumed that the build has failed. Note that the .so suffix does not change between build platforms, e.g., the .dylib suffix common to MacOS is not used. This is because the plugins are intended to be loaded dynamically at run time as set by configuration files or strings, and so to ensure that the configuration files are not platform dependent, the same library suffix is used across all platforms.

Configuration Customization

When using the provided build system, the following commands can be used to configure external packages. Note, this configuration must be present in README.md, but can be hidden with with a comment tag, i.e., <!-- -->. It is possible to define multiple external packages in README.md, and by default a number of common packages are defined in the template README.md. Each external package configuration starts with a line beginning with PKG=. If options for a package are missing, then they are taken as the default value. Note, each configuration key must begin the line and follow the syntax KEY=VALUE. The following keys are allowed.

By default, a number of common external dependencies are fully defined in README.md. It is suggested that the following names are used for these dependencies. Links to the corresponding website for each dependency are given.

Make Customization

For each external dependency specified with the configure script, a number of variables are available within the Makefile both at the top level of the package, as well in share/PACKAGE/examples. The defined variables are as follows. Note, other than DEP_USE these variables can be empty, either if the dependency has not been configured, or if the dependency does not require that variable.

The following lines in README.md define and require Pythia as a dependency.

PKG=PYTHIA8
CFG=pythia8-config
INC_PATH=--cxxflags
LIB_PATH=--libs
INC_DEPS=Pythia8/Plugins.h
REQUIRE=true

As an example, consider a Pythia installation at /hep/pythia/8.311. Then, when running ./configure, and assuming /hep/pythia/8.311/bin/pythia8-config is along PATH, then the following lines will be generated in the Makefile.inc.

PYTHIA8_USE=true
PYTHIA8_CONFIG=pythia8-config
PYTHIA8_BIN=
PYTHIA8_INCLUDE=-I/hep/pythia/8.311/include
PYTHIA8_LIB=-L/hep/pythia/8.311/lib -Wl,-rpath,/hep/pythia/8.311/lib

Note that here, the Pythia library is not included as part of the dependency. However, if the line,

LIB_DEPS=pythia8

is as added to README.md then the PYTHIA8_LIB variable will now include -lpythia8.

PYTHIA8_LIB=-L/hep/pythia/8.311/lib -Wl,-rpath,/hep/pythia/8.311/lib -lpythia8

Assuming a dependency has been defined, it is then possible to modify the Makefile to link appropriately with the dependency.

<target>: <prerequisites> 
ifeq ($(DEP_USE),true)
    <recipe, given the package exists>
else
    <recipe, given the package does not exist, or, error message>
    $(error Error: $@ requires DEP)
endif

Here, DEP should be changed to whatever dependency is being required. For the specific example of linking the package plugin library against a dependency, the plugin library rule would be changed as follows.

ifeq ($(DEP_USE),true)
$(LOCAL_LIB)/libPACKAGE.so: $(OBJECTS)
    $(CXX) $^ -o $@ $(CXX_COMMON) $(CXX_SHARED) $(CXX_SONAME)$(notdir $@)\
     $(DEP_INCLUDE) $(DEP_LIB)
else
     $(error Error: $@ requires DEP)
endif

Package Submission

Packages for PYTHIA-CONTRIB are recommended to be installed by users through the install repository. The abbreviated package installation procedure is as follows.

./enable PACKAGE[/VERSION] ... [OPTIONS]
./configure [OPTIONS]
make
make install

Here, PACKAGE is the package name, followed by an optional package version, and additional options which can be passed. If no version is given, then the latest version is taken. For a package to be available via the enable command from PYTHIA-CONTRIB, it must have a repository as part of the packages subgroup within PYTHIA-CONTRIB. To request a repository for a new package, please open an issue on the develop repository. The creation of this issue requires a freely available gitlab.com account. The issue should contain the following information.

Package Naming

Before creating a package, it is recommended to first check which names are already used in the packages subgroup of PYTHIA-CONTRIB. While no strict naming restrictions are enforced, the following conventions are encouraged. * Only use letters (i.e., no numbers or special characters like +, , -, _, etc.). Special characters like or + can easily break build systems while numbers in general should be reserved for versioning tags. * Begin the package name with a capital letter, and capitalize the first letter of each subsequent word, e.g., MyPackageName.

Repository Creation

After submitting an issue requesting a package repository as described above, the repository will be created after review by the PYTHIA-CONTRIB team. This review does not require that the package actually exist at this point, but rather is simply intended to check the package name is reasonable and that the package belongs in PYTHIA-CONTRIB, given the package description. Initially, the repository will be created as a private repository until the developers are ready for the first release. After the creation of the repository, two owners will be assigned to the repository.

  1. The package author requesting the repository.
  2. A PYTHIA-CONTRIB author.

The PYTHIA-CONTRIB owner will not make any changes to the repository without the express permission of the package owner, and is included in the case of orphaned packages or when package authors grant permission for minor maintenance to be performed on the package. The goal is that package authors will have complete autonomy over their package repository, while still allowing the PYTHIA-CONTRIB authors to maintain the system. After a repository is created, the package owner can add additional package authors to the repository via their corresponding gitlab.com accounts. The package authors can then begin committing to their repository. Note, gitlab includes a whole suite of features including CI/CD, project wikis, issue trackers, etc. There are a number of gitlab introductions, and gitlab itself provides introductory tutorials.

Version Tagging

Once a repository is created, the package authors can then develop their code as they see fit. However, for the enable script of PYTHIA-CONTRIB to be able to interact with the package repository, the repository must be made public, and versions of the package must be identified via git tags. This is done as follows.

git tag VERSION

Here, VERSION is the name of the version that is being tagged, e.g., v8317.01. It is also possible to create an annotated tag,

git tag -a VERSION -m 'MESSAGE'

where MESSAGE is now a message associated with the tag. Once a tag has been created locally, it must be pushed back to the remote repository on gitlab.com.

git push origin --tags

It is also possible to push a single tag to the remote repository.

git push origin VERSION

Tags can be very useful for any number of reasons, and so the PYTHIA-CONTRIB system tries to be minimally invasive on how tags are used for versioning. All version tags must begin with v[0-9], i.e., the letter v followed by a digit. The remainder of the version tag can then take on whatever form the package authors wish. Any tag that does not follow this convention will not be interpreted by enable from PYTHIA-CONTRIB as a version tag. In this way, package authors can use tags that are not included as releases. It is important to note that the enable script does not require a version number to be passed for a package. When this occurs, enable sorts all the v[0-9] tags from the repository by lexicographic ordering and takes the last tag as the most recent version. Care must be taken with versioning schemes, as according to this ordering v1 would be the last tag given tags v1 and v010.

While no requirements are made on the version tag format, other than beginning with v[0-9], the following versioning conventions are recommended.

Following this convention, package authors can develop separate versions of a package which are compatible with different versions of Pythia, and users will understand at a glance what the minimum version of Pythia is necessary to use that version of a package. Additionally, the most recent version will be interpreted by the enable script as the tag with the largest package version number for the most recent version of Pythia, e.g., given the tagsv8311.01 and v8310.06 the most recent version would be taken as v8311.01 and not v8310.06.

Testing

Extensive testing is an important part of any package development. For Pythia, roughly fifty tests are run before a development branch is merged, including running all the tests with checks for floating point exceptions and memory leaks as well as using a variety of compilers. This is done through the automated continuous-integration/continuous-development (CI/CD) system provided by gitlab.com. These tests are performed using docker containers. A number of Pythia Docker containers are available through Docker hub. For development, the container pythia8/dev:test is typically used which contains a number of external packages. Pythia can be built with this container as follows.

# Pull the Pythia source.
git clone git@gitlab.com:Pythia8/releases.git
cd releases

# Start the Docker container (the cap flag allows use of GDB).
docker run -i -t -v "$PWD:$PWD" -w $PWD -u `id -u` --cap-add=SYS_PTRACE --rm pythia8/dev:test bash --norc

# Configure Pythia with all packages.
export PATH=$PATH:/hep/fastjet3/bin:/hep/hepmc3/bin:/hep/lhapdf5/bin:/hep/rivet/bin:/hep/yoda/bin
./configure --with-evtgen=/hep/evtgen --with-fastjet3 --with-hepmc2=/hep/hepmc2 --with-hepmc3 --with-lhapdf5 --with-lhapdf6=/hep/lhapdf6 --with-powheg-bin=/hep/powheg/ --with-rivet --with-root --with-gzip --with-python --with-mg5mes --with-openmp --with-highfive=/hep/highfive --with-hdf5=/hep/mpich/ --with-mpich=/hep/mpich/

# Build Pythia.
make -j6

# Build the examples.
cd examples
./runmains

This container, or perhaps a more lightweight container, could be used for package testing, depending upon the preferences of the package authors.

Because testing is integral to package development, this is left to the package authors and not the PYTHIA-CONTRIB team. The PYTHIA-CONTRIB team will assume all versions provided by package authors have also been tested and validated by the package authors. However, if it becomes clear from issues submitted by users that a specific package is broken, the PYTHIA-CONTRIB team may choose to exclude this package from the enable script until the latest version of the package is fixed.

Index Generation

An index of all the available PYTHIA-CONTRIB packages is under development. However, the following information will be provided.

Advanced Usage

Some additional documentation is provided here for topics outside the intended scope of the PYTHIA-CONTRIB development system, but that are still relevant.

Additional Classes

The primary use-case for PYTHIA-CONTRIB is through the base classes listed in the introduction. However, as mentioned in the introductory example, it is possible to create a class that uses as any class available in Pythia as a base class. To create a template, the --nocheck option should be passed to generate. This type of plugin class cannot be directly passed to Pythia using the Init:plugins setting, and instead must be loaded using the make_plugin methods described in the source implementation section. This means that while some custom C++ code will need to be written by the user to access the plugin, that code itself does not need to link against the plugin package, since the run-time loading of the library is still handled by the Pythia plugin system.

If settings are required by the plugin package, then before calling the make_plugin methods described in source implementation, the settings of the package need to be registered with a Settings object. This can be done by the following method of the Settings class.

bool Settings::registerPluginLibrary(string PACKAGE, string INDEX = "")

Here, PACKAGE is the name of the plugin library optionally with path, and INDEX is the index file for the XML documentation, again with optional path. If INDEX is not provided, and PYTHIA8_PLUGIN_XML is defined in the plugin package, then this location will be used. In this way it is possible to override the default XML documentation location for the package. If PYTHIA8_PLUGIN_SETTINGS is defined within the plugin package, then this method will also be called.

For convenience, the registerPluginLibrary call has been combined with the basic make_plugin methods into the following make_plugin method.

shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME, Pythia* pythiaPtr, vector<string>& CMNDS)

This method does the following.

  1. Calls registerPluginLibrary on the Settings object from the provided pythiaPtr. At this point, all settings from the plugin library will be available.
  2. Reads in the settings passed by CMNDS using the readString method of the pythiaPtr.
  3. Creates an object of type NAME using the method,
    shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME, Pythia* pythiaPtr)

    and then returns a shared pointer of base class BASE.

In this way it is possible to with a single method create an object from a plugin library while still passing valid settings to be used in the creation of that object. A similar method is also available where the settings are passed via a file rather than a vector of strings.

shared_ptr<BASE> make_plugin<BASE>(string PACKAGE, string NAME, Pythia* pythiaPtr, string CMNDS, int subrun = SUBRUN)

Here, CMNDS is now the setting file and SUBRUN is the subrun to use from that file, where all settings are read if subruns are not being used and SUBRUN is not passed as an argument. When using generate with the --nocheck argument and an unknown base class, a test example share/PACKAGE/examples/testNAME.cc is created which uses these two methods to create an instance of the plugin class.

It is interesting to note that with this method it is possible to create a custom class that inherits from the full Pythia class, and then load it using the plugin system.

./generate --nocheck MyRecursion Pythia/MyPythia

Pythia pythia;
vector<string> cmnds = {};
shared_ptr<Pythia> plugin = make_plugin<Pythia>("libMyPackage.so", "MyPythia", &pythia, cmnds);

In this way, fully customized versions of Pythia can be provided through the PYTHIA8-CONTRIB package.

Python Packages

The PYTHIA-CONTRIB system does not yet support Python modules. However, the Pythia Python bindings are bi-directional; classes defined in Python that inherit from C++ classes can be passed back as C++ objects to Pythia. For example, it is possible to create a UserHooks class entirely in Python.

#==========================================================================

# Write own derived UserHooks class.

class MyUserHooks(pythia8.UserHooks):

    # Make sure to initialize base class.
    def __init__(self):
        pythia8.UserHooks.__init__(self)

    # Allow cross-section to be modified.
    def canModifySigma(self): return True

    # Double the cross-section.
    def multiplySigmaBy(self, sigmaProcessPtr, phaseSpacePtr, inEvent): return 2.

Here, the class simply doubles the cross-section. This can then be passed back to a Pythia object in Python code as follows.

# Generator.
pythia = pythia8.Pythia()

# Pass the user hook back to Pythia.
myUserHooks = MyUserHooks()
pythia.setUserHooksPtr(myUserHooks)

Note, it is not yet possible to load a class from a Python module, i.e., write a derived class in Python and then load that class in C++ code using the Pythia plugin system. However, this may be possible in the future using Python's ability to be embedded in C.

Library Paths

The goal of both the Pythia and PYTHIA-CONTRIB systems is to work out-of-the-box without needing to manually intervene with library paths, i.e., fixing the error cannot open shared object file reported by Pythia, even when the package library is in a valid location. When this occurs, there are a number of possible resolutions.

  1. Set LD_LIBRARY_PATH to include the folder containing the package library, e.g., LD_LIBRARY_PATH=$LD_LIBRARY_PATH:PACKAGE_PATH where PACKAGE_PATH is the relative or absolute path to package library folder. Note, LD_LIBRARY_PATH is not supported in MacOS, although the variable DYLD_LIBRARY_PATH can be used instead in a similar fashion.
  2. Link or move the package library into the same directory as the shared Pythia library, libpythia8.so or libpythia8.dylib.
  3. Link or move the package library into the same directory as the executable being run.
  4. Recompile Pythia with the flag -Wl,--disable-new-tags passed to the linker via the compiler. This can be done by modifying the following line of Makefile.inc below. If Pythia 8.312 or later is being used, this should be done automatically. Details on why this flag needs to be passed are given below.
    CXX_SHARED=-shared -Wl,--disable-new-dtags

When debugging library path issues, it can be helpful to have a further knowledge of the library path resolution. The LD_DEBUG variable can be set as follows, and a full report of the library path resolution will be given.

LD_DEBUG=libs ./EXE

Here, EXE is the executable being run. Example output from this, assuming the MyPackage example, might look like the following, depending upon the system and local development area.

     16201: find library=libMyPackage.so [0]; searching
     16201:  search path=../lib/tls/x86_64:../lib/tls:../lib/x86_64:../lib:/home/contrib/develop/releases/lib       (RPATH from file /home/contrib/develop/releases/lib/libpythia8.so)
     16201:   trying file=../lib/tls/x86_64/libMyPackage.so
     16201:   trying file=../lib/tls/libMyPackage.so
     16201:   trying file=../lib/x86_64/libMyPackage.so
     16201:   trying file=../lib/libMyPackage.so
     16201:   trying file=/home/contrib/develop/releases/lib/libMyPackage.so
     16201:  search path=../../../lib/tls/x86_64:../../../lib/tls:../../../lib/x86_64:../../../lib:/home/contrib/develop/releases/lib       (RPATH from file ./testPlugins)
     16201:   trying file=../../../lib/tls/x86_64/libMyPackage.so
     16201:   trying file=../../../lib/tls/libMyPackage.so
     16201:   trying file=../../../lib/x86_64/libMyPackage.so
     16201:   trying file=../../../lib/libMyPackage.so
     16201: 
     16201: 
     16201: calling init: ../../../lib/libMyPackage.so
     16201:

From this print out, we see that a number of system and Pythia paths are tried, and then finally the library is found along the relative path ../../../lib. Note that the path /home/contrib/develop/releases/lib is tried first. This is because this is passed as a path to the linker when the shared Pythia library is compiled; this is also why resolution 2 outlined above should work. In general, if the package library is not being found, this is a very useful way to see which paths are being tried, and their ordering.

The use of new-dtags flag of resolution 4 above is necessary because of a change in the default linker behavior for a number of Linux distributions and the difference between RPATH and RUNPATH. The library path typically resolves along the following order.

  1. RPATH: this is the run-time search path that is hard-coded into compiled libraries or executables and is typically set by passing the flag -Wl,rpath,PATH to the linker through the compiler.
  2. LD_LIBRARY_PATH: is the environment variable used by the dynamic link loader if the library is not found along RPATH. A similar, but not identical variable DYLD_LIBRARY_PATH is available for MacOS.
  3. RUNPATH: this is also passed via the flag -Wl,rpath,PATH at compile time, similar to RPATH.

For older systems, when rpath is passed to the compiler, setting RPATH is the default behavior, while for newer systems RUNPATH is set instead. Consequently, depending on whether RPATH or RUNPATH is preferred, the following options should be used.

For some systems, it is possible that the disable-new-dtags and enable-new-dtags options are not available, in which case the rpath flag will typically set RPATH and not RUNPATH. Note, it is also possible to avoid passing rpath flags to the linker by instead setting the LD_RUN_PATH environment variable before running the linker.

There is a critical difference between RPATH and RUNPATH: RPATH is transitive while RUNPATH is not. In other words, the RPATH for a dependency will be used in addition to the RPATH for the binary itself. Conversely, if RUNPATH is specified, then only that RUNPATH for that binary will be used, even if a dependency specifies an RPATH. From the Linux foundation reference specifications, the following is noted about the lack of the transitive property for RUNPATH.

The set of directories specified by a given DT_RUNPATH entry is used to find only the immediate dependencies of the executable or shared object containing the DT_RUNPATH entry. That is, it is used only for those dependencies contained in the DT_NEEDED entries of the dynamic structure containing the DT_RUNPATH entry, itself. One object's DT_RUNPATH entry does not affect the search for any other object's dependencies.

This has important implications for PYTHIA-CONTRIB. If Pythia is compiled with RUNPATH, then even if RPATH is correctly specified for the executable calling Pythia, Pythia will not use this path at run time. Similarly, if the executable is compiled with RUNPATH then Pythia will not search along this path. Consequently, for Pythia to search along a path specified by the executable, then both Pythia and the executable must be compiled using RPATH and not RUNPATH.