Top Analysis Classification Tool

The top analysis classification tool (TACT) is a frontend for multiple machine learning libraries, designed for use in typical signal vs. background classification tasks in analyses. It is available at https://github.com/brunel-physics/tact.

About the tool

TACT is designed as a frontend to multiple implementations of a range of machine learning classification techniques, configured entirely through a single YAML file. A brief introduction to the YAML syntax can be found here, and the full specification here.

TACT can read in data from ROOT files, perform a classification, and write the resluting discriminant to ROOT files compatible with the Higgs Analysis Combined Limit Tool (SWGuideHiggsAnalysisCombinedLimit) and Theta.

The training of the classifier is performed on some fraction of the input data as specified by the user. This classifier is then applied to the remainder of the input data to check the classifier performs consistently on unseen data.

Available classifiers

Boosted Decision Tree

TACT provides three different implementations of boosted decision trees (BDTs), provided by the scikit-learn, XGBoost, and LightGBM.

The scikit-learn implementation is the most primitive and least performant but still has feature-parity with TMVA and is the only one capable of using the AdaBoost boosting algorithm (gradient boosting is used by XGBoost, LightGBM, and in scikit-learn by default). More information on this classifier and its parameters can be found in the scikit-learn documentation.

XGBoost is an alternative BDT implementation, more performant in both classification strength and compute time than scikit-learn in some benchmarks. Unlike scikit-learn it is capable of using multiple CPU threads, and so will see further improvements in situations where multiple threads are available. Techniques such as L1L2 regularisation and dart boosting are available, unlike in scikit-learn. A full accounting of options can be found in the XGBoost documentation

Finally, LightGBM is a newer library with roughly the same featureset as XGBoost and similar-to-better performance. Unlike scikit-learn and XGBoost it grows trees until a maximum number of leaves is reached, rather than a maximum depth. A full list of options can be found in the LightGBM documentation.

Random Forest

A Random Forest classifier is provided via scikit-learn. Documentation.

LightGBM is also able to act as a random forest classifier if the boosting method is set to rf.

Neural Network

TACT is able to use the Keras library to construct, train, and apply neural networks. Both simple, shallow multi-layer perceptrons are possible along with deeper, more complicated networks.

Keras itself can use one of three backends for computation: Theano, TensorFlow, and CNTK. The latter two are preferred if computation is to be performed on a GPU.

Getting Started

TACT requires Python >= 2.7.13 and ROOT >= 6.06.08 with python 2 bindings.

To install TACT, download the source tree and run

pip install -e .

in the base directory. This will install TACT and a minimal set of dependencies which will enable only scikit-learn based classifiers.

For XGBoost classifiers, specify

pip install -e .[xboost]

and for LightGBM support specify

pip install -e .[lightgbm]

Finally, Keras support is enabled by

pip install -e .[MLP]

but this will not include a Keras backend, which must be installed separately.

Alternatively, TACT can be run from the base directory without installation by

python -m tact.tact

but the dependencies will have to be installed manually. These dependencies are (2018-05-10):

  • scipy
  • matplotlib
  • pandas
  • dill
  • root_pandas
  • numpy
  • root_numpy
  • more_itertools
  • scikit_learn
  • PyYAML

Optional dependencies:

  • Keras
  • xgboost
  • lightgbm

Running tact

Once installed, tact can be run via

tact configuration.yaml

where configuration.yaml name of the configuration file. A well-documnted configuration file demonstarting the creation of a scikit-learn BDT can be found here.

A confguration file can also be specfied via stdin by setting --stdin:

tact --stdin

Finally, to run TACT from its base directory:

python -m tact.tact configuration.yaml

Neural Network Specificaton

In the TACT config, neural networks can are specfied using YAML. This section includes examples how the mlp entry in a TACT configuration file can be used to specify a variety of neural networks.

Single hidden layer MLP

mlp:
    model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

    model_params:
        epochs: 100
        verbose: 1
        batch_size: 256

    compile_params:
        loss: binary_crossentropy
        optimizer: adam

The above snippet describes a simple multi-layer perceptron with a single hidden layer of 32 neurons.

The model entry under mlp describes the structure of the model itself. First the backend is specified, which can be theano, tensorflow, or cntk.

Next, class_name: Sequential specified that the model is a linear sequence of layers. This is the only model type supported by TACT. The immediately following config entry specified the layers themselves, in a sequential list. The input layer is automatically specified, so the only layers listed are the hidden and output layer.

The first layer listed is the single hidden layer. Each layer's type is given by class_name, which can be any layer provided by Keras (see the docmetation for a complete list). The Dense layer is a standard fully-connected NN layer. Additional configuration of a layer is performed in its associated config entry. Any values here are passed to the constructor of that layer type in Keras, so review the Keras documentation of that layer type for more information. In this case the activation function is set using activation to be sigmoid and the width of the layer to be 32 by specifying units.

The second and final layer listed is the output layer. The activation for this layer must be sigmoid.

model_params are passed to Keras.Sequential.fit and can be used to specify the number epochs, batch size, and verbosity.

Finally, compile_params are passed to Keras.Sequential.compile and can be used to specify the loss function to be optimised and optimizer used. For more information on the available loss functions see here and for available optimisers here. TACT does not currently support the configuration of optimisers.

Multiple hidden layers

mlp:
    model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

    model_params:
        epochs: 100
        verbose: 1
        batch_size: 256

    compile_params:
        loss: binary_crossentropy
        optimizer: adam

The above YAML snippet is identical to that used to specify the single hidden layer MLP above, but duplicates the hidden layer entry to produce a network with two hidden layers.

Callbacks

TACT currently allows two callbacks to be configured. These are triggered when certain conditions are reached during the model's training.

Early stopping

mlp:
    model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

    model_params:
        epochs: 100
        verbose: 1
        batch_size: 256

    compile_params:
        loss: binary_crossentropy
        optimizer: adam

    early_stopping_params:
        monitor: loss
        min_delta: 0.01
        patience: 5

Early stopping allows the training of the neural network to cease if a specified metric does not show any improvement. Here the loss is monitored, and if the loss (monitor) does not reduce by more than 0.001 (min_delta) for 5 consecutive epochs (patience), the training will stop.

See the Keras documentation on early stopping for more information

Learning rate reduction

mlp:
    model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

    model_params:
        epochs: 100
        verbose: 1
        batch_size: 256

    compile_params:
        loss: binary_crossentropy
        optimizer: adam

    lr_reduction_params:
        monitor: loss
        factor: 0.2
        patience: 2

Learning rate reduction is similar to early stopping, but multiplies the learning rate by factor instead of halting the training. More information.

L1L2 Regularisation

model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
            activity_regularizer:
               class_name: L1L2
               config: {l1: 1e-3, l2: 0.0}
            kernel_regularizer:
               class_name: L1L2
               config: {l1: 0.0, l2: 1e-2}
            bias_regularizer:
               class_name: L1L2
               config: {l1: 1e-3, l2: 1e-2}
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

The above model specification demonstrates how L1, L2, and L1L2 regularisation can be applied to the layers of a model.

Dropout Regularization

model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: sigmoid
            units: 32
        - class_name: Dropout
          config:
            rate: 0.2
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

Dropout regularisation is performed by adding a Dropout layer.

Self-normalising Neural Network

model:
        backend: theano
        class_name: Sequential
        config:
        - class_name: Dense
          config:
            activation: selu
            kernel_initializer: lecun_normal
            units: 32
        - class_name: AlphaDropout
          config:
              rate: 0.25
        - class_name: Dense
          config:
            activation: selu
            kernel_initializer: lecun_normal
            units: 32
        - class_name: AlphaDropout
          config:
              rate: 0.1
        - class_name: Dense
          config:
            activation: sigmoid
            units: 1

The above snippet shows how a self-normalising neural network could be specified.

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2018-05-11 - CorinJamesKeirHoad
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback