JSONCollector < Sandbox

---+ JSONCollector *(DEPRECATED)*

%TOC%

---++ Description

The JSONCollector is a tool that enables summing FFF monitoring files. The two main uses of the tool are:
   1. Binary: loading json files and generating a single output file, representing the summed output
   1. API: used by CMSSW processes to output monitoring files respecting a pre-defined structure of the monitorable fields

In order to improve efficiency, the json files are split into *legend* and *data* files. Legend files describe the structure of a data file, and are split into a few categories (e.g. monitoring files generated by the rubuilder, monitoring files generated by the filter processes, monitoring files describing data files output by the filtering processes, etc.). Data files are instances of the different monitored fields, respecting a file type described by the legend.

The JSONCollector implementation uses the _jsoncpp_ library: http://jsoncpp.sourceforge.net/

---++ File categories

There are 3 main categories of files the collector works with:

   1. JSON data files: monitoring files in the _json_ format that respect a legend (also referred to as _definition_).
   1. CSV data files: monitoring files in the _csv_ format that respect a legend. These files are used for fast monitoring in the FFF (monitoring information output and collected every second).
   1. Legend files: describe the JSON and CSV data files, providing the operation to be performed on each field of the data files

For usage examples and monitoring file types in the FFF, please see: [link]

---++ Get the code
Code is available on the cms-hlt git repo: *https://git.cern.ch/web/?p=cms-hlt.git*

---++ Useful Links

* [[FFFMetafileFormats][FFF Metafile Formats]]

---++ Binary

The command line tool loads json or csv data files from a directory (optionally using a regex for the file name) and outputs the aggregated result according to the legend. Input files must have the same legend and be consistent.

Usage: <strong>$jsoncollector [-json/-csv] [-d] [-r regex] [-o outfile] [-i indir1 indir2 ...infileN]</strong>

where:
   * *[-json/-csv]*: the type of input files
   * *[-d]* (optional): output-for-display flag. By setting this flag, the collector outputs a json file that combines the summed data together with the legend information for this field. This is useful for display purposes, providing a self-describing json file. *This file cannot be aggregated again since it no longer respects the format for data files, but should be used only when a human-readable result is required.*
   * *[-r regex]* (optional): regular expression for the names of files in the input space. The regular expression applies to the file name *stub* (which is /path/to/<strong>myMonitoringFile</strong>.jsn)
   * *[-o outfile]*: full path of the output file
   * *[-i indir1 indir2 ...infileN]*: list of input files and directories to get the files from


---++ API: writing CSV (fast) and JSON (slow) files

%SYNTAX{ syntax="cpp" }%

#include "JSONCollector/interface/JsonMonitorable.h"
#include "JSONCollector/interface/FastMonitor.h"
#include "JSONCollector/interface/JSONSerializer.h"

#include <vector>

using namespace std;
using namespace jsoncollector;

class Monitored {

public:
	IntJ processedEvents;
	IntJ acceptedEvents;
	IntJ microstate;
	IntJ ministate;
	IntJ macrostate;
};

int main() {

	Monitored mon;

        // set names of the variables to be matched with JSON Definition
	mon.processedEvents.setName("Processed Events");
	mon.acceptedEvents.setName("Accepted Events");
	mon.microstate.setName("Microstate");
	mon.macrostate.setName("Ministate");
	mon.macrostate.setName("Macrostate");

        // create a vector of all monitorable parameters to be passed to the monitor
	vector<JsonMonitorable*> monParams;
	monParams.push_back(&mon.processedEvents);
	monParams.push_back(&mon.acceptedEvents);
	monParams.push_back(&mon.macrostate);
	monParams.push_back(&mon.ministate);
	monParams.push_back(&mon.microstate);

        // create a FastMonitor using vector of monitorable parameters, a path to a JSON Definition file and the output file path
	FastMonitor monitor (monParams, "/path/to/definition.jsd");

        // change the monitored parameters
	mon.processedEvents = 100;
	mon.acceptedEvents = 76;
	mon.microstate = 3;
	mon.macrostate = 1;

        // take a snapshot of the monitored values and output to a CSV file
	monitor.snap(true, "/path/to/outputCSVFile.csv");

         // change the monitored parameters again
	mon.processedEvents = 200;
	mon.acceptedEvents = 150;
	mon.microstate = 9;
	mon.macrostate = 1;

        // same as before, updating the file
	monitor.snap(true, "/path/to/outputCSVFile.csv");

        // do something else ...

        // output a slow monitoring file, summing all the information from the snapshots
        monitor.outputFullHistoDataPoint("/path/to/outputJSONFile.jsn");

	return 0;
}
%ENDSYNTAX%

For the above code we need a definition file at the specified path. The output format will be given by the definition at _/path/to/definition.jsd_.

<verbatim>
{
   "legend" : [
      {
         "name" : "Processed Events",
         "operation" : "sum"
      },
      {
         "name" : "Accepted Events",
         "operation" : "sum"
      },
      {
         "name" : "Microstate",
         "operation" : "histo"
      },
      {
         "name" : "Ministate",
         "operation" : "histo"
      },
      {
         "name" : "Macrostate",
         "operation" : "histo"
      }
   ]
}
</verbatim>

-- Main.AndreiSpataru - 14-Aug-2013
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	JSONCollector_classes.png	r1	manage	353.1 K	2012-10-10 - 18:02	AndreiSpataru	Class diagram
Topic revision: r19 - 2013-08-14 - AndreiSpataru
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback