Auxiliary Resource Monitoring and Data Aggregation Services

Introduction

Panda services running at BNL (or any other location if desired) are using a number of service scripts that facilitate some aspects of the system operations. In particular, we are maintaining a RRD database of the load profile of the servers. In addition, we have automated data feeds into the Panda database which populate certain attributes of Panda sites based on the information they collect and aggregate, which is coming from a few different sources in Open Science Grid. There are also a group of scripts that perform periodic cleanup of various logging directories and files.

Resource Monitoring scripts

This is not to be confused with Panda Monitor described in separate pages. Essentially, we want a Ganglia-like functionality, but with added ability to focus on processes of our choice, which most often are specific instances of Apache server which host both the Panda Server and Panda Monitoring side.

In terms of implementation, we have a process that collects the data and makes entries in RRD, and a separate script that produces a graph, in the form of a graphic file that get copied to an appropriate area of the web space accessible via Panda Monitor.

Data Aggregation Services

Cleanup Scripts

At the time of this writing, removing log files left over and characterizing user jobs, older than 30 days (which is acceptable from the auditing standpoint) allows us to keep disk utilization of 7 to 15% on each production machine. These policies can be modified as needed.

SVN repository

The code is maintained in the SVN repository.


Major updates:
-- MaximPotekhin - 02 Dec 2008



Responsible: MaximPotekhin

Never reviewed

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2008-12-02 - MaximPotekhin
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback