SAME development

Source code

• re-using the SFT code
• programming language: Python
• code structure: modules used by application wrapper scripts
• identify the funtionally different parts, and split them to separate modules
  • getting environment
  • job submission
  • publishing results
  • ...
• using standard python facilities for:
  1. logging, error reporting, etc.
  2. config files
• comments:
  1. must be added to the code!
  2. CVS comments must be added
  3. changelogs at the beginning of files

Getting source code

export CVSROOT=":ext:<your CERN AFS id here>@isscvs.cern.ch:/local/reps/lcgware"
export CVS_RSH=ssh
cvs co -d same same

Directory structure

            /SAME
               /client 
                  /etc
                  /libs
                  /sensors
                          /MyProxy
                          /CE
                          /LFC
                          ...
                  /var
                     /log
                  /tmp
              /server
                  /webapps
                     /etc
                     /lib
                     ...
                  /web
                     /html
                     /etc
                     /lib
                     /var
                        /results 
                           /public              *generated*
                              /MyProxy
                              /CE
                              /LFC
                              ...
                           /secure              *generated*
                              /MyProxy
                              /CE
                              /LFC
                              ...
                     ...

SAME framework

• • etc: this will conatin files only for the overall framework
  • var/results: large amount of data, location must be configureable (disk server)
  • tmp: temporary files

Sensor Specification

• Naming
  • Sensor names should be uppercase (CE, SE, LFC, etc.) or mixed-case (MyProxy)
  • Sensor directory must have the same name as the sensor will be invoked from the command line
• Files
  • Each sensor might have their own configuration opions in SAME/sensors/sensorname/etc direcotory
  • Sensors must be invoked by script SAME/sensors/sensorname/sensorname-test
  • Definition file for the sensor
    • requirement: dependency list, as for RPM .spec files
    • execTests: tests to be executed from those that are shipped with the sensor
  • Definition file for each individual test within the sensor (!) with fields:
    • testName: Sensor test name in format sensor-testname (example: CE-rm)
    • testTitle: Short name of the test (example: Replica Manager Tests)
    • testAbbreviation: Short abbreviataion (example: rm)
    • dataType: Type of data produced by test as summaryData ('number','string')
    • dataThreshold: Threshold on summaryData which causes test to fail (numbers are converted to string)
• Command-line parameters
  • -d|--debug: verbose mode
  • -h|--help: display help message
• Code
  • No restriction on language
  • Obligatory functions
    • help(): description of command-line parameters
    • publish_env(): Using SAME libraries publishing environment to R-GMA
  • No common files should be written if the sensor is executed in parallel using threads

Test execution

• test definitions will be published in parallel with the test execution

Publishing

Internal component layout

The following figure presents the interaction between the SAME framework and the data storages, where results will be published. Well-defined protocols will have to be used to access the transport schema and the storage schema.

Schemas in Oracle

TestDef : This table holds the definitions of various SFT and SAME tests, pointers to related help docs, thresholds that define the sucess of tests etc.

        CREATE TABLE TestDef (
                          testID                           Integer,   
                          testName                     varchar2(100),
                          testTitle                       varchar2(255),
                          testAbbr                       varchar2(50),
                          testHelp                       clob,
                          dataType                      varchar2(64), 
                          dataUnit                       varchar2(30),
                          dataThreshold              varchar2(64),
                          createdTimestamp       Integer,
                          primary key (testID)
);

TestDefLog : This table holds the change logs for data units and thresholds for various tests.

         CREATE TABLE TestDefLog (
                          testID                      Integer,
                          dataType                        varchar2(64),
                          dataUnit                        varchar2(30),
                          dataThreshold           varchar2(64),
                          timestamp                       Integer,
                          Primary Key (testID, timestamp),
                          Foreign Key (testID) 
                          References TestDef(testID)
          )

TestEnv : This table maintains details about environment names and their corresponding IDs

            CREATE TABLE TestEnv (
                         envID                           Integer,
                         envName                         varchar2(100), 
                         timestamp                       Integer,
                         Primary Key (envID)        
             );

            CREATE INDEX TestEnvTS on TestEnv(timestamp);

TestEnvVars : This table maintains details about environment variables used during various instances of test runs.

            CREATE TABLE TestEnvVars (
                          envID                           Integer, 
                          name                            varchar2(100), 
                          value                           varchar2(255), 
                          Primary Key (envID, name),
                          Foreign Key (envID) 
                          References TestEnv(envID)
             );

TestStatus : This table stores information about status IDs for tests.

             CREATE TABLE TestStatus (
                         statusID                        Integer,
                         statusAbbr                      varchar2(30),
                         statusDescription       varchar2(255),
                         primary key ( statusID )
             );

TestData : This table stores the results produced by various tests and sensors.

           CREATE TABLE TestData (
                       voID                            Integer, 
                       testID                          Integer, 
                       nodeID                          Integer,
                       envID                           Integer, 
                       statusID                        Integer,
                       summaryData             varchar2(255),
                       detailedData            clob,
                       timestamp                       Integer, 
                       Primary Key (voID, testID, nodeID, timestamp),
                       Foreign Key ( voID )
                       References VO ( voID ),
                       Foreign Key ( testID )
                       References TestDef ( testID ),
                       Foreign Key (NodeID)
                       References Nodes(NodeID),
                       Foreign Key (envID) 
                       References TestEnv(envID),
                       Foreign Key (statusID)
                       References TestStatus(statusID)
);

TestCriticality : This table stores the criticality of various tests defined by VOs.

           CREATE TABLE TestCriticality (
                      voID                            Integer, 
                      testID                          Integer, 
                      testVOID                     Integer, 
                      isCritical                      char(1) DEFAULT 'N' check (
                      isCritical = 'Y' OR isCritical = 'N'),
                      Primary Key (voID, testID, testVOID),
                      Foreign Key ( voID, testVOID )
                      References VO ( voID ),
                      Foreign Key ( testID )
                      References TestDef ( testID )
           );

TestCriticalityLog : This table stores the change log (history) of criticality of various tests defined by VOs.

           CREATE TABLE TestCriticalityLog (
                      voID                            Integer, 
                      testID                          Integer, 
                      testVOID                     Integer, 
                      isCritical                      char(1) DEFAULT 'N' check (
                      isCritical = 'Y' OR isCritical = 'N'),
                      timestamp                       Integer, 
                      Primary Key (voID, testID, testVOID, timestamp),
                      Foreign Key ( voID, testVOID)
                      References VO ( voID ),
                      Foreign Key ( testID )
                      References TestDef ( testID )
           );

SFTNodeSummary : This table holds summary of various SFT Tests for a particular node.

          CREATE TABLE SFTNodeSummary (
                     voID                            Integer, 
                     nodeID                          Integer,
                     statusID                        Integer,
                     timestamp                       Integer,
                     Primary Key (voID,nodeID,timestamp),   
                     Foreign Key ( voID )
                     References VO ( voID ),
                     Foreign Key (NodeID)           
                     References Nodes(NodeID),
                     Foreign Key (statusID)       
                     References TestStatus(statusID)
          );

SFTSiteSummary : This table holds summary of various SFT Tests for a particular site.

           CREATE TABLE SFTSiteSummary (
                   voID                            Integer, 
                   siteID                          Integer,
                   statusID                        Integer, 
                   timestamp                       Integer, 
                   Primary Key (voID,siteID,timestamp),
                   Foreign Key ( voID )
                   References VO ( voID ),
                   Foreign Key (siteID)
                   References Sites(siteID),
                   Foreign Key (statusID)
                   References TestStatus(statusID)
         );

MetricDef : This table stores metric definitions

         CREATE TABLE MetricDef (
                 metricID                        Integer,
                 metricName                      varchar2(100),
                 metricTitle                     varchar2(255),
                 metricAbbr                      varchar2(50),
                 metricHelp                      clob,
                 targetType                      varchar2(40) check ( targetType in 
                                                    ( 'NODE', 'SITE', 'REGION', 'GRID' ) ),
                 Primary Key (metricID)
);

NodeMetrics : This table stores metrics for various nodes

        CREATE TABLE NodeMetrics (
               voID                            Integer, 
               nodeID                          Integer,
               metricID                        Integer,
               value                           Double Precision,  
               timestamp                       Integer,
               Primary Key (voID, nodeID, metricID, timestamp),
               Foreign Key ( voID )
               References VO ( voID ),
               Foreign Key (NodeID)
               References Nodes(NodeID),
               Foreign Key (metricID)
               References MetricDef(metricID)
       );

SiteMetrics : This table stores metrics for various sites

       CREATE TABLE SiteMetrics (
               voID                            Integer, 
               siteID                          Integer,
               metricID                        Integer,
               value                           Double Precision,       
               timestamp                       Integer,
               Primary Key (voID, siteID, metricID, timestamp),
               Foreign Key ( voID )
               References VO ( voID ),
               Foreign Key (siteID)
               References Sites(siteID),
               Foreign Key (metricID)
               References MetricDef(metricID)
       );

RegionMetrics : This table stores metrics for various grid regions

        CREATE TABLE RegionMetrics (
                voID                            Integer,
                regionID                        Integer,
                metricID                        Integer,
                value                           Double Precision,       
                timestamp                       Integer,
                Primary Key (voID, regionID, metricID, timestamp),
                Foreign Key ( voID )
                References VO ( voID ),
                Foreign Key (regionID)
                References Regions(regionID),
                Foreign Key (metricID)
                References MetricDef(metricID)
       );

GridMetrics : This table stores metrics for the entire Grid i.e. LCG.

       CREATE TABLE GridMetrics (
                voID                            Integer, 
                metricID                        Integer,
                value                           Double Precision,       
                timestamp                       Integer,
                Primary Key (voID, metricID, timestamp),
                Foreign Key ( voID )
                References VO ( voID ),
                Foreign Key (metricID)
                References MetricDef(metricID)
       );

SameUser: Table to store user DN's of the SAME web portal

      CREATE TABLE  SameUser ( 
               userID Integer, 
               userDN varchar2(255), 
               Primary Key (userID) 
      );

SamePortalUserVars: Table to store user settings of the SAME web portal

      CREATE TABLE  SamePortalUserVars ( 
               userID Integer, 
               name varchar2(100), 
               value clob, 
               Primary Key      (userID, name), 
               Foreign Key (userID)      References SameUser(userID) 
      );

Other issues

• help should be extended and mainained
• CVS tags should enable the checkout only of the desired part (only SFT but not SAME, etc.) of the code