SAME development



Source code

  • re-using the SFT code
  • programming language: Python
  • code structure: modules used by application wrapper scripts
  • identify the funtionally different parts, and split them to separate modules
    • getting environment
    • job submission
    • publishing results
    • ...
  • using standard python facilities for:
    1. logging, error reporting, etc.
    2. config files
  • comments:
    1. must be added to the code!
    2. CVS comments must be added
    3. changelogs at the beginning of files


Getting source code

Source code is in CVS :
export CVSROOT=":ext:<your CERN AFS id here>@isscvs.cern.ch:/local/reps/lcgware"
export CVS_RSH=ssh
cvs co -d same same

Directory structure


            /SAME
               /client 
                  /etc
                  /libs
                  /sensors
                          /MyProxy
                          /CE
                          /LFC
                          ...
                  /var
                     /log
                  /tmp
              /server
                  /webapps
                     /etc
                     /lib
                     ...
                  /web
                     /html
                     /etc
                     /lib
                     /var
                        /results 
                           /public              *generated*
                              /MyProxy
                              /CE
                              /LFC
                              ...
                           /secure              *generated*
                              /MyProxy
                              /CE
                              /LFC
                              ...
                     ...

SAME framework

    • etc: this will conatin files only for the overall framework
    • var/results: large amount of data, location must be configureable (disk server)
    • tmp: temporary files

Sensor Specification

  • Naming
    • Sensor names should be uppercase (CE, SE, LFC, etc.) or mixed-case (MyProxy)
    • Sensor directory must have the same name as the sensor will be invoked from the command line
  • Files
    • Each sensor might have their own configuration opions in SAME/sensors/sensorname/etc direcotory
    • Sensors must be invoked by script SAME/sensors/sensorname/sensorname-test
    • Definition file for the sensor
      • requirement: dependency list, as for RPM .spec files
      • execTests: tests to be executed from those that are shipped with the sensor
    • Definition file for each individual test within the sensor (!) with fields:
      • testName: Sensor test name in format sensor-testname (example: CE-rm)
      • testTitle: Short name of the test (example: Replica Manager Tests)
      • testAbbreviation: Short abbreviataion (example: rm)
      • dataType: Type of data produced by test as summaryData ('number','string')
      • dataThreshold: Threshold on summaryData which causes test to fail (numbers are converted to string)
  • Command-line parameters
    • -d|--debug: verbose mode
    • -h|--help: display help message
  • Code
    • No restriction on language
    • Obligatory functions
      • help(): description of command-line parameters
      • publish_env(): Using SAME libraries publishing environment to R-GMA
    • No common files should be written if the sensor is executed in parallel using threads


Test execution

submission.gif

  • test definitions will be published in parallel with the test execution


Publishing

SAME-architecture.png

Internal component layout

The following figure presents the interaction between the SAME framework and the data storages, where results will be published. Well-defined protocols will have to be used to access the transport schema and the storage schema.

protocols-1.gif


Schemas in Oracle

TestDef : This table holds the definitions of various SFT and SAME tests, pointers to related help docs, thresholds that define the sucess of tests etc.


        CREATE TABLE TestDef (
                          testID                           Integer,   
                          testName                     varchar2(100),
                          testTitle                       varchar2(255),
                          testAbbr                       varchar2(50),
                          testHelp                       clob,
                          dataType                      varchar2(64), 
                          dataUnit                       varchar2(30),
                          dataThreshold              varchar2(64),
                          createdTimestamp       Integer,
                          primary key (testID)
);

TestDefLog : This table holds the change logs for data units and thresholds for various tests.

         CREATE TABLE TestDefLog (
                          testID                      Integer,
                          dataType                        varchar2(64),
                          dataUnit                        varchar2(30),
                          dataThreshold           varchar2(64),
                          timestamp                       Integer,
                          Primary Key (testID, timestamp),
                          Foreign Key (testID) 
                          References TestDef(testID)
          )

TestEnv : This table maintains details about environment names and their corresponding IDs


            CREATE TABLE TestEnv (
                         envID                           Integer,
                         envName                         varchar2(100), 
                         timestamp                       Integer,
                         Primary Key (envID)        
             );

            CREATE INDEX TestEnvTS on TestEnv(timestamp);

TestEnvVars : This table maintains details about environment variables used during various instances of test runs.

            CREATE TABLE TestEnvVars (
                          envID                           Integer, 
                          name                            varchar2(100), 
                          value                           varchar2(255), 
                          Primary Key (envID, name),
                          Foreign Key (envID) 
                          References TestEnv(envID)
             );

TestStatus : This table stores information about status IDs for tests.

             CREATE TABLE TestStatus (
                         statusID                        Integer,
                         statusAbbr                      varchar2(30),
                         statusDescription       varchar2(255),
                         primary key ( statusID )
             );

TestData : This table stores the results produced by various tests and sensors.

           CREATE TABLE TestData (
                       voID                            Integer, 
                       testID                          Integer, 
                       nodeID                          Integer,
                       envID                           Integer, 
                       statusID                        Integer,
                       summaryData             varchar2(255),
                       detailedData            clob,
                       timestamp                       Integer, 
                       Primary Key (voID, testID, nodeID, timestamp),
                       Foreign Key ( voID )
                       References VO ( voID ),
                       Foreign Key ( testID )
                       References TestDef ( testID ),
                       Foreign Key (NodeID)
                       References Nodes(NodeID),
                       Foreign Key (envID) 
                       References TestEnv(envID),
                       Foreign Key (statusID)
                       References TestStatus(statusID)
);

TestCriticality : This table stores the criticality of various tests defined by VOs.

           CREATE TABLE TestCriticality (
                      voID                            Integer, 
                      testID                          Integer, 
                      testVOID                     Integer, 
                      isCritical                      char(1) DEFAULT 'N' check (
                      isCritical = 'Y' OR isCritical = 'N'),
                      Primary Key (voID, testID, testVOID),
                      Foreign Key ( voID, testVOID )
                      References VO ( voID ),
                      Foreign Key ( testID )
                      References TestDef ( testID )
           );

TestCriticalityLog : This table stores the change log (history) of criticality of various tests defined by VOs.

           CREATE TABLE TestCriticalityLog (
                      voID                            Integer, 
                      testID                          Integer, 
                      testVOID                     Integer, 
                      isCritical                      char(1) DEFAULT 'N' check (
                      isCritical = 'Y' OR isCritical = 'N'),
                      timestamp                       Integer, 
                      Primary Key (voID, testID, testVOID, timestamp),
                      Foreign Key ( voID, testVOID)
                      References VO ( voID ),
                      Foreign Key ( testID )
                      References TestDef ( testID )
           );

SFTNodeSummary : This table holds summary of various SFT Tests for a particular node.

          CREATE TABLE SFTNodeSummary (
                     voID                            Integer, 
                     nodeID                          Integer,
                     statusID                        Integer,
                     timestamp                       Integer,
                     Primary Key (voID,nodeID,timestamp),   
                     Foreign Key ( voID )
                     References VO ( voID ),
                     Foreign Key (NodeID)           
                     References Nodes(NodeID),
                     Foreign Key (statusID)       
                     References TestStatus(statusID)
          );

SFTSiteSummary : This table holds summary of various SFT Tests for a particular site.

           CREATE TABLE SFTSiteSummary (
                   voID                            Integer, 
                   siteID                          Integer,
                   statusID                        Integer, 
                   timestamp                       Integer, 
                   Primary Key (voID,siteID,timestamp),
                   Foreign Key ( voID )
                   References VO ( voID ),
                   Foreign Key (siteID)
                   References Sites(siteID),
                   Foreign Key (statusID)
                   References TestStatus(statusID)
         );

MetricDef : This table stores metric definitions

         CREATE TABLE MetricDef (
                 metricID                        Integer,
                 metricName                      varchar2(100),
                 metricTitle                     varchar2(255),
                 metricAbbr                      varchar2(50),
                 metricHelp                      clob,
                 targetType                      varchar2(40) check ( targetType in 
                                                    ( 'NODE', 'SITE', 'REGION', 'GRID' ) ),
                 Primary Key (metricID)
);

NodeMetrics : This table stores metrics for various nodes

        CREATE TABLE NodeMetrics (
               voID                            Integer, 
               nodeID                          Integer,
               metricID                        Integer,
               value                           Double Precision,  
               timestamp                       Integer,
               Primary Key (voID, nodeID, metricID, timestamp),
               Foreign Key ( voID )
               References VO ( voID ),
               Foreign Key (NodeID)
               References Nodes(NodeID),
               Foreign Key (metricID)
               References MetricDef(metricID)
       );


SiteMetrics : This table stores metrics for various sites

       CREATE TABLE SiteMetrics (
               voID                            Integer, 
               siteID                          Integer,
               metricID                        Integer,
               value                           Double Precision,       
               timestamp                       Integer,
               Primary Key (voID, siteID, metricID, timestamp),
               Foreign Key ( voID )
               References VO ( voID ),
               Foreign Key (siteID)
               References Sites(siteID),
               Foreign Key (metricID)
               References MetricDef(metricID)
       );

RegionMetrics : This table stores metrics for various grid regions

        CREATE TABLE RegionMetrics (
                voID                            Integer,
                regionID                        Integer,
                metricID                        Integer,
                value                           Double Precision,       
                timestamp                       Integer,
                Primary Key (voID, regionID, metricID, timestamp),
                Foreign Key ( voID )
                References VO ( voID ),
                Foreign Key (regionID)
                References Regions(regionID),
                Foreign Key (metricID)
                References MetricDef(metricID)
       );

GridMetrics : This table stores metrics for the entire Grid i.e. LCG.

       CREATE TABLE GridMetrics (
                voID                            Integer, 
                metricID                        Integer,
                value                           Double Precision,       
                timestamp                       Integer,
                Primary Key (voID, metricID, timestamp),
                Foreign Key ( voID )
                References VO ( voID ),
                Foreign Key (metricID)
                References MetricDef(metricID)
       );

SameUser: Table to store user DN's of the SAME web portal

      CREATE TABLE  SameUser ( 
               userID Integer, 
               userDN varchar2(255), 
               Primary Key (userID) 
      );

SamePortalUserVars: Table to store user settings of the SAME web portal

      CREATE TABLE  SamePortalUserVars ( 
               userID Integer, 
               name varchar2(100), 
               value clob, 
               Primary Key      (userID, name), 
               Foreign Key (userID)      References SameUser(userID) 
      );


Other issues

  • help should be extended and mainained
  • CVS tags should enable the checkout only of the desired part (only SFT but not SAME, etc.) of the code



-- Main.jnovak - 14 Jan 2006
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng SAME-architecture.png r1 manage 119.0 K 2011-06-21 - 16:11 AndresAeschlimann  
GIFgif protocols-1.gif r1 manage 10.0 K 2006-02-24 - 16:51 JuliaAnnGraySecondary  
GIFgif protocols.gif r1 manage 18.4 K 2006-02-09 - 10:43 JuliaAnnGraySecondary  
GIFgif publishing.gif r1 manage 8.6 K 2006-01-18 - 18:16 JuliaAnnGraySecondary  
GIFgif submission.gif r1 manage 11.3 K 2006-01-18 - 18:17 JuliaAnnGraySecondary  
Edit | Attach | Watch | Print version | History: r21 < r20 < r19 < r18 < r17 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r21 - 2011-06-22 - AndresAeschlimann
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback