This description is not full. Nevertheless It could help developers to unstarstand how does the Master awstats work.

MasterAWStats.py

class AWStatsConfigurator

modifies config file of awstats (/opt/dashboard/var/lib/mawstats/conf/awstats.master.conf by default).

__init__(self, confPath, remoteUser, siteDomain, remoteHosts, enableDebug)

initializes an object of the AWStatsConfigurator type. Sets the path to the awstats configuration file, the remote user name (dboard by default), and the list of all remote hosts.

getAllRemoteVHosts(self)

using getRemoteVHosts, returns the the list of all virtual hosts within the initializing of AWStatsConfigurator object.

getRemoteVHosts(self, remoteHost)

gets the names of virtual hosts for the remoteHost:

ssh remoteuser@remoteHost /usr/sbin/httpd -S

The ssh login without password must be provided for the connection from the main machine to the remote hosts.

setHostAliases(self)

writes in the awstats configuration file the names of all hosts, virtual hosts and aliases using the list ShortremoteVHosts:

self.shortRemoteVHosts = [ h.split('.')[0] for h in self.remoteVHosts ]

One should note in the current realization only host names ending on "cern.ch" can be included into the configuration file correctly.

setupLoadPlugin_dahsb_apps_vhosts(self)

writes into the awstats configuration file the string LoadPlugin=dashb_appa_vhosts with all hosts. Only host names ending on "cern.ch" are included correctly.

SetupDirData(self, localDBDir)

sets the path to the awstats data base (/opt/dashboard/var/lib/mawstats by default).

=fixAwstatsConf=(self, localDBDir)

using setHostAliases, setupLoadPlugin_dahsb_apps_vhosts and setupDirData changes the existed awstats configuration file (/opt/dashboard/var/lib/mawstats/conf/awstats.master.conf by default). This file is not deleted by fixAwstatsConf, therefore sysadmin is able to edit this file by hand.

setLogFilesToParse(self, logFilesToParse)

makes a string from the log files list logFilesToParse, and after that the standard awstats script logresolmerge.pl extracts the information from logs in the list logFilesToParse and stores it into the awstats data base in directory localDBDir (/opt/dashboard/var/lib/mawstats by default). One should note, that the log files list logFilesToParse is formed within MasterAWStats class.

class MasterAWStats

init(self, remoteUser, siteDomain, remoteHosts, awststats, awstatsConfPath, localLocalLogsDir, localDBDir, awstatsStdoutFile, awstatsDebugLevel)

initializes an object of MasterAWStats type.

remoteUser is the name of remote user,

remoteHosts is the list of the remote hosts,

awstatsConfPath is the path to the config file (/opt/dashboard/var/lib/mawstats/conf/awstats.master.conf by default),

localLogsDir is the directory where the httpd log files from remote hosts are stored.

checkConfiguration()

Within the ini method checks:

1. Whether there is the config file awstatsConfPath.

2. Whether there is the directory localDBDir for awstats data base.

3. Whether there is the directory localLogsDir for httpd log files from the remote host.

4. Whether the remote hosts exist.

runScpProcess(self, remoteHost, remoteLogFilePath, localLogFilePath)

copies httpd log files from remote host to the main machine preserving modification time, access time, and mode from the original files using passwordless scp:

scp -p remoteuser@remoteHost:/remoteLogFilePath localLogFilePath

statRemoteLogFile(self, remoteHost, remoteLogFilePath)

using passwordless ssh determines the modification time of file RemoteLogFilePath and returns it in the variable RemoteModifyTimestamp.

writeLocalLogFileTimestamp(self, localLogFilePath, timeStamp = None)

checks is there a local file LocalLogFilePath and, if so, then it determines the time of the modification and writes to a file localLogFilePath.timestamp.

downloadLogFiles

downloads httpd log files from the remote hosts. It creates two empty vocabularies:

logFilesToParse = {};
and
pidLogs={}.

and then iterates the list remoteLogFiles, which consisted from pairs (remote host name, the hhtpd log file on this remote host with the total path).

Within this iteration the method finds a file with the same name on the local machine, and if such a file differ from the remote one or does not exist at all the corresponding local file is rewritten or created by method runScpProcess. One should note that the runScpProcess is executed in parallel process (os.fork() is used). runScpProcess returns the id the subprocess.

After that the following element is added to the vocabulary pidLogs:

pidLogs[pid]= (uri of the remote httpd log file, the name of appropriate local file)

Then the same files with extention ".1" are processed.

Using passwordless ssh the method reads the timestap of this log file and compares it with the local file timestap. If the timestaps are different, the remote file is copied to the main machine.

if the log file is copied successfully then the following element is written into vocabularylogFilesToParse:

logFilesToParse[localLogFilePath]=1

Then the method downloadLogFiles returns the list of file to be processed by awstats (sorted(logFilesToParse.key())).

buildDataBase(self)

initializes an object configurator of the AWStatsConfigurator type.

The configurator receives the path to the awstats config file (/opt/dashboard/var/lib/mawstats/conf/awstats.master.conf by default), the remote user name, and the list of remote hosts.

It is worth to note that the initializing of AWStatsConfigurator object does not change the awstats configuration file itself. This file is changed by method fixAwstatsConf.

After that the pairs (host name, /var/log/httpd/access_log) is added to the list remoteLogFiles.

It should be noted, that in the current realization the path to the httpd log file on the remote machine is fixed. It must be /var/log/httpd/access_log on all remote machines.

Then the method downloadlogFiles downloads remote httpd logs and returns the file list logFilesToParse to be processed by standard awstats script awstats.pl:

... awstats.pl -update -config=...

MasterAWStatsService.py

class MasterAWStatsService

The class MasterAWStatsService has been developed to include the master awstats into the dashboard package as a dashboard-service.

__init__(self, name, configFile)

initializes an object of MasterAWStatsService type, reads the parameter values from the service configuration file (/opt/dashboard/etc/dashboard-service-config/service-config.xml), sets default values for the parameters which were not defined in the configuration file.

run(self)

initializes an object master of MasterAWStats type, using established parameter values.

Execute master.buildDataBase().

Then it sleeps 60 second (by default) to avoid the race condition between the master awstats and the log rotation (logrotate). Thus it allows logs to rotate first (give them 60 seconds). After that these logs are copied to main machine.

-- AlexanderBerezhnoy - 08-Aug-2011

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2011-08-09 - AlexanderBerezhnoy
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    ArdaGrid All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback