Batch Accounting Data Sources

Table of Contents


Introduction

This article lists various web services and databases that can be used for retrieving information related to CPU accounting.

Unless stated otherwise, all web-based APIs require a valid Kerberos certificate.

LSF - AFS

Todo

/afs/cern.ch/project/lsf/7.0/mnt/batch/work/batch/

LSF - Shares

Get information on shares.

Format: XML

Endpoint

https://service-lsfweb.web.cern.ch/service-lsfweb/xml/shares

Output

LSFSHARESINFO is a list of LSFSHARE elements.

Field Description
LSFGROUP The LSF group that owns this share.
GROUPSHARE Unknown - Assumed total allocated resources for group?
DEDICATED If yes, the share is always guaranteed to have allocated resources available.
HOSTPARTITION The host partition where the share is located. See the HOSTPARTITION values section below.
LSFSUBGROUP The LSF subgroup that this share is assigned to.
LSFSUBGROUPALIAS The alias for the LSF subgroup.
SUBGROUPFRACTION An allocated percentage of resources on the host partition assigned to the subgroup.
USERS A comma-delimited string of users associated with this share.

HOSTPARTITION values

HOSTPARTITION is observed to have the following values:

  • ATLBOGUS - dedicated partition for the ATLAS experiment.
  • CMSCAF - dedicated partition for the CMSCAF queue.
  • LHCBCAF - dedicated partition for the LHCb experiment.
  • SHARE - general use.
If DEDICATED is no, HOSTPARTITION will usually have a value of SHARE.

Example Output

<LSFSHARESINFO>
<LSFSHARE LSFGROUP="u_ALEPH" GROUPSHARE="1" DEDICATED="no" HOSTPARTITION="SHARE" LSFSUBGROUP="xu_PROD" LSFSUBGROUPALIAS="" SUBGROUPFRACTION="50" USERS="almcbew,almcqcd,almcww"/>
...
</LSFSHARESINFO>

Foreman - Hosts

Get information on Foreman hosts.

Format: JSON

Endpoint

https://judy.cern.ch:8443/api/hosts

Output

The output is a list of objects that contain information about each host.

Field Description
name The hostname.
id The ID.
hostgroup_id The hostgroup ID.
operatingsystem_id The operating system ID.

Example Output

[
{
"host": {
"name": "agrigore-server-02.cern.ch",
"id": 45211,
"hostgroup_id": 4051,
"operatingsystem_id": 72
}
},
...
]

Puppet - Nodes

Get information on Puppet nodes.

Format: JSON

Endpoint

https://judy.cern.ch:9081/v3/nodes

Output

The output is an array of objects that contain information about each host.

Field Description
name The hostname.
deactivated ??
catalog_timestamp ??
facts_timestamp ??
report_timestamp ??

Example Output

[
{ "name" : "adfs11.cern.ch", "deactivated" : null, "catalog_timestamp" : "2015-09-08T07:25:54.212Z", "facts_timestamp" : "2015-09-08T07:25:39.396Z", "report_timestamp" : "2015-09-08T07:25:44.561Z" },
...
]

HardwareDB - Hosts

This endpoint exposes detailed hardware information for physical devices.

Requirements

The hwcollect service requires a number of credentials:

  • A valid Kerberos certificate.
  • A valid local certificate and private key.
  • The CERN root CA.
curl https://lxbsu2004.cern.ch:9000/hwinfo/_design/hwinfo/_view/hostcurrent --cert [path to your cert] --key [path to your key] --negotiate -u : --cacert [path to CERN CA bundle]

Endpoint

https://lxbsu2004.cern.ch:9000/hwinfo/_design/hwinfo/_view/hostcurrent

Output

Output is pretty straightforward.

Example Output

{
    "total_rows": 13678,
    "offset": 0,
    "rows": [
        {
            "id": "CD5153026 CR018059A",
            "key": "acron11.cern.ch",
            "value": {
                "WARRANTY": "CD5153026",
                "BMC": [
                    {
                        "Board Serial": "",
                        "IPMI Version": "2.0",
                        "Vendor": "Winbond",
                        "Product Serial": "CR018059A",
                        "MAC Address": "00:25:90:70:43:04",
                        "Manufacturer Name": "Winbond",
                        "mc Product Name": "Unknown (0xAA11)",
                        "Firmware Revision": "2.50",
                        "IP Address": "10.9.202.40",
                        "Product Asset Tag": "CD5153026",
                        "Device Available": "yes",
                        "Serial": "CR018059A",
                        "Model": null,
                        "Board Mfg": "Supermicro",
                        "Board Mfg Date": "Mon Jan  1 00:00:00 1996"
                    }
                ],
                "ATA": {
                    "CONTROLLERS": {
                        "GENERIC0": {
                            "Vendor": "Advanced Micro Devices [AMD] nee ATI",
                            "Cache": null,
                            "Driver": "ahci",
                            "Ports": { /* An entry for each ATA device, keyed by {Model} {Serial} */
                                "WD2003FYYS-02W0B1 WD-WCAY01097802": {
                                    "Vendor": "WDC",
                                    "Enclosure": null,
                                    "Version": "01.01D02",
                                    "Device": "sdb",
                                    "Serial": "WD-WCAY01097802",
                                    "Port": 1,
                                    "Model": "WD2003FYYS-02W0B1",
                                    "Smart": "Enabled",
                                    "Size": 1907729
                                },
... }, "Type=": "sata", "Version": null, "Location": "/sys/devices/pci0000:00/0000:00:11.0", "Serial": null, "Model": "SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]", "BBU": null } }, "Total Space": "5723187.0", "No. of Disks": "3" }, "NETWORKDETAILS": { "cluster": "Unknown", "interfaces": {}, "hostname": "acron11.cern.ch" }, "NIC": { /* An entry for each network interface card keyed by MAC address */ "00:25:90:70:45:05": { "boundipv4": null, "boundipv6": null, "maxspeed": 1024000, "bus": "pci", "netmask": null, "version": "1.53, 0x0000f0f6", "location": "/sys/devices/pci0000:00/0000:00:04.0/0000:03:00.1", "media": "ethernet", "hwaddr": "00:25:90:70:45:05", "model": "82576 Gigabit Network Connection", "manufacturer": "Intel Corporation" },
... }, "RAM": { "No. of DIMMS": "8", "Total Memory": "65536", "DIMMS": { /* An entry for each memory module keyed by the serial */ "1294D7BC": { "Vendor": "Samsung", "DataRate": "1600", "Location": "P1_DIMM3A", "Model": "M393B1K70DH0-YK0", "Type": "DDR3", "Serial": "1294D7BC", "Size": "8192" },
... } }, "CPU": { "Physical Processors": "2", "Logical Processors": "16", "CPUS": { /* An entry for each physical CPU */ "CPU0": { "Vendor": "AuthenticAMD", "SMT": true, "LogicalCores": 16, "Cores": 8, "Model": "AMD Opteron(TM) Processor 6276", "Speed": 2300 },
... } }, "FCA": [], "DOC FROM": "bmc", "SERIAL": "CR018059A", "BIOS": { "version": "2.0b (07/13/2012)", "vendor": "American Megatrends Inc.", "settings": {} } } },
... ] }

LANDB

host = pdb-s.cern.ch

port = 10121

service = PDB_CERNDB1.cern.ch

?? Needs sorting out; hwacct credentials seem to not have the permissions to read from the table it needs

Meter - Elastic

Retrieves node information from Meter. This is an Elastic cluster API endpoint.

Format: JSON

Endpoint

https://meter.cern.ch/public/_nodes

Input

Refer to https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html.

Output

Refer to https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-info.html.

Elastic - HEPSPEC

todo: fill this out.

Format: JSON

meter-cli -m 13273 --start "2015-07-01 00:00:00" -n "hepspec-jmakai" --limit 20000 --json

Endpoint

meter.cern.ch

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r6 - 2015-09-09 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback