EventServiceDeletion

EventService Deletion

  • Currently EventService deletion is handled in two workflow: 1) Rucio deletion 2) OS LifeCycle deleteion
  • By default, Rucio will delete objects older than 30 days and OS LifeCycle is defined with expiration=29 days(remove objects older than 29 days).
  • OS LifeCycle should remove all things older than 29 days. If one found that rucio deletion is working to delete things, one should check why OS lifecycle is not working.

Rucio Deleteion

  • It consists of two daemons: rucio-light-injector and rucio-light-reaper, running on "rucio-daemon-int-05". The logs can be found under /var/log/rucio/ directory.
  • rucio-light-injector works to list all older(30 days) objects and inject them to rucio temp_dids table.
  • rucio-light-reaper works to delete objects from rucio temp_dids table.

Object LifeCycle:

  • Below is an example how to configure and check the OS LifeCycle.

        import boto
        import boto.s3.connection
        from boto.s3.key import Key
        from boto.s3.lifecycle import Lifecycle, Rule, Expiration

        access_key = '**********'
        secret_key = "********"
        hostname = 's3.cern.ch'
        port = 443
        is_secure = True

        conn = boto.connect_s3(
                aws_access_key_id = access_key,
                aws_secret_access_key = secret_key,
                host = hostname,
                port = port,
                is_secure=is_secure,               # uncommmnt if you are not using ssl
                calling_format = boto.s3.connection.OrdinaryCallingFormat(),
               )

        bucket_names = []
        rs = conn.get_all_buckets()
        for b in rs:
             bucket_names.append(b.name)

        for bucket_name in bucket_names:
            print bucket_name
            bucket = conn.get_bucket(bucket_name)
            if 'eventservice' in bucket_name:
                 expiration = Expiration(days=29)
                 rule = Rule(id='expire_rule_id', status='Enabled', expiration=expiration)
                 lifecycle = Lifecycle()
                 lifecycle.append(rule)
                 bucket.configure_lifecycle(lifecycle)
            current = bucket.get_lifecycle_config()
            print current[0].transition
            print current[0].expiration

    

-- WenGuan - 2019-05-08

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2019-05-08 - WenGuan
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback