On-going

• OTG:0063097 – Intermittent network issues affecting CMSR database – from Fri Mar 30th, 20:30 – last update Thu Apr 08, 2021 09:23 - Random network connectivity issues when connecting to the CMS offline (Oracle) database. The problem seems related to IP resolution of some DNS entries used the clusterware software. The DNS definition is correct, but it is not properly handled somehow. Root cause: Problem mainly linked with extreme high load caused by Rucio software.

News

• OTG:0062990 - New data retention for MONIT metrics in HDFS - Tue Jun 01st, 10:00 - A new data retention policy for MONIT metrics stored in HDFS will be introduce. All metrics older than 13 months will start to be deleted by default starting from the 1st June 2021. If a different retention policy is required, this can be done by opening a SNOW ticket to the MONIT team. More details at the ASDF of 25th March 2021: https://indico.cern.ch/event/1003452/.

Incidents

On-going

• OTG:0062275 – "screen" security vulnerability that might require future action on interactive clusters – No updates. Recently a vulnerability in "screen" has been published. A patch has not yet been released by RedHat; however, there are no known remote code execution exploits in the wild for the moment. Given its prevalent usage on the interactive LXPLUS and AIADM clusters, however, its users should refrain from using "screen" where possible and consider "tmux" as an alternative as the Computer Security Team might decide to temporarily block "screen" once an exploit abusing this vulnerability becomes public.
• OTG:0062882 – 2FA with Yubikey to aiadm may fail – from Fri Mar 19th, 08:00 – No updates. Login to aiadm.cern.ch using a Yubikey as 2nd factor is apparently not working. Using TOTP with aiadm.cern.ch is working. Using TOTP or a Yubikey with aiadm8.cern.ch is working. Situation under investigation.

Completed Interventions

• OTG:0062816 – Latest OS updates on CMSONR database – Tue Mar 23rd, 10:30–11:30 – CERN IT will apply the latest OS patches to CMSONR database. The intervention is rolling, databases will be available, however some applications may experience session failures.
• OTG:0062814 – Latest OS updates on CMSR database – Mon Mar 29th, 09:30–11:30 – CERN IT will apply the latest OS patches to CMSR database. The intervention is rolling, databases will be available, however some applications may experience session failures.

Upcoming Service Changes

• OTG:0062832 – Removal of lxplus-cloud.cern.ch – Wed Mar 31st, 10:00 – The LxPlus sub service lxplus-cloud.cern.ch will be removed from service (lxplus-cloud will no longer exist). The recommended service to use for up to date compatible openstack clients is lxplus8.cern.ch. Cloud Client Documentation: https://clouddocs.web.cern.ch/clients/lxplus.html
• OTG:0062891 – Removing password expiration – Thu Apr 15th, 12:00 – The annual expiration of accounts' passwords will be removed. Password expiration as a security measure is no longer considered necessary after the introduction of checks on potentially compromised passwords in the new Single Sign On service (see OTG:0062834 for more details).

Reminders

• More than ever, it's recommended to make new requests to the VOC via the CMSVOC Jira project; and when writing to the VOC, please use cms-voc@cernNOSPAMPLEASE.ch, rather than writing to me individually. By adhering to those methods, the incoming and outgoing VOCs, as well as backup VOCs will always be in the loop, and your request will be answered more quickly.