Investigate ncm-yum and yum instead of ncm-spma and spma.

Background

Currently CDB configuration contains lists of rpms that exactly define the rpm names and versions to be installed on a box. This list is particularly hard to maintain and is getting harder and harder as:

• The rest of the world uses yum so multiple dependancies are cheap.
• There are many more packages these days as more software is expected and software is splitting into sub packages.

Advantages of yum over spma

• More standard to the rest of the world. Hardly anyone is not using yum basically. Even other conf engines such as puppet, chef use yum under the hood.
• It is of course what the SLC desktop users use and primary support case for linux support team.
• It's probably a lot quicker, will check some time. Certainly it's a lot quicker if resolutions are resolved by yum rather than failing in cdb commit-deploy cycle.
• This is more because of how we it use but since there is an intermediate swrep repository between SLC yum repos some packages for what ever reason are added swrep and so spma machines diverge from what SLC desktop machines are running. This understandably reduces synergy between the desktop world and quattor world especially in the area of support. In particular packages some packages are pinned to versions of ncm-components within CDB that are the not same as is present in SLC.
• Every one knows how to use it when they arrive at cern cc and we don't have to teach them local ways.
• yum has matured with countless features that were missing from it when spma was written. New features include:
  • yum downgrade, upgrade of packages including in the same transaction.
  • yum disto-sync, regardless of what I have installed fix my machine package versions to match the repository I am looking at.
  • yum security, give me a list of outstanding CVEs on my box.
  • yum versionlock. Pin a package version at version and stay there.
  • python api.
• A yum configuration in CDB would only contain the top level packages and group names of what is needed rather than every package. e.g on lxplus6 there are only 711 packages to define rather 2059 packages. There other 1300 odd packages are just dependencies of the 700. The 711 is reduced further since many explicit additions would be satisfied by the fact that a number of yum groups would provide these packages.

Disadvantages and mitigation of yum over spma

• Cost of migration - do for SL6 deployments , we have no public services that matter yet.
• Unknown problems and odd corner cases that have not been thought about.

YUM Repositories

Snap Shot Repositories.

A common and understandable fear with yum is that since it is so good at updating packages does this mean my box will be picking up random packages that have just been added to the repository. What happens if package is added mid lxbatch update .....

I would suggest to completely remove this problem by only ever pointing machines at snapshot repositories. Linux support already provide snapshots of OS and EPEL repositories at http://linuxsoft.cern.ch/internal . These snap shot repositories once created never ever change. In order to move a box forward the yum configuration must be pointed a new timestamp configuration. To this end the CDB template services/yum/slc6/dated-repos.tpl uses the existing OSDATESTAMP variable to point yum configuration to the snapshot repositories.

Of course configuration to point at the rolling SLC updates is perfectly possible also for non mission critical boxes where it makes sense to always just update. We get the best of both worlds.

Initial discussions with linux support suggest the acceptance of new repositories into the snapshot mechanism are possible, e.g IGTF, EMI, Quattor (if there was one), of course early days.

Everything has to be in YUM.

Of course in order to use yum to manage machines every candidate RPM must be in a yum repository. Running yum list extras on typical lxplus6 boxes shows around 200 packages that are pulled from swrep that are not the SLC offering, mostly these are CERN-CC packages and extra or different version ncm-components. Clearly at the moment SL6 swrep is not completely full of junk to actual requirements. Any suggestions of just using the swrep repository as a yum repository are not possible. Within it packages both override and underride SLC.... Also it is enormous since it contains the now redundant OS.

Currently for testing purposes the 200 packages are in my home directory http://cern.ch/straylen/spma2yum-fudges. Any real solution would require a yum repository for ELFMS, CERN-CC packages as well as any other software packages that come from random places.. Some servies would even no doubt there own packages. ...

ncm-yum

ncm-yum (v1.0.1-1) was written by Jan Ivan around 4 years ago and is the vast majority quattor code required. It's possible to:

• Define YUM repositories to be configured in /etc/yum.repos.d/
• Define packages and groups to be installed which fall into 3 cases:

  Define a group for installation

  At ncm-yum time the group name is added to ncm maintained list of groups /var/ncm/ncm-yum-groupinstall.list. The group is subsequently installed by ncm-yum calling yum on that group. Any groups being removed from the list of groups are also removed.

  Define a package for installation

  At ncm-yum time the package name is added to an ncm maintained list of package /var/ncm/ncm-yum-yuminstall.list the package is then installed using yum install pkg. Any packages removed within an ncm-yum run are removed.

  Define a package for installation with exact version number and/or exact arch

  Such a package name is added to the list of packages in /var/ncm/ncm-yum-yuminstall.list but is also added to yum's versionlock file. The package is installed at this version as long as it is not installed at a newer version. This last point obviously wrong since a versionlocking a package at an older version should work, see below for improvements to ncm-yum to be made in particular the use of distro-sync.

Note in the versionlock case all other operations below respect the versionlock. A versionlocked package essentially causes any other version in the yum repository to be completely ignored by the yum client allways.

ncm-yum bugs, improvements and new features

ncm-yum has few bugs and missing features to be fixed or improved upon. Mostly these are because of new features available in yum which make thing easier to do and ncm-yum has to do less work to try and do itself.

New feature - use a single transaction for multiple yum calls.

• ncm-yum will currently under certain circumstances of reconfiguration run the following separate shell commands commands:

        # yum remove xulrunner-1.2.3
        # yum install xulrunner-1.2.4
        # yum groupremove graphics
        # yum groupinstall science
        # yum groupupdate base
        

  in particular the first two are very bad since remove and install is very different to an upgrade or downgrade. Instead ncm-yum can be adapted to run:

       yum shell <<EOF
       remove xulrunner-1.2.3
       install  xulrunner-1.2.4
       install @graphics
       remove @science
       install @base
       run
       EOF
       

  which will then be resolved within a single transaction, should be easy. Just as spma would of course

Bug - schema bug.

The schema for ncm-yum contains *"arch" ? string with match (SELF, 'i[3456]86\|x86_64\|ia64\|noarch') which is good in principal but fails if you actually set an arch. Need to check schema syntax , should be trivial.

New feature, Add support for repo priorities

Add support for priority=N for each repository. yum-priorities are a lot richer than yum-protectbase, is trivial.

New feature, Add an exclude pkg option.

It's very common to say that I want the @base group but I don't want a particular package from it and I never ever want to install that package. yum supports exclude= concept for this. We just need to expose it in ncm-yum. Easy.

Update for the current perl-LC , there are various perl-LC warnings to be cleaned up.

Use distro-sync rather than install.

The install method only updates or installs. The distro-sync method will update, install or downgrade all in one method.

yum-exact or removing packages that should not be there.

The spma command is very good at is removing packages that have been installed outside of spma and so brings the box back to some known state. We require the same functionality with a yum based system. Our inputs to define what should be installed on a system are the ncm maintained files ncm-yum-yuminstall and ncm-yum-groupinstall.

A first yum script yum-exact.py now exists that does this. Here is an example of installing a random package znc that pulls in a dependency c-ares. (The output has been edited for clarity)

# yum  -y install znc
Installed:
  znc.x86_64 0:0.098-2.el6
Dependency Installed:
  c-ares.x86_64 0:1.7.0-5.el6
Complete!

and then cleaning that back up with yum-exact.py

# ./yum-exact.py 
Erasing: znc 100/100 [1/1]
Erasing: c-ares 100/100 [1/1]

The current logic of yum-exact is:

1. Calculate a list of installed leaf packages that are installed. A leaf is an package which has no other packages depending upon it.
2. Calculate a list of packages to be preserved from ncm-yum-yuminstall and the expansion of the groups in ncm-yum-groupinstall to packages.
3. Remove any installed package leaves that are not explicitly list of packages to preserve.
4. Repeat this process till there are no leaves left to remove.

This is purposely different to the other way of doing this which would be to expand desired groups and packages , calculate the dependencies and then fix to that allways. I think the way I have done it is better but I am open to comments on this one.... Certainly this way is a lot quicker. A few seconds rather than probably a minute or two.

Note that packages installed on the box which are not present in the yum repository should also be removed.

Fixing My Package versions to latest or desired version and how do I roll back my whole system to previous snapshot.

The yum command yum distro-sync is marvelous here. It changes all the packages on your machine where they are present in the yum repository to move to the latest package in that repository, if a versionlock is defined on a package this will be respected.

It will happily upgrade or downgrade any such packages as required. So with reference to the snapshot repositories above it is perfectly possible to switch yum configuration to an older timestamp and then downgrade to that timestamp - how cool is that

Of course it should be noted that on an interactive case boxes can all ways be rolled back with yum history magic but status is of course transient till the next yum update (or distro-sync)

Open Ends

Things I have yet to think about or experiment with yet:

• Multiarch issues - it would be good to be able to specify arch as, default, exact or all. By default I mean yum default which is often both.
• What should be the actuall run of ncm-yum, Some subset of configure yum, fix up changes in package list, yum distro-sync, yum-exact.
• What should be run routinely as cron, and if so subsequent lemon alarm for that. The marvelous spma_wrong but some sub casing of that should be possible:
  • You have extra packages installed to what you asked for.
  • distro-sync is going to fail unless you do something next time it is run.
• Installations, obviously we use yum through out the installation, I don't expect any particular difficulties here.

Current Status

lxdev62 is to all intents and purposes a box managed by yum under quattor. The operations above that are complete apparently work. Relevant templates for inspection are cluster/lxplus/yumtime.tpl services/yum/*.