Bamboo Analysis

Worklog

December 6th 2012: Suggested infrastructure

• For the server, a virtual node is enough as zero jobs will be executed there.

• The folder where all the artifacts are stored should be mapped to a high capacity and reliable system such as AFS.

• For the database, we can store it in the central service. In case of problems with the server, it could be easily reinstalled by only pointing to the central database.

• For the remote agents, the future Agile infrastructure is the most obvious target. They will be created in Openstack, based on minimal images with a little customization needed to correctly run the jobs coming from Bamboo. This nodes will be reverted to the checkpoint simply by adding a new stage on the job (that should be mandatory or converted to a plugin). From the user's point of view, he will have a static pool of agents always ready to run a job.

• A script to restore the checkpoints is needed. It can be based on the current script used in ETICS to control the pool, but simpler. It has only to run as frequent as possible, check the nodes that are powered off, restart the initial check point, and start them.

December 6th 2012: Investigating how to do remote deployment tests

The main problem to had clean nodes was how to poweroff the node after a job. With the pre-post plugin it works. It is necessary to add a stage, can be called poweroff, and create inside a task which post command in case of success and failed is "sudo poweroff". Of course bamboo user has to be added to the /etc/sudoers in this way:

bamboo ALL=(ALL) NOPASSWD: ALL

This will shout down the node at the end of the job.

A small script, that is a more basic copy of the ETICS pool control, has to be created to run every minute. This script will restore the initial checkpoint in the stopped nodes and start them. More logic can be added later to provide capacities to manage high platform demands, etc.

The artifacts are stored in the main server before the restarting of the agent, so it is not a problem that the agent is reverted to the clean checkpoint.

Steps to create an agent:

• Use the minimal image
• Add Bamboo user
• Add Bamboo user to /etc/sudoers
• Add Bamboo to Mock in /etc/group
• Install Bamboo software and mark it to run under Bamboo user
• Stop the node
• Create a checkpoint

December 5th 2012: Investigating how to do remote deployment tests

Disabled remote agent authentication on the Bamboo server to be able to start/stop nodes without having to manually approve them in the tool. In production the nodes have to be secured: https://confluence.atlassian.com/display/BAMBOO/Securing+your+remote+agents

This plugin (https://marketplace.atlassian.com/plugins/com.sysbliss.bamboo.plugins.prepost-build-command) will be review for allowing execute the poweroff at the end of the build

The Bamboo software in the nodes has to be running as Bamboo user. This user need to be part of Mock in /etc/group: mock:x:498:bamboo

December 4th 2012: Installation of remote agents and test jobs on them

If there is an error like: libwrapper: ../lib/libwrapper.so: wrong ELF class: ELFCLASS32 the last java version for that platform (ex: yum install java-1.7.0-openjdk.x86_64) has to be installed.

The same executables needed by the configuration and installed in the local server have been installed in order to run the jobs. More information about how to add remote agents capabilities here: https://confluence.atlassian.com/display/BAMBOO033/Editing+a+Capability

First success remote build: http://bamboo-test-srv/browse/LCGDM-DPMDSI-MOCKBUILD-46/log

December 3rd 2012: Installation of remote agents

Remote agents has been installed.

From the basic installation, these packages have to be installed: java, glibs.i686

The required bamboo-agent-home folder has been create in /opt/bamboo-agent-home

The steps for the installation can be found here: https://confluence.atlassian.com/display/BAMBOO/Bamboo+remote+agent+installation+guide

Jobs has to be launched to those nodes to verify that they are working

November 30th 2012: lcgdm-dsi job replication

The configuration has been successfully cloned manually. Due to the node is a SL6 and not SL5, few modification had to be done in the configuration to make it work

November 27th 2012: Installation of testing Bamboo server (attemp 3)

Creating a server (SL6) and two agents (SL5 and SL6 64bits) in CVI.

Machine names:

• SL6: bamboo-test-srv
• SL6: bamboo-test-a1
• SL5: bamboo-test-a2

Mid November - November 26th 2012: Installation of testing Bamboo server (attemp 2)

Machine created in Openstack by Andrew. Bamboo server installed and configured by Andres.

Checkpoints created with the empty machine just after the creation and after the installation and configuration of Bamboo.

Tried to import the data from tomtools to avoid replicate all the configurations (plans) manually. It failed and it was not possible to recreate the checkpoints previously created.

End October - Mid November 2012: Installation of testing Bamboo server (attemp 1)

Installation done by Andrew and Andres. Server configured using Puppet and Openstack. The server was down after some days and it was not possible to recover it.

Details: Server address: gt-bamboo1.cern.ch

For the moment there is not a web access, so ports need to be forwarded: ssh root@gt-bamboo1NOSPAMPLEASE.cern.ch -L 8085:localhost:8085

And then it can be accessed from the local computer. Example: http://127.0.0.1:8085/

Creation of remote agents

Created initially in CVI to check if the deployment tests can be done using Bamboo. To be discussed with Andrew for porting to Openstack.

Discussion with Alejandro about how emi.favorite is doing the deployments

Discussed with Alejandro. emi.favourite, the configuration which is doing the deployment tests in ETICS, is simply fetching the files needed and executing the tests. Run as Root privileges are needed.

The information is passed by parameters. The machines have to be clean ones.

It is necessary to copy the reports back.

Free plugin analysis

• XebiaLabs Deployit Plugin --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.xebialabs.deployit.plugin.bamboo-deployit-plugin

For doing deployment

• Bamboo Artifactory Plugin --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/org.jfrog.bamboo.bamboo-artifactory-plugin

The plugin lets you capture information about deployed artifacts, resolved dependencies and environment data associated with Bamboo build runs and have full traceability for your builds.

• Bamboo SCP/SSH Task --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugins.bamboo-scp-plugin

Allows secure copying of artifacts

• SameTime Notification Plugin --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/be.sofico.bamboo.plugins.bamboo-sametime-plugin

This plug-in integrates SameTime with Bamboo which makes it possible to notify the participants of a build about build failures, comments...

• Agent Smith Wallboard --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugin.agent-smith-wallboard

To quickly assess the agent and build queue health of your Bamboo instance

• Bamboo Goto Build bar --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugins.gotobar.gotobar

Improves navigation by adding shortcuts to quickly access top level plans and builds from anywhere in Bamboo

• Agent Utilities - Basic --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.pronetbeans.bamboo.agentutils-basic

To provide the ability to use Server Notifications When Bamboo Agents go offline, are disabled/enabled, or have capabilities changed.

• Atlassian Connector for Eclipse --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.atlassian.ide.plugins.eclipse.jira

To get Bamboo build information right within Eclipse

• Bamboo Build Log Exporter --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.pronetbeans.bamboo.buildlogexport

For allowing to export all the build result log files for an entire plan

• Bamboo Secure SSH Plugin --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/bamboo-ssh-plugin

Uses Java libraries to provide password support to execute shell commands on a remote host.

• Bamboo Project Graph Plugin --> Recommended: To be Analyzed

https://marketplace.atlassian.com/plugins/com.sysbliss.bamboo.plugins.bamboo-projectgraph-plugin

To generate a visual graph of the build order for a given project.

Adding remote nodes: Analysis

Questions to Atlassian

1. Reading Atlassian documents, the system with remote agents and elastic agents is working as it is described in the picture:

The bamboo server will send the job to the remote agent that match the requirements and this will forward the job to the elastic agent to be executed. The logs and artifacts produced are send back to the remote agent that sent the job to be stored before the elastic agent disappear. Is it correct? Otherwise, how is it working?

2. Are there the result copy back from EC2 to the remote agents or how are the log and artifact accessed?

3. Instead of Amazon EC2 we are planning to connect our OpenStack infrastructure for the elastic agents. Is there something already done for this system or should we do it? In case we do it, what is the collaboration/help we can expect from Atlassian? Amazon EC2 and OpenStack are not so different and both, CERN and Atlassian, could benefit from this collaboration: CERN will not spent a lot of time in the implementation and Atlassian and the community will obtain a new add-on developed with the same design as the existing Amazon EC2, being easy to maintain.

Atlassian answers

Let me try to answer your questions:

Question 1:

The bamboo server will send the job to the remote agent that match the requirements and this will forward the job to the elastic agent to be executed.

According to this document (https://confluence.atlassian.com/display/BAMBOO/About+Elastic+Bamboo), the flow diagram is a little different. The Bamboo server won't contact the Remote Agents to, then, contact the Elastic Agents. The Bamboo server will send the job directly to the elastic agent that matches the requirements. An elastic agent inherits its capabilities from the elastic image (https://confluence.atlassian.com/display/BAMBOO/elastic+image) that it was created from.

The logs and artifacts produced are send back to the remote agent that sent the job to be stored before the elastic agent disappear. Is it correct? Otherwise, how is it working?

Once a Job's build has completed running on an elastic agent, its results are made available (like those of any other Job build executed on a non-elastic agent). The elastic agent and instance will continue to run until they are shut down. Shutting down an elastic instance will terminate the agent, not take it offline. However, Bamboo will store historical information about the terminated elastic agent, such as the Job builds which it has run. Elastic agent activity is logged inside the elastic instance where the elastic agent runs. To access the elastic agent logs (atlassian-bamboo.log and bamboo-elastic-agent.out) use ssh to log in to your elastic instance as described in Viewing an elastic instance (https://confluence.atlassian.com/display/BAMBOO/Viewing+an+elastic+instance) and retrieve the logs. If you want to persist the log files after an elastic instance is terminated, you need to configure the Elastic Agent to record the log files in an Elastic Block Store (EBS) volume (https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+instances+to+use+the+EBS), which is a paid service (http://aws.amazon.com/pricing/ebs/). With regards to the artifacts, they're automatically transferred to the Bamboo server's BAMBOO_HOME/artifacts directory.

Question 2:

Are there the result copy back from EC2 to the remote agents or how are the log and artifact accessed?

I think this is answered above, but just to confirm:

The artifacts are automatically transferred from the Elastic Instance to the Bamboo server's BAMBOO_HOME/artifacts directory. The logs remain available in the elastic instance until the elastic instance is terminated. So you have two options: Use ssh to log in to your elastic instane as described in Viewing an elastic instance (https://confluence.atlassian.com/display/BAMBOO/Viewing+an+elastic+instance) and retrieve the logs before the elastic instance is terminated. Configure (https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+instances+to+use+the+EBS) the elastic instance to store the logs in the Amazon EBS volume.

Question 3:

Instead of Amazon EC2 we are planning to connect our OpenStack infrastructure for the elastic agents. Is there something already done for this system or should we do it?

I've discussed about this with our product managers, and we unfortunately don't support OpenStack. They've mentioned that it's currently impossible to make it work.

In case we do it, what is the collaboration/help we can expect from Atlassian?

If you want to develop a plugin or customization to make your Bamboo instance be compatible with OpenStack, we have an active community of developers in our ecosystem, who offer help via Answers (http://answers.atlassian.com/), or via paid services through our experts. We also have a community chat room for development questions. Details can be found here (http://blogs.atlassian.com/developer/2010/01/the_atlassian_community_irc_channel.html). We also welcome you to Get Involved in the Atlassian Developer Network (https://developer.atlassian.com/display/DOCS/Getting+Involved+in+the+Atlassian+Developer+Network). Please see Atlassian Support Offerings (https://confluence.atlassian.com/display/Support/Atlassian+Support+Offerings) for more information.

Please let me know if you need any further clarifications.

Kind regards, Felipe Kraemer Atlassian Support

Useful links

Openstack storage: http://blog.42.nl/articles/joss-tutorial-using-joss-to-access-openstack-storage

How to configure Amazon cloud for elastic agents: https://confluence.atlassian.com/display/BAMBOO/Configuring+Elastic+Bamboo

Agents monitoring: https://confluence.atlassian.com/display/BAMBOO/Monitoring+agent+status

Scaling up: http://www.atlassian.com/software/bamboo/overview/scaling-up

Configuring elastic agent capabilities: https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+agent+capabilities

Elastic Bamboo: https://confluence.atlassian.com/display/BAMBOO/About+Elastic+Bamboo

Working with elastic Bamboo: https://confluence.atlassian.com/display/BAMBOO/Working+with+Elastic+Bamboo

Disabling an elastic agent: https://confluence.atlassian.com/display/BAMBOO/Disabling+an+Elastic+Agent