Installing bare metal Concourse CI for Windows .Net deployments
The DevOps mindset is at the core of how we maintain a robust infrastructure here at Judopay, and we’re always on the lookout for anything that can improve our existing toolset.
With that in mind, we have recently looked at the Concourse CI system as a possible continuous integration/build server. The product looks very strong, but documentation for installing is sparse as it’s still in its infancy, so we thought we’d go through some detail and gotchas here for anyone else thinking of giving it a try.
Concourse brings the idea of infrastructure as code to Continuous integration. The benefits of versionability and reliability that this brings to Infrastructure can apply just as well to the CI/Build process, with the added bonus that the specifics of a build for a particular version of our code can be stored in the same repository as the software itself.
This appealed to us a lot at Judopay as it solves the problem of how to deploy older versions of code, should we ever wish to, even after the build pipeline has long since changed. When old code is deployed, we can simply switch seamlessly back to the old build pipeline for that exact code version.
Another benefit is that Concourse is free. As Judopay is growing rapidly, a solution that can be scaled without rising licensing costs was a big draw for us.
A word of warning
Before going into the specifics of bare metal install, a word of warning: Concourse is not for the faint-hearted.
The documentation on the Concourse website is pretty lightweight, and assumes a certain amount of knowledge upfront. The community of people using the product is small in comparison to a big player like Jenkins, and Concourse isn’t simply a Teamcity/Jenkins-a-like so the concepts will require a fair bit of learning before you can put them into practice. If you are looking for free CI, rather than specifically CI as code, I would advise looking at Jenkins first.
That all being said, Concourse looks like a very flexible and powerful CI system, so if you’re still sure you want to go ahead, here are some things we’ve learned at Judo about installing on bare metal.
Bare metal install
When getting started with Concourse, there is the option to download a Vagrant or Docker instance that will allow you to experiment right away, but once you’re ready for production you’re going to want to try and install onto a server, to allow full access to all the options.
A very simplified description of the Concourse architecture (a full description of which is here) is that a central server is used to coordinate tasks on workers. The server coordinates the workers and has a web front end that is used to monitor the progress of builds only and cannot be used for configuration. When first installed, it should look something like this:
Reading the Concourse documentation, you might be forgiven for thinking that Concourse can be installed on either Linux, Windows or Mac, but, in our experience, only the workers can be run on all three platforms.
In fact, the only platform on which the Concourse server reliably installs and functions correctly is Ubuntu. It is possible to complete an install on either Windows or Centos of the Concourse server, but our experience was that it did not function afterwards on any of these systems.
After spending some time on forums, I found other people who shared this experience, so while it may be theoretically possible to run on anything, we would strongly advise the use of Ubuntu Linux at the time of writing this post.
Thankfully, the process of getting a worker running on any system is very straightforward, so once you have a central Ubuntu server set up, you can spin up workers on any other system to handle your builds.
Steps to install
The install can be split into four basic sections:
- Create an Ubuntu server
- Install PostgreSQL and create users and databases
- Create SSL keys
- Spin up Concourse, specifying all three things above correctly.
Create an Ubuntu server
There is no need to cover how to install Ubuntu in this document, as it is well covered here. We used Ubuntu version 16.04 and followed the instructions for a standard install with no extras.
Install PostgreSQL and create users and databases
This document will assume you are logged in as root unless otherwise specified, so does not use the sudo command.
apt-get install postgresql postgresql-contrib
Now that postgres is installed, some configuration changes need to be made to allow it to work with Concourse.
The Concourse server requires:
– A database called ATC
– A user (which will be called “Concourse” in this post)
– A database with the same name as the user (required by the postgres authentication system)
These steps can be done as follows:
Switch to Postgres user (created automatically during postgres install)
Log into postgresql as Postgres default superuser
Run the following commands to create databases and user
create database atc;
create database concourse;
create user concourse password ‘password’;
Log out of Postgres back to root
Now that the blank ATC database and user required by Concourse have been created, certain settings need to be changed as follows:
Find the postgres data directory
Find -name pg_hba.conf
Browse to postgres data directory and edit pg_hba.conf
vi [path to Postgresql data directory]/pg_hba.conf
Add the following lines
host concourse concourse 127.0.0.1/32 md5
host atc concourse 127.0.0.1/32 md5
Enable SSL in postgresql.conf
vi [path to Postgresql data directory]/postgresql.conf
ssl = true
Note: As the default is ssl = off, you might expect ssl = on to the be the positive, but the above ssl = true is actually correct.
Install openssl and create certificates
apt-get install openssl
cd [path to postgresql data directory]
openssl req -nodes -new -x509 -keyout server.key -out server.crt -nodes
Note: It isn’t compulsory to create the certificates in the data directory of postgresql, but if you move them to a different location you must update that location in the postgresql.conf file.
Note: Check the Concourse download page for the appropriate most recent version link
In the same folder as the location of the Concourse binary (above we just use home, but you might prefer to make a better place), certain certificates need to be created to allow workers to communicate with the Concourse server.
These certificates get specified at the time of starting a server or worker, which will be covered a little later in this blog.
Create server and worker keys
ssh-keygen -t rsa -f host_key -N ”
ssh-keygen -t rsa -f worker_key -N ”
ssh-keygen -t rsa -f session_signing_key -N ”
cp worker_key.pub authorized_worker_keys
Start the Concourse server
Now that the database and keys have been created, all that remains is to start the server itself. As the file is executable it needs to be given permissions to run if you haven’t done so already. It also needs to be told where the database is, and which user will be accessing it.
Give the file permissions to run
Chmod 700 concourse
(Assuming that you are in the correct directory and have downloaded the binary called Concourse)
Run the Concourse server (fill in [ip address of server] as appropriate)
concourse –basic-auth-username concourse –basic-auth-password password –session-signing-key session_signing_key –tsa-host-key host_key –tsa-authorized-keys worker_key.pub –external-url http://[ip address of server] –postgres-data-source postgres://concourse:password@localhost/atc
If this works, you will see a series of logs as the server populates the database and gets going. If this doesn’t work, study the errors it shows and see if you have any incorrect information in that command.
Common mistakes are to get the names of keys wrong, or the username or password for the postgres user isn’t correct (in this case we use “concourse”, “password” specified early).
One way to troubleshoot is to try actually logging into the postgres server with the user specified above. This will look something like:
Su postgres #because you must be postgres not for logins to work as below
Psql -U concourse –password
Once you have successfully run the Concourse server, you might find you want to create a service for this, rather than running it straight from the console – it can make starting, stopping and checking its status easier.
Note: The output never ends while the server is running in the console. You will not be presented with a command prompt again at that login until you stop the server. Until then, it will output logs from the server onto the screen.
Browse to your server
Once the command is successful and the server has finished configuring (it takes a minute or two), browse to the server at the specified IP, and you should see the Concourse login screen as above. Once you are at this point, it is time to spin up a worker.
To build Microsoft .Net sites will require a Windows worker on which to run msbuild.
Thankfully, the process of spinning up a worker is extremely simple. All that is required is to download the Windows binary from the download page listed above, copy in the appropriate keys as created in the install, and then run the following command (remembering to fill in the IP of the server):
.\concourse_windows_amd64.exe worker /work-dir .\work /tsa-host [ip of server] /tsa-public-key .\host_key.pub /tsa-worker-private-key .\worker_key
Note: Don’t worry about creating work-dir in the location you choose to run this, it will create during the starting of the worker automatically.
As with the server, this will create a log output to the command window. If you have any issues starting the worker, check the log output and it will give you clues as to any possible spelling mistakes or missing files you may have.
As long as you have the Concourse worker, the keys and the ability to access the server on ports 8080 and 2222, it should work fine.
That’s it! I hope this helped you getting started with a bare metal install. If you are just thinking of having a play with Concourse and don’t want to go to the effort of the steps above, it’s definitely worth checking out the Vagrant box install.
About Judopay · Judopay simplifies in-app payments, enable frictionless checkouts and intelligently prevents fraud for leading companies globally. Our payments and mobile experts help guide businesses and their development partners to create best in class apps to make paying faster, easier and more secure. Founded by serial financial technology entrepreneurs in 2012, Judopay is backed by leading venture investors and supported by banking and card scheme partners to offer in-app payments that are simple, frictionless and protected.