Migrating Jenkins on AWS using Terraform

This is a rundown of how I recently moved one of our Jenkins instances that was sitting on a repurposed AWS EC2 instance doing multiple jobs and serving many other purposes. In order to ensure we had a dedicated server for our build, test and deploy pipeline, I decided to move this onto its’ own server using Terraform.

If you already have your Jenkins config and want to fast forward to infra provisioning you can find all the code here on Github. You only need to update the bucket name in both the iam.tf file and the s3.tf file and you are good to go.

Pre-Requisites

You will need the following to run with this guide:

• AWS Account
• Terraform Installed
• SSH access to the server that you are migrating from
• Terminal Access (either iTerm or within VS Code, or other chosen method)

Retrieve Jenkins Config

First off we need to get your current Jenkins configuration from the old EC2 server. So SSH onto the server with either the ec2 user or your own user.

ssh {username}@server-ip

Once on you will need to navigate to the Jenkins Home on the server. This can usually be found in /var/lib/jenkins, however, if it is not here you can find it in the following location through the Jenkins UI (Jenkins > Manage Jenkins > System Information)

After you have verified the location of the Jenkins directory then navigate to the /var/lib/, and run the following command:

tar -cvzf jenkins-migrate.tgz jenkins/

This command will in order of cvzf (compress, verbosely, zip and name the file) allow us to get the files we need from the server easier. If you list the items in the directory you will now see an item named jenkins-migrate.tgz. At this point you can exit from the server onto your local machine.

To get the file from the server onto your local machine you are going to first create a new folder to store your config, so a simple mkdir Proj_jenk_migr8should do the trick or a name of your own choosing. If you now go into that folder (cd Proj_jenkins_migr8) you can now run scp in order to get the folder onto your local machine:

scp -r {username}@server-ip:/var/lib/ jenkins-migrate.tgz .

This will then secure copy your tar file to your current folder. Once it is finished copying from the server you can then untar it in the folder it is currently in:

tar -xvzf jenkins-migrate.tgz jenkins/

The above command, you will notice is quite similar to compressing the folder but now we are extracting the contents.

Don’t forget your keys!

You will need to ssh onto the server again and confirm the jenkins user home which will probably look something like this — cd /home/jenkins. Once confirmed you can then create a new directory to hold the jenkins .ssh folder with something like mkdir jenkins-ssh and and run the following command:

sudo cp -R /home/jenkins/.ssh /jenkins-ssh

You will then need to change the permissions in order to be able to copy it onto your local machine with:

sudo chown -R {your-username}:{your-username} jenkins-ssh

You can now log off from the box and on your local machine copy the folder to be used later. Run this command:

scp -r {username}@server-ip:/home/{username}/jenkins-ssh .

You will add this key to the s3 bucket and then afterwards move it to the EC2 server. At this point you can log back onto the box and delete the new folder (jenkins-ssh) you created in your home directory.

Options

The next step is to provision the AWS infrastructure and install Jenkins. This can be achieved in multiple ways:

• Create and provision all infrastructure and then install Jenkins
• Spin up infra with Terraform and then install Jenkins manually
• Add user data to install Jenkins via User Data specified in your Terraform file
• Call Ansible via Terraform to install Jenkins on the EC2 instance (Overkill)

We are going to use the third one because nobody wants to spend a lot of time manually constructing the resources necessary or installing Jenkins. The use of Ansible for a one-time set-up of a Jenkins pipeline is also a bit excessive.

Terraform

For this we will need to create (or find) a Terraform file. In the project (Proj_jenkins_migr8) directory create a folder named terraform. Add the following files:

• main.tf
• variables.tf
• compute.tf
• network.tf
• terraform.tfvars
• install_jenkins.sh
• iam.tf
• s3.tf

Below will see the contents of each file and a brief description:

main.tf — This sets your provider as AWS and sets up your SSH keypair

variables.tf — This contains your variables to be used throughout. You can change the default region here.

compute.tf — This sets up your EC2 instance that will host Jenkins and run your install Jenkins script via User Data. Please Note* You may need to update your instance type depending on the size of your jenkins data.

network.tf — This will set up your VPC, subnets, route tables and Internet Gateway for access to your Jenkins instance

terraform.tfvars — You can set region, VPC IP, Subnet CIDR blocks, instance type and keyname here.

install_jenkins.sh — Install Jenkins via bash script

iam.tf — This will give you access to the S3 bucket to upload the Jenkins config from

s3.tf — You will use this to upload your Jenkins Configuration

Once you have the files in place you will need to update the bucket name in both S3.tf and iam.tf (put-your-bucket-name-here).

Initialise Terraform by running the following terraform init

Deployment

To deploy the above into Production you will first need to sign into AWS and create a new Terraform User if you haven’t already done so with permissions to the resources you need to spin up. Once you have created a new User, then you will create a new Access Key under the Security Credentials tab. From here you will get an access key and secret access key:

In your terminal if you type aws configure then you will be able to set your keys along with default region (eu-west-2) and default output (json). This of course isn’t the only way to sign in (AWS Configure).

You should now be able to run the following in terminal:

• terraform plan
• terraform apply -auto-approve

When you run plan you will see the resources that will be spun up. When you run apply the resources will then be created within AWS.

You will get an output within the terminal window which is the EC2 IP address to access Jenkins. Copy this and paste it into your browser then append it with :8080 (port) and you should be able to see a Jenkins window with Unlock Jenkins.

Jenkins Migration

At this point you will open up the AWS console and navigate to your newly created bucket in S3 (put-your-bucket-name-here). Now you are going to upload your jenkins configuration tarball (jenkins-migrate.tgz) into your S3 bucket by clicking Add files and then select your compressed jenkins config folder. It may take a while depending on how many plugins you had that you didn’t use :)

You can also add your jenkins key pair folder jenkins-ssh to the S3 bucket using the same method.

Once the folder has been uploaded you can navigate to EC2 on the console and click the Connect button. This will take you to another page with 3 tabs, on the one labelled EC2 Instance Connect. Press the Connect button again and a terminal window will be opened.

From here you can run the following commands

• aws s3 ls s3://put-your-bucket-name-here — To list the items in your bucket
• aws s3 cp s3://put-your-bucket-name-here/jenkins-migrate.tgz jenkins-migrate.tgz . — To copy the compressed file to your EC2 instance.

Some Linux commands…. …and hey presto!

Once it is on your EC2 instance you can untar it with the following command:

tar -xvzf jenkins-migrate.tgz

With the folder contents unzipped you can now run the following commands:

Stop Jenkins — sudo service jenkins stop

Copy current configuration to the /tmp folder (just in case) — sudo cp -R /var/lib/jenkins /tmp

Remove the current Jenkins folder — sudo rm -R /var/lib/jenkins

Copy the migrated folder across — sudo mv jenkins /var/lib

Update permissions from ec2 user to jenkins — sudo chown -R jenkins:jenkins /var/lib/jenkins

Start Jenkins — sudo service jenkins start

Some more Linux Commands… …don’t forget your keys!

In order to have as little disruption as possible to your migration we can now add the ssh keys into the Jenkins User on the new EC2 instance. This will allow your new server to talk to your application servers. Whilst still on the EC2 SSH Connect terminal switch to the Jenkins user with the following:

sudo su -s /bin/bash jenkins

cd — Drop to the Jenkins root directory

mkdir .ssh — Create directory to copy keys to

aws s3 cp s3://put-your-bucket-name-here/jenkins-ssh/ .ssh --recursive — Copy keys from S3

Change permissions on the key:

sudo chown -R jenkins:jenkins /var/lib/jenkins/.ssh

At this point you want to make sure your .ssh folder and its’ content have the correct permissions, .ssh folder — 700, public key — 644, private key and authorized_keys — 600. There are a plethora of articles on the inter-web that go through this all pertaining to the use of chmod. Check them out (google/stackoverflow) if you are not sure.

Remove unnecessary folder:

rm -r /var/lib/jenkins/jenkins-ssh

Clean-Up (Optional)

If you want to tear down the infrastructure and were using this purely as a skill up/refresh exercise you can pull it down with one command:

terraform destroy

NOTE: You will need to empty the contents of your S3 bucket in order to enable terraform to remove the S3 bucket. If you plan on bringing this infrastructure up and down multiple times you can leave the S3 bucket where it is and it will connect each time. Just remember to remove it when you are finished.

Wrap up

Now if you navigate to the Output IP from your terraform Output with port 8080 you should now be able to log into your old Jenkins Instance only on new and improved Infrastructure.

From here you want to test that your build still works so you may need to update the configuration settings in Jenkins to match various items such as the new server IP and various folder pathways that may need to be reconfigured.

This is also a great opportunity to upgrade both Jenkins and your plugins which you may as well do if you have came this far. Remember to keep your old set-up running for a while until you are at least 99% sure that your new Jenkins instance works as desired — ie, you have successfully promoted code to your upper environments (Dev, Test, Production).

Next you would want to be able to access your Jenkins interface securely from a browser with DNS and SSL. We will visit this in another post.

Any updates, improvements or feedback is very much welcomed and appreciated. Thanks for reading!