My blog

Using Packer and Terraform to Setup Jenkins Master-Slave Architecture

Written by

admin

in

, , , , , ,

Automation is everywhere and it is better to adopt it as soon as possible. Today, in this blog post, we are going to discuss creating the infrastructure. For this, we will be using AWS for hosting our deployment pipeline. Packer will be used to create AMI’s and Terraform will be used for creating the master/slaves. We will be discussing different ways of connecting the slaves and will also run a sample application with the pipeline.

Please remember the intent of the blog is to accumulate all the different components together, this means some of the code which should be available in development code repo is also included here. Now that we have highlighted the required tools, 10000 ft view and intent of the blog. Let’s begin.

Using Packer to Create AMI’s for Jenkins Master and Linux Slave

Hashicorp has bestowed with some of the most amazing tools for simplifying our life. Packer is one of them. Packer can be used to create custom AMI from already available AMI’s. We just need to create a JSON file and pass installation script as part of creation and it will take care of developing the AMI for us. Install packer depending upon your requirement from Packer downloads page. For simplicity purpose, we will be using Linux machine for creating Jenkins Master and Linux Slave. JSON file for both of them will be same but can be separated if needed.

Note: user-data passed from terraform will be different which will eventually differentiate their usage.

We are using Amazon Linux 2 – JSON file for the same.

{
  "builders": [
  {
    "ami_description": "{{user `ami-description`}}",
    "ami_name": "{{user `ami-name`}}",
    "ami_regions": [
      "us-east-1"
    ],
    "ami_users": [
      "XXXXXXXXXX"
    ],
    "ena_support": "true",
    "instance_type": "t2.medium",
    "region": "us-east-1",
    "source_ami_filter": {
      "filters": {
        "name": "amzn2-ami-hvm-2.0*x86_64*",
        "root-device-type": "ebs",
        "virtualization-type": "hvm"
      },
      "most_recent": true,
      "owners": [
        "amazon"
      ]
    },
    "sriov_support": "true",
    "ssh_username": "ec2-user",
    "tags": {
      "Name": "{{user `ami-name`}}"
    },
    "type": "amazon-ebs"
  }
],
"post-processors": [
  {
    "inline": [
      "echo AMI Name {{user `ami-name`}}",
      "date",
      "exit 0"
    ],
    "type": "shell-local"
  }
],
"provisioners": [
  {
    "script": "install_amazon.bash",
    "type": "shell"
  }
],
  "variables": {
    "ami-description": "Amazon Linux for Jenkins Master and Slave ({{isotime \"2006-01-02-15-04-05\"}})",
    "ami-name": "amazon-linux-for-jenkins-{{isotime \"2006-01-02-15-04-05\"}}",
    "aws_access_key": "",
    "aws_secret_key": ""
  }
}

As you can see the file is pretty simple. The only thing of interest here is the install_amazon.bash script. In this blog post, we will deploy a Node-based application which is running inside a docker container. Content of the bash file is as follows:

#!/bin/bash

set -x

# For Node
curl -sL https://rpm.nodesource.com/setup_10.x | sudo -E bash -

# For xmlstarlet
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

sudo yum update -y

sleep 10

# Setting up Docker
sudo yum install -y docker
sudo usermod -a -G docker ec2-user

# Just to be safe removing previously available java if present
sudo yum remove -y java

sudo yum install -y python2-pip jq unzip vim tree biosdevname nc mariadb bind-utils at screen tmux xmlstarlet git java-1.8.0-openjdk nc gcc-c++ make nodejs

sudo -H pip install awscli bcrypt
sudo -H pip install --upgrade awscli
sudo -H pip install --upgrade aws-ec2-assign-elastic-ip

sudo npm install -g @angular/cli

sudo systemctl enable docker
sudo systemctl enable atd

sudo yum clean all
sudo rm -rf /var/cache/yum/
exit 0
@velotiotech

Now there are a lot of things mentioned let’s check them out. As mentioned earlier we will be discussing different ways of connecting to a slave and for one of them, we need xmlstarlet. Rest of the things are packages that we might need in one way or the other.

Update ami_users with actual user value. This can be found on AWS console Under Support and inside of it Support Center.

Validate what we have written is right or not by running packer validate amazon.json.

Once confirmed, build the packer image by running packer build amazon.json.

After completion check your AWS console and you will find a new AMI created in “My AMI’s”.

It’s now time to start using terraform for creating the machines. 

Prerequisite:

1. Please make sure you create a provider.tf file.

provider "aws" {
  region                  = "us-east-1"
  shared_credentials_file = "~/.aws/credentials"
  profile                 = "dev"
}

The ‘credentials file’ will contain aws_access_key_id and aws_secret_access_key.

2.  Keep SSH keys handy for server/slave machines. Here is a nice article highlighting how to create it or else create them before hand on aws console and reference it in the code.

3. VPC:

# lookup for the "default" VPC
data "aws_vpc" "default_vpc" {
  default = true
}

# subnet list in the "default" VPC
# The "default" VPC has all "public subnets"
data "aws_subnet_ids" "default_public" {
  vpc_id = "${data.aws_vpc.default_vpc.id}"
}

Creating Terraform Script for Spinning up Jenkins Master

Creating Terraform Script for Spinning up Jenkins Master. Get terraform from terraform download page.

We will need to set up the Security Group before setting up the instance.

# Security Group:
resource "aws_security_group" "jenkins_server" {
  name        = "jenkins_server"
  description = "Jenkins Server: created by Terraform for [dev]"

  # legacy name of VPC ID
  vpc_id = "${data.aws_vpc.default_vpc.id}"

  tags {
    Name = "jenkins_server"
    env  = "dev"
  }
}

###############################################################################
# ALL INBOUND
###############################################################################

# ssh
resource "aws_security_group_rule" "jenkins_server_from_source_ingress_ssh" {
  type              = "ingress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["<Your Public IP>/32", "172.0.0.0/8"]
  description       = "ssh to jenkins_server"
}

# web
resource "aws_security_group_rule" "jenkins_server_from_source_ingress_webui" {
  type              = "ingress"
  from_port         = 8080
  to_port           = 8080
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "jenkins server web"
}

# JNLP
resource "aws_security_group_rule" "jenkins_server_from_source_ingress_jnlp" {
  type              = "ingress"
  from_port         = 33453
  to_port           = 33453
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["172.31.0.0/16"]
  description       = "jenkins server JNLP Connection"
}

###############################################################################
# ALL OUTBOUND
###############################################################################

resource "aws_security_group_rule" "jenkins_server_to_other_machines_ssh" {
  type              = "egress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins servers to ssh to other machines"
}

resource "aws_security_group_rule" "jenkins_server_outbound_all_80" {
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins servers for outbound yum"
}

resource "aws_security_group_rule" "jenkins_server_outbound_all_443" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = "${aws_security_group.jenkins_server.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins servers for outbound yum"
}

Now that we have a custom AMI and security groups for ourselves let’s use them to create a terraform instance.

# AMI lookup for this Jenkins Server
data "aws_ami" "jenkins_server" {
  most_recent      = true
  owners           = ["self"]

  filter {
    name   = "name"
    values = ["amazon-linux-for-jenkins*"]
  }
}

resource "aws_key_pair" "jenkins_server" {
  key_name   = "jenkins_server"
  public_key = "${file("jenkins_server.pub")}"
}

# lookup the security group of the Jenkins Server
data "aws_security_group" "jenkins_server" {
  filter {
    name   = "group-name"
    values = ["jenkins_server"]
  }
}

# userdata for the Jenkins server ...
data "template_file" "jenkins_server" {
  template = "${file("scripts/jenkins_server.sh")}"

  vars {
    env = "dev"
    jenkins_admin_password = "mysupersecretpassword"
  }
}

# the Jenkins server itself
resource "aws_instance" "jenkins_server" {
  ami                    		= "${data.aws_ami.jenkins_server.image_id}"
  instance_type          		= "t3.medium"
  key_name               		= "${aws_key_pair.jenkins_server.key_name}"
  subnet_id              		= "${data.aws_subnet_ids.default_public.ids[0]}"
  vpc_security_group_ids 		= ["${data.aws_security_group.jenkins_server.id}"]
  iam_instance_profile   		= "dev_jenkins_server"
  user_data              		= "${data.template_file.jenkins_server.rendered}"

  tags {
    "Name" = "jenkins_server"
  }

  root_block_device {
    delete_on_termination = true
  }
}

output "jenkins_server_ami_name" {
    value = "${data.aws_ami.jenkins_server.name}"
}

output "jenkins_server_ami_id" {
    value = "${data.aws_ami.jenkins_server.id}"
}

output "jenkins_server_public_ip" {
  value = "${aws_instance.jenkins_server.public_ip}"
}

output "jenkins_server_private_ip" {
  value = "${aws_instance.jenkins_server.private_ip}"
}

As mentioned before, we will be discussing multiple ways in which we can connect the slaves to Jenkins master. But it is already known that every time a new Jenkins comes up, it generates a unique password. Now there are two ways to deal with this, one is to wait for Jenkins to spin up and retrieve that password or just directly edit the admin password while creating Jenkins master. Here we will be discussing how to change the password when configuring Jenkins. (If you need the script to retrieve Jenkins password as soon as it gets created than comment and I will share that with you as well).

Below is the user data to install Jenkins master, configure its password and install required packages.

#!/bin/bash

set -x

function wait_for_jenkins()
{
  while (( 1 )); do
      echo "waiting for Jenkins to launch on port [8080] ..."
      
      nc -zv 127.0.0.1 8080
      if (( $? == 0 )); then
          break
      fi

      sleep 10
  done

  echo "Jenkins launched"
}

function updating_jenkins_master_password ()
{
  cat > /tmp/jenkinsHash.py <<EOF
import bcrypt
import sys
if not sys.argv[1]:
  sys.exit(10)
plaintext_pwd=sys.argv[1]
encrypted_pwd=bcrypt.hashpw(sys.argv[1], bcrypt.gensalt(rounds=10, prefix=b"2a"))
isCorrect=bcrypt.checkpw(plaintext_pwd, encrypted_pwd)
if not isCorrect:
  sys.exit(20);
print "{}".format(encrypted_pwd)
EOF

  chmod +x /tmp/jenkinsHash.py
  
  # Wait till /var/lib/jenkins/users/admin* folder gets created
  sleep 10

  cd /var/lib/jenkins/users/admin*
  pwd
  while (( 1 )); do
      echo "Waiting for Jenkins to generate admin user's config file ..."

      if [[ -f "./config.xml" ]]; then
          break
      fi

      sleep 10
  done

  echo "Admin config file created"

  admin_password=$(python /tmp/jenkinsHash.py ${jenkins_admin_password} 2>&1)
  
  # Please do not remove alter quote as it keeps the hash syntax intact or else while substitution, $<character> will be replaced by null
  xmlstarlet -q ed --inplace -u "/user/properties/hudson.security.HudsonPrivateSecurityRealm_-Details/passwordHash" -v '#jbcrypt:'"$admin_password" config.xml

  # Restart
  systemctl restart jenkins
  sleep 10
}

function install_packages ()
{

  wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
  rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
  yum install -y jenkins

  # firewall
  #firewall-cmd --permanent --new-service=jenkins
  #firewall-cmd --permanent --service=jenkins --set-short="Jenkins Service Ports"
  #firewall-cmd --permanent --service=jenkins --set-description="Jenkins Service firewalld port exceptions"
  #firewall-cmd --permanent --service=jenkins --add-port=8080/tcp
  #firewall-cmd --permanent --add-service=jenkins
  #firewall-cmd --zone=public --add-service=http --permanent
  #firewall-cmd --reload
  systemctl enable jenkins
  systemctl restart jenkins
  sleep 10
}

function configure_jenkins_server ()
{
  # Jenkins cli
  echo "installing the Jenkins cli ..."
  cp /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar /var/lib/jenkins/jenkins-cli.jar

  # Getting initial password
  # PASSWORD=$(cat /var/lib/jenkins/secrets/initialAdminPassword)
  PASSWORD="${jenkins_admin_password}"
  sleep 10

  jenkins_dir="/var/lib/jenkins"
  plugins_dir="$jenkins_dir/plugins"

  cd $jenkins_dir

  # Open JNLP port
  xmlstarlet -q ed --inplace -u "/hudson/slaveAgentPort" -v 33453 config.xml

  cd $plugins_dir || { echo "unable to chdir to [$plugins_dir]"; exit 1; }

  # List of plugins that are needed to be installed 
  plugin_list="git-client git github-api github-oauth github MSBuild ssh-slaves workflow-aggregator ws-cleanup"

  # remove existing plugins, if any ...
  rm -rfv $plugin_list

  for plugin in $plugin_list; do
      echo "installing plugin [$plugin] ..."
      java -jar $jenkins_dir/jenkins-cli.jar -s http://127.0.0.1:8080/ -auth admin:$PASSWORD install-plugin $plugin
  done

  # Restart jenkins after installing plugins
  java -jar $jenkins_dir/jenkins-cli.jar -s http://127.0.0.1:8080 -auth admin:$PASSWORD safe-restart
}

### script starts here ###

install_packages

wait_for_jenkins

updating_jenkins_master_password

wait_for_jenkins

configure_jenkins_server

echo "Done"
exit 0

There is a lot of stuff that has been covered here. But the most tricky bit is changing Jenkins password. Here we are using a python script which uses brcypt to hash the plain text in Jenkins encryption format and xmlstarlet for replacing that password in the actual location. Also, we are using xmstarlet to edit the JNLP port for windows slave. Do remember initial username for Jenkins is admin.

Command to run: Initialize terraform – terraform init , Check and apply – terraform plan -> terraform apply

After successfully running apply command go to AWS console and check for a new instance coming up. Hit the <public ip=””>:8080 and enter credentials as you had passed and you will have the Jenkins master for yourself ready to be used. </public>

Note: I will be providing the terraform script and permission list of IAM roles for the user at the end of the blog.

Creating Terraform Script for Spinning up Linux Slave and connect it to master

We won’t be creating a new image here rather use the same one that we used for Jenkins master.

VPC will be same and updated Security groups for slave are below:

resource "aws_security_group" "dev_jenkins_worker_linux" {
  name        = "dev_jenkins_worker_linux"
  description = "Jenkins Server: created by Terraform for [dev]"

# legacy name of VPC ID
  vpc_id = "${data.aws_vpc.default_vpc.id}"

  tags {
    Name = "dev_jenkins_worker_linux"
    env  = "dev"
  }
}

###############################################################################
# ALL INBOUND
###############################################################################

# ssh
resource "aws_security_group_rule" "jenkins_worker_linux_from_source_ingress_ssh" {
  type              = "ingress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_linux.id}"
  cidr_blocks       = ["<Your Public IP>/32"]
  description       = "ssh to jenkins_worker_linux"
}

# ssh
resource "aws_security_group_rule" "jenkins_worker_linux_from_source_ingress_webui" {
  type              = "ingress"
  from_port         = 8080
  to_port           = 8080
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_linux.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "ssh to jenkins_worker_linux"
}


###############################################################################
# ALL OUTBOUND
###############################################################################

resource "aws_security_group_rule" "jenkins_worker_linux_to_all_80" {
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_linux.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker to all 80"
}

resource "aws_security_group_rule" "jenkins_worker_linux_to_all_443" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_linux.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker to all 443"
}

resource "aws_security_group_rule" "jenkins_worker_linux_to_other_machines_ssh" {
  type              = "egress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_linux.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker linux to jenkins server"
}

resource "aws_security_group_rule" "jenkins_worker_linux_to_jenkins_server_8080" {
  type                     = "egress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = "${aws_security_group.dev_jenkins_worker_linux.id}"
  source_security_group_id = "${aws_security_group.jenkins_server.id}"
  description              = "allow jenkins workers linux to jenkins server"
}

Now that we have the required security groups in place it is time to bring into light terraform script for linux slave.

data "aws_ami" "jenkins_worker_linux" {
  most_recent      = true
  owners           = ["self"]

  filter {
    name   = "name"
    values = ["amazon-linux-for-jenkins*"]
  }
}

resource "aws_key_pair" "jenkins_worker_linux" {
  key_name   = "jenkins_worker_linux"
  public_key = "${file("jenkins_worker.pub")}"
}

data "local_file" "jenkins_worker_pem" {
  filename = "${path.module}/jenkins_worker.pem"
}

data "template_file" "userdata_jenkins_worker_linux" {
  template = "${file("scripts/jenkins_worker_linux.sh")}"

  vars {
    env         = "dev"
    region      = "us-east-1"
    datacenter  = "dev-us-east-1"
    node_name   = "us-east-1-jenkins_worker_linux"
    domain      = ""
    device_name = "eth0"
    server_ip   = "${aws_instance.jenkins_server.private_ip}"
    worker_pem  = "${data.local_file.jenkins_worker_pem.content}"
    jenkins_username = "admin"
    jenkins_password = "mysupersecretpassword"
  }
}

# lookup the security group of the Jenkins Server
data "aws_security_group" "jenkins_worker_linux" {
  filter {
    name   = "group-name"
    values = ["dev_jenkins_worker_linux"]
  }
}

resource "aws_launch_configuration" "jenkins_worker_linux" {
  name_prefix                 = "dev-jenkins-worker-linux"
  image_id                    = "${data.aws_ami.jenkins_worker_linux.image_id}"
  instance_type               = "t3.medium"
  iam_instance_profile        = "dev_jenkins_worker_linux"
  key_name                    = "${aws_key_pair.jenkins_worker_linux.key_name}"
  security_groups             = ["${data.aws_security_group.jenkins_worker_linux.id}"]
  user_data                   = "${data.template_file.userdata_jenkins_worker_linux.rendered}"
  associate_public_ip_address = false

  root_block_device {
    delete_on_termination = true
    volume_size = 100
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "jenkins_worker_linux" {
  name                      = "dev-jenkins-worker-linux"
  min_size                  = "1"
  max_size                  = "2"
  desired_capacity          = "2"
  health_check_grace_period = 60
  health_check_type         = "EC2"
  vpc_zone_identifier       = ["${data.aws_subnet_ids.default_public.ids}"]
  launch_configuration      = "${aws_launch_configuration.jenkins_worker_linux.name}"
  termination_policies      = ["OldestLaunchConfiguration"]
  wait_for_capacity_timeout = "10m"
  default_cooldown          = 60

  tags = [
    {
      key                 = "Name"
      value               = "dev_jenkins_worker_linux"
      propagate_at_launch = true
    },
    {
      key                 = "class"
      value               = "dev_jenkins_worker_linux"
      propagate_at_launch = true
    },
  ]
}

And now the final piece of code, which is user-data of slave machine.

#!/bin/bash

set -x

function wait_for_jenkins ()
{
    echo "Waiting jenkins to launch on 8080..."

    while (( 1 )); do
        echo "Waiting for Jenkins"

        nc -zv ${server_ip} 8080
        if (( $? == 0 )); then
            break
        fi

        sleep 10
    done

    echo "Jenkins launched"
}

function slave_setup()
{
    # Wait till jar file gets available
    ret=1
    while (( $ret != 0 )); do
        wget -O /opt/jenkins-cli.jar http://${server_ip}:8080/jnlpJars/jenkins-cli.jar
        ret=$?

        echo "jenkins cli ret [$ret]"
    done

    ret=1
    while (( $ret != 0 )); do
        wget -O /opt/slave.jar http://${server_ip}:8080/jnlpJars/slave.jar
        ret=$?

        echo "jenkins slave ret [$ret]"
    done
    
    mkdir -p /opt/jenkins-slave
    chown -R ec2-user:ec2-user /opt/jenkins-slave

    # Register_slave
    JENKINS_URL="http://${server_ip}:8080"

    USERNAME="${jenkins_username}"
    
    # PASSWORD=$(cat /tmp/secret)
    PASSWORD="${jenkins_password}"

    SLAVE_IP=$(ip -o -4 addr list ${device_name} | head -n1 | awk '{print $4}' | cut -d/ -f1)
    NODE_NAME=$(echo "jenkins-slave-linux-$SLAVE_IP" | tr '.' '-')
    NODE_SLAVE_HOME="/opt/jenkins-slave"
    EXECUTORS=2
    SSH_PORT=22

    CRED_ID="$NODE_NAME"
    LABELS="build linux docker"
    USERID="ec2-user"

    cd /opt
    
    # Creating CMD utility for jenkins-cli commands
    jenkins_cmd="java -jar /opt/jenkins-cli.jar -s $JENKINS_URL -auth $USERNAME:$PASSWORD"

    # Waiting for Jenkins to load all plugins
    while (( 1 )); do

      count=$($jenkins_cmd list-plugins 2>/dev/null | wc -l)
      ret=$?

      echo "count [$count] ret [$ret]"

      if (( $count > 0 )); then
          break
      fi

      sleep 30
    done

    # Delete Credentials if present for respective slave machines
    $jenkins_cmd delete-credentials system::system::jenkins _ $CRED_ID

    # Generating cred.xml for creating credentials on Jenkins server
    cat > /tmp/cred.xml <<EOF
<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey plugin="ssh-credentials@1.16">
  <scope>GLOBAL</scope>
  <id>$CRED_ID</id>
  <description>Generated via Terraform for $SLAVE_IP</description>
  <username>$USERID</username>
  <privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\$DirectEntryPrivateKeySource">
    <privateKey>${worker_pem}</privateKey>
  </privateKeySource>
</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
EOF

    # Creating credential using cred.xml
    cat /tmp/cred.xml | $jenkins_cmd create-credentials-by-xml system::system::jenkins _

    # For Deleting Node, used when testing
    $jenkins_cmd delete-node $NODE_NAME
    
    # Generating node.xml for creating node on Jenkins server
    cat > /tmp/node.xml <<EOF
<slave>
  <name>$NODE_NAME</name>
  <description>Linux Slave</description>
  <remoteFS>$NODE_SLAVE_HOME</remoteFS>
  <numExecutors>$EXECUTORS</numExecutors>
  <mode>NORMAL</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy\$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.5">
    <host>$SLAVE_IP</host>
    <port>$SSH_PORT</port>
    <credentialsId>$CRED_ID</credentialsId>
  </launcher>
  <label>$LABELS</label>
  <nodeProperties/>
  <userId>$USERID</userId>
</slave>
EOF

  sleep 10
  
  # Creating node using node.xml
  cat /tmp/node.xml | $jenkins_cmd create-node $NODE_NAME
}

### script begins here ###

wait_for_jenkins

slave_setup

echo "Done"
exit 0

This will not only create a node on Jenkins master but also attach it.

Command to run: Initialize terraform – terraform init, Check and apply – terraform plan -> terraform apply

One drawback of this is, if by any chance slave gets disconnected or goes down, it will remain on Jenkins master as offline, also it will not manually attach itself to Jenkins master.

Some solutions for them are:

1. Create a cron job on the slave which will run user-data after a certain interval.

2. Use swarm plugin.

3. As we are on AWS, we can even use Amazon EC2 Plugin.

Maybe in a future blog, we will cover using both of these plugins as well.

Using Packer to create AMI’s for Windows Slave

Windows AMI will also be created using packer. All the pointers for Windows will remain as it were for Linux.

{
  "variables": {
    "ami-description": "Windows Server for Jenkins Slave ({{isotime \"2006-01-02-15-04-05\"}})",
    "ami-name": "windows-slave-for-jenkins-{{isotime \"2006-01-02-15-04-05\"}}",
    "aws_access_key": "",
    "aws_secret_key": ""
  },

  "builders": [
    {
      "ami_description": "{{user `ami-description`}}",
      "ami_name": "{{user `ami-name`}}",
      "ami_regions": [
        "us-east-1"
      ],
      "ami_users": [
        "XXXXXXXXXX"
      ],
      "ena_support": "true",
      "instance_type": "t3.medium",
      "region": "us-east-1",
      "source_ami_filter": {
        "filters": {
          "name": "Windows_Server-2016-English-Full-Containers-*",
          "root-device-type": "ebs",
          "virtualization-type": "hvm"
        },
        "most_recent": true,
        "owners": [
          "amazon"
        ]
      },
      "sriov_support": "true",
      "user_data_file": "scripts/SetUpWinRM.ps1",
      "communicator": "winrm",
      "winrm_username": "Administrator",
      "winrm_insecure": true,
      "winrm_use_ssl": true,
      "tags": {
        "Name": "{{user `ami-name`}}"
      },
      "type": "amazon-ebs"
    }
  ],
  "post-processors": [
  {
    "inline": [
      "echo AMI Name {{user `ami-name`}}",
      "date",
      "exit 0"
    ],
    "type": "shell-local"
  }
  ],
  "provisioners": [
    {
      "type": "powershell",
      "valid_exit_codes": [ 0, 3010 ],
      "scripts": [
        "scripts/disable-uac.ps1",
        "scripts/enable-rdp.ps1",
        "install_windows.ps1"
      ]
    },
    {
      "type": "windows-restart",
      "restart_check_command": "powershell -command \"& {Write-Output 'restarted.'}\""
    },
    {
      "type": "powershell",
      "inline": [
        "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
        "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown"
      ]
    }
  ]
}

Now when it comes to windows one should know that it does not behave the same way Linux does. For us to be able to communicate with this image an essential component required is WinRM. We set it up at the very beginning as part of user_data_file. Also, windows require user input for a lot of things and while automating it is not possible to provide it as it will break the flow of execution so we disable UAC and enable RDP so that we can connect to that machine from our local desktop for debugging if needed. And at last, we will execute install_windows.ps1 file which will set up our slave. Please note at the last we are calling two PowerShell scripts to generate random password every time a new machine is created. It is mandatory to have them or you will never be able to login into your machines.

There are multiple user-data in the above code, let’s understand them in their order of appearance.

SetUpWinRM.ps1:

<powershell>

write-output "Running User Data Script"
write-host "(host) Running User Data Script"

Set-ExecutionPolicy Unrestricted -Scope LocalMachine -Force -ErrorAction Ignore

# Don't set this before Set-ExecutionPolicy as it throws an error
$ErrorActionPreference = "stop"

# Remove HTTP listener
Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse

$Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName "packer"
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint -Force

# WinRM
write-output "Setting up WinRM"
write-host "(host) setting up WinRM"

cmd.exe /c winrm quickconfig -q
cmd.exe /c winrm set "winrm/config" '@{MaxTimeoutms="1800000"}'
cmd.exe /c winrm set "winrm/config/winrs" '@{MaxMemoryPerShellMB="1024"}'
cmd.exe /c winrm set "winrm/config/service" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/client" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/client/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{CredSSP="true"}'
cmd.exe /c winrm set "winrm/config/listener?Address=*+Transport=HTTPS" "@{Port=`"5986`";Hostname=`"packer`";CertificateThumbprint=`"$($Cert.Thumbprint)`"}"
cmd.exe /c netsh advfirewall firewall set rule group="remote administration" new enable=yes
cmd.exe /c netsh firewall add portopening TCP 5986 "Port 5986"
cmd.exe /c net stop winrm
cmd.exe /c sc config winrm start= auto
cmd.exe /c net start winrm

</powershell>

The content is pretty straightforward as it is just setting up WInRM. The only thing that matters here is the <powershell> and </powershell>. They are mandatory as packer will not be able to understand what is the type of script. Next, we come across disable-uac.ps1 & enable-rdp.ps1, and we have discussed their purpose before. The last user-data is the actual user-data that we need to install all the required packages in the AMI.

Chocolatey: a blessing in disguise – Installing required applications in windows by scripting is a real headache as you have to write a lot of stuff just to install a single application but luckily for us we have chocolatey. It works as a package manager for windows and helps us install applications as we are installing packages on Linux. install_windows.ps1 has installation step for chocolatey and how it can be used to install other applications on windows.

See, such a small script and you can get all the components to run your Windows application in no time (Kidding… This script actually takes around 20 minutes to run :P)

Remaining user-data can be found here.

Now that we have the image for ourselves let’s start with terraform script to make this machine a slave of your Jenkins master.

Creating Terraform Script for Spinning up Windows Slave and Connect it to Master

This time also we will first create the security groups and then create the slave machine from the same AMI that we developed above.

resource "aws_security_group" "dev_jenkins_worker_windows" {
  name        = "dev_jenkins_worker_windows"
  description = "Jenkins Server: created by Terraform for [dev]"

  # legacy name of VPC ID
  vpc_id = "${data.aws_vpc.default_vpc.id}"

  tags {
    Name = "dev_jenkins_worker_windows"
    env  = "dev"
  }
}

###############################################################################
# ALL INBOUND
###############################################################################

# ssh
resource "aws_security_group_rule" "jenkins_worker_windows_from_source_ingress_webui" {
  type              = "ingress"
  from_port         = 8080
  to_port           = 8080
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "ssh to jenkins_worker_windows"
}

# rdp
resource "aws_security_group_rule" "jenkins_worker_windows_from_rdp" {
  type              = "ingress"
  from_port         = 3389
  to_port           = 3389
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["<Your Public IP>/32"]
  description       = "rdp to jenkins_worker_windows"
}

###############################################################################
# ALL OUTBOUND
###############################################################################

resource "aws_security_group_rule" "jenkins_worker_windows_to_all_80" {
  type              = "egress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker to all 80"
}

resource "aws_security_group_rule" "jenkins_worker_windows_to_all_443" {
  type              = "egress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker to all 443"
}

resource "aws_security_group_rule" "jenkins_worker_windows_to_jenkins_server_33453" {
  type              = "egress"
  from_port         = 33453
  to_port           = 33453
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["172.31.0.0/16"]
  description       = "allow jenkins worker windows to jenkins server"
}

resource "aws_security_group_rule" "jenkins_worker_windows_to_jenkins_server_8080" {
  type                     = "egress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = "${aws_security_group.dev_jenkins_worker_windows.id}"
  source_security_group_id = "${aws_security_group.jenkins_server.id}"
  description              = "allow jenkins workers windows to jenkins server"
}

resource "aws_security_group_rule" "jenkins_worker_windows_to_all_22" {
  type              = "egress"
  from_port         = 22
  to_port           = 22
  protocol          = "tcp"
  security_group_id = "${aws_security_group.dev_jenkins_worker_windows.id}"
  cidr_blocks       = ["0.0.0.0/0"]
  description       = "allow jenkins worker windows to connect outbound from 22"
}

Once security groups are in place we move towards creating the terraform file for windows machine itself. Windows can’t connect to Jenkins master using SSH the method we used while connecting the Linux slave instead we have to use JNLP. A quick recap, when creating Jenkins master we used xmlstarlet to modify the JNLP port and also added rules in sg group to allow connection for JNLP. Also, we have opened the port for RDP so that if any issue occurs you can get in the machine and debug it.

Terraform file:

# Setting Up Windows Slave 
data "aws_ami" "jenkins_worker_windows" {
  most_recent      = true
  owners           = ["self"]

  filter {
    name   = "name"
    values = ["windows-slave-for-jenkins*"]
  }
}

resource "aws_key_pair" "jenkins_worker_windows" {
  key_name   = "jenkins_worker_windows"
  public_key = "${file("jenkins_worker.pub")}"
}

data "template_file" "userdata_jenkins_worker_windows" {
  template = "${file("scripts/jenkins_worker_windows.ps1")}"

  vars {
    env         = "dev"
    region      = "us-east-1"
    datacenter  = "dev-us-east-1"
    node_name   = "us-east-1-jenkins_worker_windows"
    domain      = ""
    device_name = "eth0"
    server_ip   = "${aws_instance.jenkins_server.private_ip}"
    worker_pem  = "${data.local_file.jenkins_worker_pem.content}"
    jenkins_username = "admin"
    jenkins_password = "mysupersecretpassword"
  }
}

# lookup the security group of the Jenkins Server
data "aws_security_group" "jenkins_worker_windows" {
  filter {
    name   = "group-name"
    values = ["dev_jenkins_worker_windows"]
  }
}

resource "aws_launch_configuration" "jenkins_worker_windows" {
  name_prefix                 = "dev-jenkins-worker-"
  image_id                    = "${data.aws_ami.jenkins_worker_windows.image_id}"
  instance_type               = "t3.medium"
  iam_instance_profile        = "dev_jenkins_worker_windows"
  key_name                    = "${aws_key_pair.jenkins_worker_windows.key_name}"
  security_groups             = ["${data.aws_security_group.jenkins_worker_windows.id}"]
  user_data                   = "${data.template_file.userdata_jenkins_worker_windows.rendered}"
  associate_public_ip_address = false

  root_block_device {
    delete_on_termination = true
    volume_size = 100
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "jenkins_worker_windows" {
  name                      = "dev-jenkins-worker-windows"
  min_size                  = "1"
  max_size                  = "2"
  desired_capacity          = "2"
  health_check_grace_period = 60
  health_check_type         = "EC2"
  vpc_zone_identifier       = ["${data.aws_subnet_ids.default_public.ids}"]
  launch_configuration      = "${aws_launch_configuration.jenkins_worker_windows.name}"
  termination_policies      = ["OldestLaunchConfiguration"]
  wait_for_capacity_timeout = "10m"
  default_cooldown          = 60

  #lifecycle {
  #  create_before_destroy = true
  #}


  ## on replacement, gives new service time to spin up before moving on to destroy
  #provisioner "local-exec" {
  #  command = "sleep 60"
  #}

  tags = [
    {
      key                 = "Name"
      value               = "dev_jenkins_worker_windows"
      propagate_at_launch = true
    },
    {
      key                 = "class"
      value               = "dev_jenkins_worker_windows"
      propagate_at_launch = true
    },
  ]
}

Finally, we reach the user-data for the terraform plan. It will download the required jar file, create a node on Jenkins and register itself as a slave.

<powershell>

function Wait-For-Jenkins {

  Write-Host "Waiting jenkins to launch on 8080..."

  Do {
  Write-Host "Waiting for Jenkins"

   Nc -zv ${server_ip} 8080
   If( $? -eq $true ) {
     Break
   }
   Sleep 10

  } While (1)

  Do {
   Write-Host "Waiting for JNLP"
      
   Nc -zv ${server_ip} 33453
   If( $? -eq $true ) {
    Break
   }
   Sleep 10

  } While (1)      

  Write-Host "Jenkins launched"
}

function Slave-Setup()
{
  # Register_slave
  $JENKINS_URL="http://${server_ip}:8080"

  $USERNAME="${jenkins_username}"
  
  $PASSWORD="${jenkins_password}"

  $AUTH = -join ("$USERNAME", ":", "$PASSWORD")
  echo $AUTH

  # Below IP collection logic works for Windows Server 2016 edition and needs testing for windows server 2008 edition
  $SLAVE_IP=(ipconfig | findstr /r "[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*" | findstr "IPv4 Address").substring(39) | findstr /B "172.31"
  
  $NODE_NAME="jenkins-slave-windows-$SLAVE_IP"
  
  $NODE_SLAVE_HOME="C:\Jenkins\"
  $EXECUTORS=2
  $JNLP_PORT=33453

  $CRED_ID="$NODE_NAME"
  $LABELS="build windows"
  
  # Creating CMD utility for jenkins-cli commands
  # This is not working in windows therefore specify full path
  $jenkins_cmd = "java -jar C:\Jenkins\jenkins-cli.jar -s $JENKINS_URL -auth admin:$PASSWORD"

  Sleep 20

  Write-Host "Downloading jenkins-cli.jar file"
  (New-Object System.Net.WebClient).DownloadFile("$JENKINS_URL/jnlpJars/jenkins-cli.jar", "C:\Jenkins\jenkins-cli.jar")

  Write-Host "Downloading slave.jar file"
  (New-Object System.Net.WebClient).DownloadFile("$JENKINS_URL/jnlpJars/slave.jar", "C:\Jenkins\slave.jar")

  Sleep 10

  # Waiting for Jenkins to load all plugins
  Do {
  
    $count=(java -jar C:\Jenkins\jenkins-cli.jar -s $JENKINS_URL -auth $AUTH list-plugins | Measure-Object -line).Lines
    $ret=$?

    Write-Host "count [$count] ret [$ret]"

    If ( $count -gt 0 ) {
        Break
    }

    sleep 30
  } While ( 1 )

  # For Deleting Node, used when testing
  Write-Host "Deleting Node $NODE_NAME if present"
  java -jar C:\Jenkins\jenkins-cli.jar -s $JENKINS_URL -auth $AUTH delete-node $NODE_NAME
  
  # Generating node.xml for creating node on Jenkins server
  $NodeXml = @"
<slave>
<name>$NODE_NAME</name>
<description>Windows Slave</description>
<remoteFS>$NODE_SLAVE_HOME</remoteFS>
<numExecutors>$EXECUTORS</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy`$Always`"/>
<launcher class="hudson.slaves.JNLPLauncher">
  <workDirSettings>
    <disabled>false</disabled>
    <internalDir>remoting</internalDir>
    <failIfWorkDirIsMissing>false</failIfWorkDirIsMissing>
  </workDirSettings>
</launcher>
<label>$LABELS</label>
<nodeProperties/>
</slave>
"@
  $NodeXml | Out-File -FilePath C:\Jenkins\node.xml 

  type C:\Jenkins\node.xml

  # Creating node using node.xml
  Write-Host "Creating $NODE_NAME"
  Get-Content -Path C:\Jenkins\node.xml | java -jar C:\Jenkins\jenkins-cli.jar -s $JENKINS_URL -auth $AUTH create-node $NODE_NAME

  Write-Host "Registering Node $NODE_NAME via JNLP"
  Start-Process java -ArgumentList "-jar C:\Jenkins\slave.jar -jnlpCredentials $AUTH -jnlpUrl $JENKINS_URL/computer/$NODE_NAME/slave-agent.jnlp"
}

### script begins here ###

Wait-For-Jenkins

Slave-Setup

echo "Done"
</powershell>
<persist>true</persist>

Command to run: Initialize terraform – terraform init, Check and apply – terraform plan -> terraform apply

Same drawbacks are applicable here and the same solutions will work here as well.

Congratulations! You have a Jenkins master with Windows and Linux slave attached to it.

IAM roles for reference

‍Jenkins Master

Linux Slave

Windows Slave

Bonus:

If you want to associate IAM permissions to the user but cannot assign FULL ACCESS here is a curated list below for reference:

Packer Policy

Terraform Policy

Conclusion:

This blog tries to highlight one of the ways in which we can use packer and Terraform to create AMI’s which will serve as Jenkins master and slave. We not only covered their creation but also focused on how to associate security groups and checked some of the basic IAM roles that can be applied. Although we have covered almost all the possible scenarios but still depending on use case, the required changes would be very less and this can serve as a boiler plate code when beginning to plan your infrastructure on cloud.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts