Introduction

You've started down the path of using configuration management and decided to use Ansible. You began with a big monolithic playbook and now it's time to break it down into modular roles that can be applied to groups of hosts as needed.

As you create roles, how do you ensure that your roles are working as expected, before applying the roles to your servers? You also have a heterogeneous environment with multiple operating systems. How do you ensure that updates to existing roles will continue to work as expected?

This guide will show you how to test your roles against multiple operating systems on your local development workstation before committing your changes to version control. (You are version controlling your roles, right?)

Stack

We will use the following tools to create a local Ansible development environment.

Setup

Installing the KVM Hypervisor

You will first need to setup the KVM hypervisor on your workstation, assuming your CPU supports virtualization extensions (most recent CPUs do). If you don't have KVM already installed, install the virtualization group packages:

sudo dnf install @virtualization

Start the libvirtd service and have it start at boot time:

sudo systemctl start libvirtd.service
sudo systemctl enable libvirtd.service

Installing Packages with DNF

This setup requires Vagrant, a tool to create and configure lightweight virtual machines. It works great for creating development environments locally using virtual machines. Vagrant has plugins that support libvirt as a provider. Therefore, we can use Vagrant to launch virtual machines using the local KVM hypervisor.

sudo dnf install vagrant vagrant-libvirt

The Vagrant installation page warns that some distributions have much older versions in their repositories. At the time of this writing, Fedora 25 is one minor release (1.8.5) behind the latest available (1.9.2) and is sufficient for this guide.

Other development packages are also required for building Molecule:

sudo dnf install gcc python-devel openssl-devel libffi-devel

Installing Packages with pip

The remainder of the required packages are not available in the Fedora 25 repositories. Therefore, we'll go directly to PyPi to fetch these packages:

sudo pip install molecule testinfra python-vagrant ansible

If you already have a version of Ansible installed from the repositories and don't want to update, use pip list to check the current version and then specify the existing version when calling pip install:

pip list | grep -w ^ansible
sudo pip install molecule testinfra python-vagrant ansible==2.2.1.0

Initializing a Role with Molecule

Molecule can take of initializing a new role. It will create the default directories, similar to ansible-galaxy init. It will also create a molecule.yml and a playbook.yml file. These files will instruct Molecule which Vagrant boxes to use, and run TestInfra tests against the virtual machine after the role has been applied. Create a new NTP role:

molecule init --role ansible-role-ntp --driver vagrant --verifier testinfra

This will create a new directory named ansible-role-ntp. Change to this directory and start building out the role:

cd ansible-role-ntp

Building an NTP Role

We need a role to test. If you already have a role that you would like to test, skip to the next section (Setting up the Tests).

A role for NTP makes a good example because it requires multiple Ansible modules for installing the package, customizing the configuration, and starting the service. Also, the NTP daemon name is different between CentOS and Debian, which requires us to use role variables.

Tasks

Configure the tasks that Ansible will carry out to install and configure NTP both CentOS 7 and Debian 8. We will provide a name for each task which serves as built-in documentation for each task. Add the following tasks to tasks/main.yml:

---
- name: Include OS-specific variables.
  include_vars: "{{ ansible_os_family }}.yml"

- name: Install NTP.
  package:
    name: ntp
    state: present

- name: Set Time Zone.
  file:
    src: "/usr/share/zoneinfo/{{ ntp_timezone }}"
    dest: /etc/localtime
    state: link
    force: yes

- name: Configure main NTP configuration file.
  template:
    src: ntp.conf.j2
    dest: /etc/ntp.conf
    owner: root
    group: root
    mode: 0644
  notify:
    - restart ntp

- name: Ensure the NTP service is running and enabled at boot time.
  service:
    name: "{{ ntp_daemon }}"
    enabled: yes
    state: started

There are three variables in the tasks above: ansible_os_family, ntp_timezone, and ntp_daemon.

Here, we take advantage of Ansible's fact gathering. When the role is applied, Ansible will retrieve facts about the remote system and return them as variables that can be used throughout the role. The ansible_os_family variable will resolve to either RedHat or Debian. Therefore, the role will include all variables in either vars/RedHat.yml or vars/Debian.yml, depending on which OS is being tested. This design allows us to support multiple platforms without duplicating tasks, one for each OS.

The other two variables in this role are custom variables. It is a good practice to prefix custom variables with the name of the role, in this case ntp_. This will avoid clashing of variables when applying multiple roles at the same time.

Vars

Variables in Ansible can exist in multiple places. Our role will put variables in two places: vars/ and defaults/. We'll use the vars/ directory for effectively hardcoding variables based on the operating system. For example, we know that the NTP daemon on Debian is named ntp, while on RedHat it is named ntpd.

Edit the vars/Debian.yml file:

---
ntp_daemon: ntp

Edit the vars/RedHat.yml file:

---
ntp_daemon: ntpd

Defaults

What about that other custom variables? We'll put the other custom variables in defaults/main.yml so that users can modify them when applying the role.

Edit the defaults/main.yml file:

---
ntp_timezone: America/New_York
ntp_servers:
    - 0.us.pool.ntp.org
    - 1.us.pool.ntp.org
    - 2.us.pool.ntp.org
    - 3.us.pool.ntp.org

Handlers

The handler is responsible for restarting the NTP service after Ansible detects a change in the configuration. Edit the handlers/main.yml file:

- name: restart ntp
  service:
    name: "{{ ntp_daemon }}"
    state: restarted

Templates

Ansible uses the Jinja2 templating engine. We will replace certain parameters in the ntp.conf file with variables. When Ansible applies the role, the values of the variables are replaced.

Edit the templates/ntp.conf.j2 file:

# {{ ansible_managed }}
#
# Keep ntpd from panicking in the event of a large clock skew
# when a VM guest is suspended and resumed.
{% if ansible_virtualization_role == 'guest' %}
tinker panic 0
{% endif %}

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

{% for server in ntp_servers %}
server {{ server }} iburst
{% endfor %}

# Driftfile
driftfile /var/lib/ntp/drift

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

Edit the templates/timezone.j2 file:

{{ ntp_timezone }}

Setting up the Tests

The role is complete. Now it's time to setup the tests. TestInfra aims to be a Python equivalent for Serverspec. It allows you write tests to make sure your servers are configured correctly. The idea is to configure your servers with Ansible and test the state of the servers after configuration has been applied.

If you are not using the NTP role and instead using one of your own roles while following this guide, you can add Molecule to an existing role:

cd my-role
molecule init --driver vagrant --verifier testinfra

TestInfra has several modules to run specific tests. We will use some of these modules to add some tests for our role.

Edit the tests/test_default.py file:

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


def test_ntp_package(Package):
    assert Package("ntp").is_installed


def test_ntp_file(File):
    f = File("/etc/ntp.conf")
    assert f.exists
    assert f.is_file


def test_ntp_file_content(File):
    f = File("/etc/ntp.conf")
    assert f.contains("0.us.pool.ntp.org")


def test_ntp_service(Service, SystemInfo):
    if SystemInfo.distribution == "debian":
        ntp_daemon = "ntp"
    elif SystemInfo.distribution == "centos":
        ntp_daemon = "ntpd"

    s = Service(ntp_daemon)
    assert s.is_running
    assert s.is_enabled

Configuring Molecule

The role is complete and the tests are ready. Now, the last piece is to configure Molecule to use Vagrant with two platforms: CentOS 7 and Debian 8. We also instruct Molecule to use the libvirt provider with the kvm driver. The virtual machines will be configured with 1024MB of RAM and 1 vCPU.

Edit the molecule.yml file:

---
driver:
  name: vagrant
vagrant:
  platforms:
    - name: centos7
      box: centos/7
    - name: debian8
      box: debian/jessie64
  providers:
    - name: libvirt
      type: libvirt
      options:
        memory: 1024
        cpus: 1
      driver: kvm
  instances:
    - name: ansible-role-ntp
      ansible_groups:
        - group1
verifier:
  name: testinfra

Testing the Role

Testing the role is simple after everything else is in place:

molecule test

This command will launch Vagrant, which will download a CentOS 7 image and create a new local virtual machine. When the virtual machine is ready, which typically takes a few seconds, Ansible will apply the role and then Molecule will call TestInfra to run the tests.

Here is the output from the command above:

--> Destroying instances...
--> Checking playbook's syntax...

playbook: playbook.yml
--> Creating instances...
Bringing machine 'ansible-role-ntp' up with 'libvirt' provider...
==> ansible-role-ntp: Creating image (snapshot of base box volume).
==> ansible-role-ntp: Creating domain with the following settings...
==> ansible-role-ntp:  -- Name:              ansible-role-ntp_ansible-role-ntp
==> ansible-role-ntp:  -- Domain type:       kvm
==> ansible-role-ntp:  -- Cpus:              1
==> ansible-role-ntp:  -- Memory:            1024M
==> ansible-role-ntp:  -- Management MAC:
==> ansible-role-ntp:  -- Loader:
==> ansible-role-ntp:  -- Base box:          centos/7
==> ansible-role-ntp:  -- Storage pool:      default
==> ansible-role-ntp:  -- Image:             /var/lib/libvirt/images/ansible-role-ntp_ansible-role-ntp.img (41G)
==> ansible-role-ntp:  -- Volume Cache:      default
==> ansible-role-ntp:  -- Kernel:
==> ansible-role-ntp:  -- Initrd:
==> ansible-role-ntp:  -- Graphics Type:     vnc
==> ansible-role-ntp:  -- Graphics Port:     5900
==> ansible-role-ntp:  -- Graphics IP:       127.0.0.1
==> ansible-role-ntp:  -- Graphics Password: Not defined
==> ansible-role-ntp:  -- Video Type:        cirrus
==> ansible-role-ntp:  -- Video VRAM:        9216
==> ansible-role-ntp:  -- Keymap:            en-us
==> ansible-role-ntp:  -- TPM Path:
==> ansible-role-ntp:  -- INPUT:             type=mouse, bus=ps2
==> ansible-role-ntp:  -- Command line :
==> ansible-role-ntp: Creating shared folders metadata...
==> ansible-role-ntp: Starting domain.
==> ansible-role-ntp: Waiting for domain to get an IP address...
==> ansible-role-ntp: Waiting for SSH to become available...
    ansible-role-ntp:
    ansible-role-ntp: Vagrant insecure key detected. Vagrant will automatically replace
    ansible-role-ntp: this with a newly generated keypair for better security.
    ansible-role-ntp:
    ansible-role-ntp: Inserting generated public key within guest...
    ansible-role-ntp: Removing insecure key from the guest if it's present...
    ansible-role-ntp: Key inserted! Disconnecting and reconnecting using new SSH key...
==> ansible-role-ntp: Setting hostname...
==> ansible-role-ntp: Configuring and enabling network interfaces...
==> ansible-role-ntp: Rsyncing folder: /home/torresgi/tmp/ansible-role-ntp/ => /vagrant
==> ansible-role-ntp: Machine not provisioned because `--no-provision` is specified.
--> Starting Ansible Run...

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [ansible-role-ntp]

TASK [ansible-role-ntp : Include OS-specific variables] ************************
ok: [ansible-role-ntp]

TASK [ansible-role-ntp : Install NTP] ******************************************
changed: [ansible-role-ntp]

TASK [ansible-role-ntp : Set Time Zone] ****************************************
changed: [ansible-role-ntp]

TASK [ansible-role-ntp : Configure main NTP configuration file] ****************
--- before: /etc/ntp.conf
+++ after: dynamically generated
@@ -1,55 +1,27 @@
-# For more information about this file, see the man pages
-# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
-
-driftfile /var/lib/ntp/drift
+# Ansible managed: Do NOT edit this file manually!
+#
+# Keep ntpd from panicking in the event of a large clock skew
+# when a VM guest is suspended and resumed.
+tinker panic 0

 # Permit time synchronization with our time source, but do not
 # permit the source to query or modify the service on this system.
-restrict default nomodify notrap nopeer noquery
+restrict default kod nomodify notrap nopeer noquery
+restrict -6 default kod nomodify notrap nopeer noquery

 # Permit all access over the loopback interface.  This could
 # be tightened as well, but to do so would effect some of
 # the administrative functions.
-restrict 127.0.0.1
-restrict ::1
+restrict 127.0.0.1
+restrict -6 ::1

-# Hosts on local network are less restricted.
-#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
+server 0.us.pool.ntp.org iburst
+server 1.us.pool.ntp.org iburst
+server 2.us.pool.ntp.org iburst
+server 3.us.pool.ntp.org iburst

-# Use public servers from the pool.ntp.org project.
-# Please consider joining the pool (http://www.pool.ntp.org/join.html).
-server 0.centos.pool.ntp.org iburst
-server 1.centos.pool.ntp.org iburst
-server 2.centos.pool.ntp.org iburst
-server 3.centos.pool.ntp.org iburst
-
-#broadcast 192.168.1.255 autokey   # broadcast server
-#broadcastclient           # broadcast client
-#broadcast 224.0.1.1 autokey       # multicast server
-#multicastclient 224.0.1.1     # multicast client
-#manycastserver 239.255.254.254        # manycast server
-#manycastclient 239.255.254.254 autokey # manycast client
-
-# Enable public key cryptography.
-#crypto
-
-includefile /etc/ntp/crypto/pw
-
-# Key file containing the keys and key identifiers used when operating
-# with symmetric key cryptography.
-keys /etc/ntp/keys
-
-# Specify the key identifiers which are trusted.
-#trustedkey 4 8 42
-
-# Specify the key identifier to use with the ntpdc utility.
-#requestkey 8
-
-# Specify the key identifier to use with the ntpq utility.
-#controlkey 8
-
-# Enable writing of statistics records.
-#statistics clockstats cryptostats loopstats peerstats
+# Driftfile
+driftfile /var/lib/ntp/drift

 # Disable the monitoring facility to prevent amplification attacks using ntpdc
 # monlist command when default restrict does not include the noquery flag. See

changed: [ansible-role-ntp]

TASK [ansible-role-ntp : Make sure the NTP service is enabled and running] *****
changed: [ansible-role-ntp]

RUNNING HANDLER [ansible-role-ntp : restart ntp] *******************************
changed: [ansible-role-ntp]

PLAY RECAP *********************************************************************
ansible-role-ntp           : ok=7    changed=5    unreachable=0    failed=0

--> Idempotence test in progress (can take a few minutes)...
--> Starting Ansible Run...
Idempotence test passed.
--> Executing ansible-lint...
--> Executing flake8 on *.py files found in tests/...
--> Executing testinfra tests found in tests/...
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /home/torresgi/tmp/ansible-role-ntp, inifile:
plugins: testinfra-1.5.4
collected 4 itemss

tests/test_default.py ....

============================ pytest-warning summary ============================
WP1 None Module already imported so can not be re-written: testinfra
================= 4 passed, 1 pytest-warnings in 3.12 seconds ==================
--> Destroying instances...
==> ansible-role-ntp: Removing domain...

Looking at the above output, Molecule does a few things after applying the role under test:

  • run the playbook a second time to check for idempotence
  • runs ansible-lint against the role
  • runs flake8 against tests

These checks encourage good coding practices. Idempotence is very important for creating stable and predictable roles. The tests ensure that our role works as expected after making continuous improvements.

If you want to test a different platform configured in molecule.yml, use the --platform option to specify a platform:

molecule test --platform debian8

Tips

Disable NFS synced folder

I've noticed that some Vagrant boxes will prompt for a password to enable NFS on the host for sharing with the virtual machines. You may see a similar error as the following:

==> ansible-role-ntp: Mounting NFS shared folders...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

set -e
mkdir -p /vagrant
mount -o vers=4 192.168.121.1:/home/torresgi/tmp/ansible-role-ntp /vagrant
if command -v /sbin/init && /sbin/init --version | grep upstart; then
  /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=/vagrant
fi


Stdout from the command:



Stderr from the command:

stdin: is not a tty
mount.nfs: access denied by server while mounting 192.168.121.1:/home/torresgi/tmp/ansible-role-ntp

ERROR: Command '['/usr/bin/vagrant', 'up', '--no-provision']' returned non-zero exit status 1

To workaround this and bypass the need for NFS, which is not needed for this guide, add the following snippet to your local Vagrantfile, ~/.vagrant.d/Vagrantfile:

Vagrant.configure('2') do |config|
  config.vm.synced_folder '.', '/vagrant', disabled: true
end

Taming the Password Prompt

When Molecule calls Vagrant, it may prompt several times for your sudo password to create the virtual machine. To avoid this, you can add your user to the libvirt group:

usermod -aG libvirt giovanni

Note: Be careful when doing this. This is probably okay on your local workstation, but not suitable on a production hypervisor.

Conclusion

Using a combination of Molecule, TestInfra, Vagrant, Ansible and Git, it becomes easier to write testable roles and version control the roles alongside the tests in the same repository.


Comments

comments powered by Disqus