Operating loop results with dict or array

The scenario is that we have two different loops, and we want to operate their result together with a with_nested. So, here we have two loops:

- name: Get home files with names from 1 to 4 
  shell: "ls /home/ansible/{{ item }}"
  ignore_errors: yes
  register: files_1_4
  with_items:
    - 1
    - 2
    - 3
    - 4 #doesn't exist to do the example more interesting

- name: Get home files with names from 5 to 8
  shell: "ls /home/ansible/{{ item }}"
  ignore_errors: yes
  register: files_5_8
  with_items:
    - 5
    - 6
    - 7
    - 8

If you want to operate this together you will need to create a dict (or array) in order to be able to operate them.

So, what we should do is use set_fact to create this dict:

- name: Creating the dict with files that exists
  set_fact:
    existing_files_1_4: "{{ existing_files_1_4|default([]) + [ {'name': item.item } ] }}"
  when: ( item.rc == 0 ) # Just as example
  with_items:
    - "{{ files_1_4.results }}"

- name: Creating the dict with files that exists
  set_fact: 
    existing_files_5_8: "{{ existing_files_5_8|default([]) + [ {'name': item.item } ] }}"
  when: ( item.rc == 0 ) # Just as example
  with_items:
    - "{{ files_5_8.results }}"

And now we’re able to use with_nested with both items. Take into account that we were saving item.item, but you could save item.stdout_lines for example.

- name: Echo result
  shell: "echo {{ item.0.name }} {{ item.1.name }}"
  with_nested:
    - "{{ existing_files_1_4 }}"
    - "{{ existing_files_5_8 }}"
  ignore_errors: yes

And if you check the result you will see something like:

TASK [Echo result] *************************************************************
changed: [server] => (item=[{'name': 1}, {'name': 5}])
changed: [server] => (item=[{'name': 1}, {'name': 6}])
changed: [server] => (item=[{'name': 1}, {'name': 7}])
changed: [server] => (item=[{'name': 2}, {'name': 5}])
changed: [server] => (item=[{'name': 2}, {'name': 6}])
changed: [server] => (item=[{'name': 2}, {'name': 7}])
changed: [server] => (item=[{'name': 3}, {'name': 5}])
changed: [server] => (item=[{'name': 3}, {'name': 6}])
changed: [server] => (item=[{'name': 3}, {'name': 7}])

If instead of using a dict, you want to use an array, you should set_fact as follows

- set_fact:
    existing_files_1_4: "{{ existing_files_1_4|default({}) | combine ( { item.item: item.stdout } ) }}"
  when: (item.rc == 0)
  with_items:
    - "{{ files_1_4.results }}"

Debug (properly) data registered with a loop

If you take a look at official documentation of Ansible to learn how to use register with a loop, you will see that the data that is registered is not really usable when debugging.

Example:

- shell: echo "{{ item }}"
  with_items:
    - one
    - two
  register: echo

If you want to show only the ‘stdout_lines’ of this task you will need to do:

- debug:
  msg: "{{ echo.results|map(attribute='stdout_lines')|list }}"

The output won’t be perfect, but for sure will be more usable:

ok: [server] => {
 "msg": [
     [
         "one"
     ],
     [
         "two"
     ]
   ]
}

Using tags for each role

I normally put a tags for each role, so it makes me easier to choose what I need to do for my hosts with only one tag. Then if I need to specify something inside the role, I can specify a second tag and apply both tags easily.

So for example I have here frontend.yml playbook with two roles. Each role deploys one application and each of them has its own tag name:

- name: "Deploy of frontend applications: frontend1, frontend2"
  hosts: frontend
  roles:
    - { role: frontend1,
        tags: frontend1 }
    - { role: frontend2, 
        tags: frontend2 }

So then I can easily choose what I want to do if I run:

ansible-playbook -i inventory/prod frontend.yml -t frontend1
ansible-playbook -i inventory/prod frontend.yml -t frontend2

Then, imagine that inside each playbook, as part of the deployment process, I stop and start my app. Then I can add a tag called restart for that specific tasks. For example inside roles/frontend1/tasks/main.yml:

[...]
- name: Stop {{ service_name }}
  shell: /etc/init.d/{{ service_name }} stop
  tags:
    - restart

- name: Start {{ service_name }}
  shell: /etc/init.d/{{ service_name }} start"
  tags:
    - restart
[...]

Then, you could think that in order to restart the app you could run:

ansible-playbook -i inventory/prod frontend.yml -t frontend1,restart

Sadly you can’t do that because it will restart also frontend2 app. In order to fix that you should add –skip-tags:

ansible-playbook -i inventory/prod frontend.yml -t frontend1,restart --skip-tags frontend2

I know that it doesn’t look really nice, but that’s the best way I’ve found to do it right now. Also I recommend to use an Operation Tool like Rundeck to perform these kind of tasks.

How to register the result of a mysql query with Ansible

When you run a query on mysql you normally have a output like:

+-----------+
| USERNAME  |
+-----------+
| rcastells |
+-----------+
1 row in set (0.00 sec)

If you need to register only ‘rcastells’, how can you register the result of query above with ansible? You need to add following args to your mysql query:

  • -N, –skip-column-names: Don’t write column names in results.
  • -s, –silent: Be more silent. Print results with a tab as separator, each row on new line.

So your task will look like:

- name:  Get username
  command: mysql -u {{ username }} -p {{ mysql_root_password }} {{ database }} -Ns -e "<your_query>"
  register: username

- debug: msg="{{ username.stdout_lines }}"

 

Jinja2: How to gather_facts from other hosts

If you have this content in your inventory file:

[loadbalancers]
ha01 ansible_host=ec2-52-50-164-246.eu-west-1.compute.amazonaws.com
ha02 ansible_host=ec2-52-48-101-228.eu-west-1.compute.amazonaws.com

[webservers]
web01 ansible_host=ec2-52-50-143-215.eu-west-1.compute.amazonaws.com
web02 ansible_host=ec2-52-31-14-253.eu-west-1.compute.amazonaws.com

And you want to create a template inside your loadbalancers with information from webservers, like for example following haproxy configuration:

{% for host in groups['webservers'] %}
server {{ hostvars[host]['ansible_hostname'] }} {{ hostvars[host]['ansible_default_ipv4']['address'] }}:80
{% endfor %}

You first need to connect to webservers. No need to do anything. So, your playbook will look like:

---
# No need to run a task for webservers. This will gather their facts.
- hosts: webservers
  become: yes

- hosts: loadbalancers
  become: yes
  roles:
- haproxy-ha

This will connect first to your webservers and gather their facts that will be used in haproxy.cfg.j2 template file.

Role: HAProxy in HA with Keepalived in Ubuntu

https://github.com/rcastells/ansible

This role is created to work with:

  • 2 LoadBalancers
  • 1 or more WebServers

First of all review your inventory file and create a group for loadbalancers and another one for webservers:

[loadbalancers]
 ha01 ansible_host=ec2-52-50-164-246.eu-west-1.compute.amazonaws.com
 ha02 ansible_host=ec2-52-48-101-228.eu-west-1.compute.amazonaws.com

[webservers]
 web01 ansible_host=ec2-52-50-143-215.eu-west-1.compute.amazonaws.com
 web02 ansible_host=ec2-52-31-14-253.eu-west-1.compute.amazonaws.com

Then, take a look at host_vars directory and inside the role at default vars:

host_vars

You must define for ha01 and ha02 which one will be the master and the VIP between servers:

keepalived_interval: 2
keepalived_debug: 2
keepalived_interface: eth0
keepalived_MASTER_SLAVE: MASTER (or SLAVE)
keepalived_priority: 101 (or 100)
keepalived_virtualip: 172.31.39.157 (your VIP)

default vars

You could need to change these vars.

---
# Set default haproxy installation package if not defined at host_vars
haproxy_url: http://www.haproxy.org/download/1.6/src/haproxy-1.6.7.tar.gz
haproxy_version: 1.6.7

Finally take a look at the initial playbook (haproxy-ha.yml):

---
# HAProxy servers
- hosts: webservers
become: yes

- hosts: loadbalancers
become: yes
roles:
- haproxy-ha

We want to connect to webservers just to gather their facts and then use them in haproxy.cfg template file.

To run the playbook you must run:

ansible-playbook -i <inventory_file>  haproxy-ha.yml -k -K -u <username>

As an example if you have ssh key connection to all servers you could run:

ansible-playbook -i inventory/production haproxy-ha.yml

 

Using vars_prompt to create menu with options

Normally I prefer to reduce the probability that a user does mistakes. So sometimes is better if you ask through a menu what do you want to do when running a playbook. For example, if you’re using the same playbook to perform deployments of different applications is a good idea to create a menu with options. So you can reuse the same playbook for different actions. If you have following menu created by using vars_prompt:

vars_prompt:
 - name: "application"
   prompt: "Which application do you want to deploy?\n1- Web\n2- Wiki\n"
   private: no

You will have following prompt when running the playbook:

Which application do you want to deploy?
1- Web
2- Wiki

Then inside your playbook you can work with this condition.

For example you can define variables using set_fact:

 - set_fact: app=web
 when: ( application == "1" )
 
 - set_fact: app=wiki
 when: ( application == "2" )

Or perform actions taking into account which option you have chosen:

 - name: Build the war for website
   shell: /opt/builds/buildwebwar.sh
   when: ( application == "1" ) 


- name: Build the war for wiki
   shell: /opt/builds/buildwikiwar.sh
   when: ( application == "2" )

In that way you can be sure that user can do only one mistake: press the wrong number 🙂

Using variables in hosts declaration inside playbooks

When you declare in which hosts do you want to run a playbook you usually do it like this:

- hosts: databases

But let’s say that you want to be more flexible and you want to have some variable inside your hosts declaration. How can you do that? One possible solution is using extra vars.

If we decide to call our variable ‘env’ we should define our hosts inside the playbook like:

- hosts: "{{ env }}_master"

And then run the playbook as follows:

ansible-playbook -i inventory/databases binlogs.yml --extra-vars "env=database01"

Obviously inside your inventory file, these hosts must exist:

database01_master ansible_host=db01
database02_master ansible_host=db03
database03_master ansible_host=db05

 

Jinja2: lstrip_blocks to manage indentation

According to Jinja2 documentation you can manage whitespace and tabular indentation with lstrip_blocks and trim_blocks options:

  • trim_blocks: If this is set to True the first newline after a block is removed (block, not variable tag!). Defaults to False.
  • lstrip_blocks: If this is set to True leading spaces and tabs are stripped from the start of a line to a block. Defaults to False.

By default Ansible templates have trim_blocks true and lstrip_blocks false. If you want to manage them you can do it if you set them at the very beginning of your template file as follows:

#jinja2: lstrip_blocks: "True (or False)", trim_blocks: "True (or False)"

So, if you have following Ansible code for /etc/hosts template:

{% for host in groups['webservers'] %}
    {% if inventory_hostname in hostvars[host]['ansible_fqdn'] %}
{{ hostvars[host]['ansible_default_ipv4']['address'] }} {{ hostvars[host]['ansible_fqdn'] }} {{ hostvars[host]['inventory_hostname'] }} MYSELF
    {% else %}
{{ hostvars[host]['ansible_default_ipv4']['address'] }} {{ hostvars[host]['ansible_fqdn'] }} jcs-server{{ loop.index }} {{ hostvars[host]['inventory_hostname'] }}
    {% endif %}
{% endfor %}

By default Ansible will indent as you’re doing in your code:

     192.168.1.2 web01
               192.168.1.3 web02
               192.168.1.4 web03

But if you add at the top of your template the following code:

#jinja2: lstrip_blocks: “True”

Result will be without indentation:

192.168.1.2 web01
192.168.1.3 web02
192.168.1.4 web03

That makes you able to write more readable jinja2 code in your templates.

Please, note that you can’t delete indentation of the code that will be written in your final file. So in our case we’ve defined hostvars variables at the very beginning of our line because we wanted them there. If you indent that code, your result will be indented.

Jinja2: Using loop.index and loop.length. Examples: /etc/hosts and workers.properties

It’s really important to know how Jinja2 works if you want to create powerful templates for your playbooks. Today we’re gonna work with:

  • loop.index: The current iteration of the loop. (1 indexed)
  • loop.length:  The number of items in the sequence

Two different examples for different files: /etc/hosts and workers.properties:

/etc/hosts

We want to generate the following snip of code inside /etc/hosts:

192.168.1.2 web01 BB-WS1
192.168.1.3 web02 BB-WS2
192.168.1.4 web03 BB-WS3
192.168.1.5 web04 BB-WS4

web01, web02, web03 and web04 are hostnames, but how can we generate the second field?

Let’s say we have a group called ‘webservers’ in our inventory file. We can use loop.index if we want to create the code above:

{% for host in groups['webservers'] %}
{{ hostvars[host]['ansible_default_ipv4']['address'] }} {{ hostvars[host]['ansible_fqdn'] }} BB-WS{{ loop.index }}
{% endfor %}

workers.properties

We want to generate the following snip of code for Apache workers properties:

worker.loadbalancer.balance_workers=bb-ws1_gf,bb-ws2_gf,bb-ws3_gf,bb-ws4_gf

In that case, not only it’s important to know the loop.index value. Also we want to know if we’re in the latest item. Because the latest item it’s without comma. To do that we will use loop.index combined with loop.length:

worker.loadbalancer.balance_workers=
{% for host in groups['webservers'] %}
{% if loop.index == loop.length %}
bb-ws{{ loop.index }}_gf
{% else %}
bb-ws{{ loop.index }}_gf,
{% endif %}
{% endfor %}

As you can see if we’re in the latest item ( loop.index == loop.length ) we should write the code without comma.