summaryrefslogtreecommitdiffhomepage
path: root/ansible.html.markdown
blob: bfb1406b59aa0770cd9549d13bb29109cc499dcc (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
---
category: tool
tool: ansible
contributors:
    - ["Jakub Muszynski" , "http://github.com/sirkubax"]
    - ["Pat Myron" , "https://github.com/patmyron"]
    - ["Divay Prakash", "https://github.com/divayprakash"]
filename: LearnAnsible.txt
---

## Introduction

```yaml
---
"{{ Ansible }}" is an orchestration tool written in Python.
...
```

Ansible is (one of many) orchestration tools. It allows you to control your
environment (infrastructure and code) and automate the manual tasks.

Ansible has great integration with multiple operating systems (even Windows)
and some hardware (switches, Firewalls, etc). It has multiple tools that
integrate with the cloud providers. Almost every noteworthy cloud provider is
present in the ecosystem (AWS, Azure, Google, DigitalOcean, OVH, etc...).

But ansible is way more! It provides execution plans, an API, library, and callbacks.

### Main pros and cons

#### Pros

* It is an agent-less tools In most scenarios, it use ssh as a transport layer.
In some way you can use it as 'bash on steroids'.
* It is very easy to start. If you are familiar with ssh concept - you already
know Ansible (ALMOST).
* It executes 'as is' - other tools (salt, puppet, chef - might execute in
different scenario than you would expect)
* Documentation is at the world-class standard!
* Writing your own modules and extensions is fairly easy.
* Ansible AWX is the open source version of Ansible Tower we have been waiting
for, which provides an excellent UI.

#### Cons

* It is an agent-less tool - every agent consumes up to 16MB ram - in some
environments, it may be noticable amount.
* It is agent-less - you have to verify your environment consistency
'on-demand' - there is no built-in mechanism that would warn you about some
change automatically (this can be achieved with reasonable effort)
* Official GUI - Ansible Tower - is great but expensive.
* There is no 'small enterprise' payment plan, however Ansible AWX is the free
open source version we were all waiting for.

#### Neutral

Migration - Ansible <-> Salt is fairly easy - so if you would need an
event-driven agent environment - it would be a good choice to start quick with
Ansible, and convert to Salt when needed.

#### Some concepts

Ansible uses ssh or paramiko as a transport layer. In a way you can imagine
that you are using a ssh with API to perform your action. The simplest way is
to execute remote command in more controlled way (still using ssh).
On the other hand - in advanced scope - you can wrap Ansible (use python Ansible
code as a library) with your own Python scripts! It would act a
bit like Fabric then.

## Example

An example playbook to install apache and configure log level

```yaml
---
- hosts: apache

  vars:
      apache2_log_level: "warn"

  handlers:
  - name: restart apache
    service:
      name: apache2
      state: restarted
      enabled: True
    notify:
      - Wait for instances to listen on port 80
    become: True

  - name: reload apache
    service:
      name: apache2
      state: reloaded
    notify:
      - Wait for instances to listen on port 80
    become: True

  - name: Wait for instances to listen on port 80
    wait_for:
      state: started
      host: localhost
      port: 80
      timeout: 15
      delay: 5

  tasks:
  - name: Update cache
    apt:
      update_cache: yes
      cache_valid_time: 7200
    become: True

  - name: Install packages
    apt:
      name={{ item }}
    with_items:
      - apache2
      - logrotate
    notify:
      - restart apache
    become: True

  - name: Configure apache2 log level
    lineinfile:
      dest: /etc/apache2/apache2.conf
      line: "LogLevel {{ apache2_log_level }}"
      regexp: "^LogLevel"
    notify:
      - reload apache
    become: True
...
```

## Installation

```bash
# Universal way
$ pip install ansible

# Debian, Ubuntu
$ apt-get install ansible
```

* [Appendix A - How do I install ansible](#infrastructure-as-a-code)
* [Additional Reading.](http://docs.ansible.com/ansible/latest/intro_installation.html)

### Your first ansible command (shell execution)

```bash
# Command pings localhost (defined in default inventory: /etc/ansible/hosts)
$ ansible -m ping localhost
# You should see this output
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
```

### Shell Commands

There are few commands you should know about

* `ansible` (to run modules in CLI)
* `ansible-playbook` (to run playbooks)
* `ansible-vault` (to manage secrets)
* `ansible-galaxy` (to install roles from github/galaxy)

### Module

A program (usually python) that executes, does some work and returns proper
JSON output. This program performs specialized task/action (like manage
instances in the cloud, execute shell command). The simplest module is called
`ping` - it just returns a JSON with `pong` message.

Example of modules:

* Module: `ping` - the simplest module that is useful to verify host connectivity
* Module: `shell` - a module that executes shell command on a specified host(s).


```bash
$ ansible -m ping all
$ ansible -m shell -a 'date; whoami' localhost #hostname_or_a_group_name
```

* Module: `command` - executes a single command that will not be processed
through the shell, so variables like `$HOME` or operands like ``|` `;`` will not
work. The command module is more secure, because it will not be affected by the
user’s environment. For more complex commands - use shell module.

```bash
$ ansible -m command -a 'date; whoami' # FAILURE
$ ansible -m command -a 'date' all
$ ansible -m command -a 'whoami' all
```

* Module: `file` - performs file operations (stat, link, dir, ...)
* Module: `raw` - executes a low-down and dirty SSH command, not going through
the module subsystem (useful to install python2.7)

### Task

Execution of a single Ansible **module** is called a **task**. The simplest
module is called `ping` as you could see above.

Another example of the module that allow you to execute command remotly on
multiple resources is called `shell`. See above how you were using them already.

### Playbook

**Execution plan** written in a form of script file(s) is called **playbook**.
Playbook consist of multiple elements -
* a list (or group) of hosts that 'the play' is executed against
* `task(s)` or `role(s)` that are going to be executed
* multiple optional settings (like default variables, and way more)

Playbook script language is YAML. You can think that playbook is very advanced
CLI script that you are executing.

#### Example of the playbook

This example-playbook would execute (on all hosts defined in inventory) two tasks:
* `ping` that would return message *pong*
* `shell` that execute three commands and return the output to our terminal

```yaml
- hosts: all

  tasks:
    - name: "ping all"
      ping:

    - name: "execute a shell command"
      shell: "date; whoami; df -h;"
```

Run the playbook with the command:

```bash
$ ansible-playbook path/name_of_the_playbook.yml
```

Note: Example playbook is explained in the next chapter: 'Roles'

### More on ansible concept

### Inventory

Inventory is a set of objects or hosts, against which we are executing our
playbooks or single tasks via shell commands. For these few minutes, let's
assume that we are using the default ansible inventory (which in Debian based
system is placed in `/etc/ansible/hosts`).

```
localhost

[some_group]
hostA.mydomain.com
hostB.localdomain
1.2.3.4

[a_group_of_a_groups:children]
some_group
some_other_group
```

* [Additional Reading.](http://docs.ansible.com/ansible/latest/intro_inventory.html)

### ansible-roles (a 'template-playbooks' with right structure)

You already know that the tasks (modules) can be run via CLI. You also know the
playbooks - the execution plans of multiple tasks (with variables and logic).

A concept called `role` was introduced for parts of the code (playbooks) that
should be reusable.

**Role** is a structured way to manage your set of tasks, variables, handlers,
default settings, and way more (meta, files, templates). Roles allow reusing
the same parts of code in multiple playbooks (you can parametrize the role
'further' during its execution). Its a great way to introduce `object oriented`
management for your applications.

Role can be included in your playbook (executed via your playbook).


```yaml
- hosts: all

  tasks:
      - name: "ping all"
        ping:
      - name: "execute a shell command"
        shell: "date; whoami; df -h;"

  roles:
      - some_role
      - { role: another_role, some_variable: 'learnxiny', tags: ['my_tag'] }

  pre_tasks:
      - name: some pre-task
        shell: echo 'this task is the last, but would be executed before roles, and before tasks'
```

#### For remaining examples we would use additional repository
This example install ansible in `virtualenv` so it is independend from a system.
You need to initialize it into your shell-context with `source environment.sh`
command.

We are going to use this repository with examples: [https://github.com/sirkubax/ansible-for-learnXinYminutes]()

```bash
$ # The following example contains a shell-prompt to indicate the venv and relative path
$ git clone git@github.com:sirkubax/ansible-for-learnXinYminutes.git
user@host:~/$ cd ansible-for-learnXinYminutes
user@host:~/ansible-for-learnXinYminutes$ source environment.sh
$
$ # First lets execute the simple_playbook.yml
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_playbook.yml
```

Run the playbook with roles example

```bash
$ source environment.sh
$ # Now we would run the above playbook with roles
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_role.yml
```

#### Role directory structure

```
roles/
   some_role/
     defaults/      # contains default variables
     files/         # for static files
     templates/     # for jinja templates
     tasks/         # tasks
     handlers/      # handlers
     vars/          # more variables (higher priority)
     meta/          # meta - package (role) info
```

#### Role Handlers
Handlers are tasks that can be triggered (notified) during execution of a
playbook, but they execute at the very end of a playbook. It is the best way to
restart a service, check if the application port is active (successful
deployment criteria), etc.

Get familiar with how you can use roles in the simple_apache_role example

```
playbooks/roles/simple_apache_role/
├── tasks
│   └── main.yml
└── templates
    └── main.yml
```

### ansible - variables

Ansible is flexible - it has 21 levels of variable precedence.
[read more](http://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable)
For now you should know that CLI variables have the top priority.
You should also know, that a nice way to pool some data is a **lookup**

### Lookups
Awesome tool to query data from various sources!!! Awesome!
query from:
* pipe  (load shell command output into variable!)
* file
* stream
* etcd
* password management tools
* url

```bash
# read playbooks/lookup.yml
# then run
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/lookup.yml
```

You can use them in CLI too

```yaml
ansible -m shell -a 'echo "{{ my_variable }}"' -e 'my_variable="{{ lookup("pipe", "date") }}"' localhost
ansible -m shell -a 'echo "{{ my_variable }}"' -e 'my_variable="{{ lookup("pipe", "hostname") }}"' all

# Or use in playbook

(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/lookup.yml
```

### Register and Conditional

#### Register

Another way to dynamically generate the variable content is the `register` command.
`Register` is also useful to store an output of a task and use its value
for executing further tasks.

```
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/register_and_when.yml
```

```yaml
---
- hosts: localhost
  tasks:
   - name: check the system capacity
     shell: df -h /
     register: root_size

   - name: debug root_size
     debug:
        msg: "{{ root_size }}"

   - name: debug root_size return code
     debug:
       msg:  "{{ root_size.rc }}"

# when: example

   - name: Print this message when return code of 'check the system capacity' was ok
     debug:
       msg:  "{{ root_size.rc }}"
     when: root_size.rc == 0
...
```

#### Conditionals - when:

You can define complex logic with Ansible and Jinja functions. Most common is
usage of `when:`, with some variable (often dynamically generated in previous
playbook steps with `register` or `lookup`)

```yaml
---
- hosts: localhost
  tasks:
   - name: check the system capacity
     shell: df -h /
     when: some_variable in 'a string'
  roles:
   - { role: mid_nagios_probe, when: allow_nagios_probes }
...
```

### ansible - tags, limit

You should know about a way to increase efficiency by this simple functionality

#### TAGS

You can tag a task, role (and its tasks), include, etc, and then run only the
tagged resources

```
ansible-playbook playbooks/simple_playbook.yml --tags=tagA,tag_other
ansible-playbook playbooks/simple_playbook.yml -t tagA,tag_other

There are special tags:
    always

--skip-tags can be used to exclude a block of code
--list-tags to list available tags
```

[Read more](http://docs.ansible.com/ansible/latest/playbooks_tags.html)

#### LIMIT

You can limit an execution of your tasks to defined hosts

```
ansible-playbook playbooks/simple_playbook.yml --limit localhost

--limit my_hostname
--limit groupname
--limit some_prefix*
--limit hostname:group #JM
```

### Templates

Templates are a powerful way to deliver some (partially) dynamic content.
Ansible uses **Jinja2** language to describe the template.

```
Some static content

{{ a_variable }}

{% for item in loop_items %}
    this line item is {{ item }}
{% endfor %}
```

Jinja may have some limitations, but it is a powerful tool that you might like.

Please examine this simple example that installs apache2 and generates
index.html from the template
"playbooks/roles/simple_apache_role/templates/index.html"

```bash
$ source environment.sh
$ # Now we would run the above playbook with roles
(venv) user@host:~/ansible-for-learnXinYminutes$ ansible-playbook playbooks/simple_role.yml --tags apache2
```

#### Jinja2 CLI

You can use the jinja in the CLI too

```bash
ansible -m shell -a 'echo {{ my_variable }}' -e 'my_variable=something, playbook_parameter=twentytwo' localhost
```

In fact - jinja is used to template parts of the playbooks too

```yaml
# check part of this playbook: playbooks/roles/sys_debug/tasks/debug_time.yml
- local_action: shell date +'%F %T'
  register: ts
  become: False
  changed_when: False

- name: Timestamp
  debug: msg="{{ ts.stdout }}"
  when: ts is defined and ts.stdout is defined
  become: False
```

#### Jinja2 filters

Jinja is powerful. It has many built-in useful functions.

```
# get first item of the list
{{ some_list | first() }}
# if variable is undefined - use default value
{{ some_variable | default('default_value') }}
```

[Read More](http://docs.ansible.com/ansible/latest/playbooks_filters.html)

### ansible-vault

To maintain **infrastructure as code** you need to store secrets. Ansible
provides a way to encrypt confidential files so you can store them in the
repository, yet the files are decrypted on-the-fly during ansible execution.

The best way to use it is to store the secret in some secure location, and
configure ansible to use during runtime.

```bash
# Try (this would fail)
$ ansible-playbook playbooks/vault_example.yml

$ echo some_very_very_long_secret > ~/.ssh/secure_located_file

# in ansible.cfg set the path to your secret file
$ vi ansible.cfg
  ansible_vault_password_file = ~/.ssh/secure_located_file

#or use env
$ export ANSIBLE_VAULT_PASSWORD_FILE=~/.ssh/secure_located_file

$ ansible-playbook playbooks/vault_example.yml

  # encrypt the file
$ ansible-vault encrypt path/somefile

  # view the file
$ ansible-vault view path/somefile

  # check the file content:
$ cat path/somefile

  # decrypt the file
$ ansible-vault decrypt path/somefile
```

### dynamic inventory

You might like to know, that you can build your inventory dynamically.
(For Ansible) inventory is just JSON with proper structure - if you can
deliver that to ansible - anything is possible.

You do not need to reinvent the wheel - there are plenty of ready to use
inventory scripts for most popular Cloud providers and a lot of in-house
popular usecases.

[AWS example](http://docs.ansible.com/ansible/latest/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script)

```bash
$ etc/inv/ec2.py --refresh
$ ansible -m ping all -i etc/inv/ec2.py
```

[Read more](http://docs.ansible.com/ansible/latest/intro_dynamic_inventory.html)

### ansible profiling - callback

Playbook execution takes some time. It is OK. First make it run, then you may
like to speed things up. Since ansible 2.x there is built-in callback for task
execution profiling.

```
vi ansible.cfg
# set this to:
callback_whitelist = profile_tasks
```

### facts-cache and ansible-cmdb

You can pull some information about your environment from another hosts.
If the information does not change - you may consider using a facts_cache
to speed things up.

```
vi ansible.cfg

# if set to a persistent type (not 'memory', for example 'redis') fact values
# from previous runs in Ansible will be stored.  This may be useful when
# wanting to use, for example, IP information from one group of servers
# without having to talk to them in the same playbook run to get their
# current IP information.
fact_caching = jsonfile
fact_caching_connection = ~/facts_cache
fact_caching_timeout = 86400
```

I like to use `jsonfile` as my backend. It allows to use another project
`ansible-cmdb` [(project on github)](https://github.com/fboender/ansible-cmdb) that generates a HTML page of your inventory
resources. A nice 'free' addition!

### Debugging ansible [chapter in progress]

When your job fails - it is good to be effective with debugging.

1. Increase verbosity by using multiple -v **[ -vvvvv]**
2. If variable is undefined -
`grep -R path_of_your_inventory -e missing_variable`
3. If variable (dictionary or a list) is undefined -
`grep -R path_of_your_inventory -e missing_variable`
4. Jinja template debug
5. Strange behaviour - try to run the code 'at the destination'

### Infrastructure as code

You already know, that ansible-vault allows you to store your confidential data
along with your code. You can go further - and define your
ansible installation and configuration as code.
See `environment.sh` to learn how to install the ansible itself inside a
`virtualenv` that is not attached to your operating system (can be changed by
non-privileged user), and as additional benefit - upgrading version of ansible
is as easy as installing new version in new virtualenv. What is more, you can
have multiple versions of Ansible present at the same time.

```bash
# recreate ansible 2.x venv
$ rm -rf venv2
$ source environment2.sh

# execute playbook
(venv2)$ ansible-playbook playbooks/ansible1.9_playbook.yml # would fail - deprecated syntax

# now lets install ansible 1.9.x next to ansible 2.x
(venv2)$ deactivate
$ source environment.1.9.sh

# execute playbook
(venv1.9)$ ansible-playbook playbooks/ansible1.9_playbook.yml # works!

# please note that you have both venv1.9 and venv2 present - you need to (de)activate one - that is all
```

#### become-user, become

In Ansible - to become `sudo` - use the `become` parameter. Use `become_user`
to specify the username.

```
- name: Ensure the httpd service is running
  service:
    name: httpd
    state: started
  become: true
```

Note: You may like to execute Ansible with `--ask-sudo-pass` or add the user to
sudoers file in order to allow non-supervised execution if you require 'admin'
privilages.

[Read more](http://docs.ansible.com/ansible/latest/become.html)

## Tips and tricks

#### --check -C

Always make sure that your playbook can execute in 'dry run' mode (--check),
and its execution is not declaring 'Changed' objects.

#### --diff -D

Diff is useful to see nice detail of the files changed.
It compare 'in memory' the files like `diff -BbruN fileA fileB`.


#### Execute hosts with 'regex'

```bash
ansible -m ping web*
```

#### Host groups can be joined, negated, etc

```bash
ansible -m ping web*:!backend:monitoring:&allow_change
```

#### Tagging

You should tag some (not all) objects - a task in a playbook, all tasks
included form a role, etc. It allows you to execute the chosen parts of the
playbook.

#### no_logs: True

You may see, that some roles print a lot of output in verbose mode. There is
also a debug module. This is the place where credentials may leak. Use `no_log`
to hide the output.

#### Debug module

allows to print a value to the screen - use it!

#### Register the output of a task

You can register the output (stdout), rc (return code), stderr of a task with
the `register` command.

#### Conditionals: when:

#### Loop: with, with\_items, with\_dict, with\_together

[Read more](http://docs.ansible.com/ansible/latest/playbooks_conditionals.html)

## Additional Resources

* [Servers For Hackers: An Ansible Tutorial](https://serversforhackers.com/c/an-ansible-tutorial)
* [A system administrator's guide to getting started with Ansible - FAST!](https://www.redhat.com/en/blog/system-administrators-guide-getting-started-ansible-fast)
* [Ansible Tower](https://www.ansible.com/products/tower) - Ansible Tower provides a web UI, dashboard and rest interface to ansible.
* [Ansible AWX](https://github.com/ansible/awx) - The Open Source version of Ansible Tower.