“High Availability” AWX Ansible

I recently did a two part video section on HA AWX Ansible. Check out part 1 here if you haven’t seen it.

In this post I’ll be sharing some of the more intricate details of the things I did in the video.

To start off I need to clear up the statement of “High Availability”. This is not an HA solution but rather a “reasonably resilient to failure” solution.

In my lab I have infrastructure in two different data centers on opposite sides of the country. Each site is protected by a firewall with an IPSEC VPN tunnel between them.

For this all to work, I’ll need to have an external postgres database server, separate from AWX host.

To make things easy I’ll be using Jujucharms and Metal as a Service to do provisioning of the infrastructure and bringing postgres to life.

To make sure that the database machines get placed in opposite datacenters, I need to add a ‘tag’ of awx_db to a machine in US East inside of MAAS and then instruct JuJu to build a postgresql instance on an Ubuntu 18.04 server, using the tag as a constraint. Once the primary server is provisioned, I’ll add the same tag to a second machine in US West and then “scale” or add another postgresql unit to the model.

root@maas-region-ctl:~# juju add-model ha-awx-ansible
root@maas-region-ctl:~# juju deploy postgresql awx-postgres-db --series bionic --constraints tags=awx_db
root@maas-region-ctl:~# juju add-unit awx-postgres-db 

Once the database servers are up I’ll use the JuJu GUI to make a small change to the configuration setting extra_pg_auth option which will allow external applications to connect to the postgres server. I set the string to
host all all 0.0.0.0/0 md5

Once this is done I’ll ssh to both units, switch to the postgres user and set the postgres password.

 
root@maas-region-ctl:~# juju ssh juju ssh awx-postgres-db/0
ubuntu@use-med-vm-04:~$ sudo su - postgres
postgres@use-med-vm-04:~$ psql
 psql (10.6 (Ubuntu 10.6-0ubuntu0.18.04.1))
 Type "help" for help.
 postgres=# \password
 Enter new password:
 Enter it again:
 postgres=# \q

After this I can use pgadmin to connect and create a new database for the AWX data. Click here to read Getting Started with pgadmin.

I can then instruct JuJu to build two more servers on the same model which I’ll use as the AWX hosts. The same way I provisioned the postgres instances, I’ll use a tag of awx_web to stand up the AWX Web servers and give them a friendly name of awx-web-primary and awx-web-secondary.

 
root@maas-region-ctl:~#  juju deploy ubuntu  awx-web-primary --series bionic --constraints tags=awx_web
root@maas-region-ctl:~#  juju deploy ubuntu  awx-web-secondary --series bionic --constraints tags=awx_web 

Now that I have hosts for the web tier, I need to install Ansible, the Docker engine, and finally pull down the AWX code from github. Check out the docker install instructions here.

root@maas-region-ctl:~# juju ssh awx-web-primary/0
 ubuntu@usw-med-vm-03:~$ sudo apt-get update 
 ubuntu@usw-med-vm-03:~$  sudo apt-get install software-properties-common 
 ubuntu@usw-med-vm-03:~$  sudo apt-add-repository --yes --update ppa:ansible/ansible 
 ubuntu@usw-med-vm-03:~$ sudo apt-get install ansible

 ubuntu@usw-med-vm-03:~$ sudo su - 
 root@usw-med-vm-03:~# git clone https://github.com/ansible/awx.git
 root@usw-med-vm-03:~# cd awx/installer/ 

Once the AWX repository has been cloned, I open up the inventory file inside the installer folder and fill out all the required parameters. Remember to uncomment “pg_hostname=” to enable a remote postgres server. If “pg_hostname=” is not set a local docker container is started for the postgres data.

To install AWX you’ll actually use ansible and declare the inventory file with all the settings and use the install.yml playbook.

root@usw-med-vm-03:~/awx/installer# ansible-playbook -i inventory install.yml

After the playbook run completes successfully there should be four docker containers running.

At this point, I have a working copy of AWX. I repeat this process for the second server and point that AWX configuration to the live postgres server as well.

To move the database from US East to US West, I simply instruct JuJu to destroy the live master server. This will release the machine back into the MAAS pool as well as promote the Replica server to the Live Master.

 
root@maas-region-ctl:~# juju remove-unit awx-postgres-db/0

Once the database is live in US West I simply ssh into the AWX web servers and edit the inventory file again making sure to point “pg_hostname=” to the IP address of the live postgres server in the West and then run the installer again

root@usw-med-vm-03:~/awx/installer# ansible-playbook -i inventory install.yml