AAP2 and Ansible Navigator - Execution Environments

[Travis]

The ansible-navigator command was introduced when Ansible Automation Platform 2 (AAP2) was released. Ansible navigator allows the use of Execution Environments (EEs)to leverage container-based images known as Execution Environment Images (EEIs). Navigator is capable of leveraging the default container runtimes on the system to launch the EE in order to run the playbook or perform other Ansible functions and sub-commands (https://ansible.readthedocs.io/projects/navigator/) and (https://ansible.readthedocs.io/projects/navigator/installation/#install-ansible-navigator-windows). 

There is a separate configuration file for ansible-navigator which is called ansible-navigator.yml and contains the basic settings and configuration information for how it should interact with and launch the container images (EEIs). The overview of the settings can be found (https://ansible.readthedocs.io/projects/navigator/settings/) and it is possible to modify some of the container behavior with this configuration file. If there are multiple container runtimes (container engines) it is possible to specify which container engine is used by the ansible-navigator command.

   execution-environment:
     container-engine: podman

The other thing to keep in mind when using Ansible Navigator is that localhost now has a different meaning. Prior to running the Ansible playbook within an EE, the playbook was launched locally from a control node with the ansible-playbook command. Now, the playbook as well as anything else in the working directory is mounted inside the container and executed from the EE. What results from this is that any assets that would be copied and written to localhost now reside within the temporary (ephemeral) filesystem of the running container and are deleted as soon as the ansible-navigator command exits.

I have created a quick demo using a set of dummy playbooks to illustrate the differences between localhost and workstation in the RH294/DO374/DO467 classroom environments. There are two playbooks where you have the opportunity to run and observe the container as there is a wait_for module used to look for a file in a specific directory. The demo is located (https://github.com/tmichett/AnsiblePlaybooks/tree/master/AAP2/navigator) and named Localhost_Navigator_Demo.yml and Workstation_Navigator_Demo.yml. It is set up to run directly from the RH294 classroom. Changes will need to be made to the EEI in the ansible-navigator.yml file for the other courses to use the correct EEI.

When running the playbooks, you should also open another terminal window so that you can use a podman exec -it <Container_Name> /bin/bash command to open the container and look around. This will provide the most useful information. The playbook looks for a file called /tmp/navdemo on the system and expects to find DEMO COMPLETE in that file. It will wait at the Ansible task until that file exists with the correct content (similar to something else you might need in a real Ansible task).

Running the demo is easy …

Terminal 1:

[student@workstation navigator]$ ansible-navigator run Localhost_Navigator_Demo.yml

… OUTPUT OMITTED …

TASK [Display message to screen] *****************************************************
ok: [localhost] => {
    "msg": "Hello, I'm waiting on a file on localhost /tmp/navdemo. I will continue waiting until the file exists. Open a new terminal, use PODMAN to go into the container and then create the file /tmp/navdemo with contents 'DEMO COMPLETE'."
}

TASK [Wait until the string "DEMO COMPLETE" is in the file /tmp/foo before continuing] ***

Terminal 2

[student@workstation navigator]$ ls /tmp | grep navdemo

[student@workstation navigator]$ podman exec -it ansible_runner_65c393c1-0d78-4126-a0db-d759f0884041 /bin/bash
bash-4.4#

bash-4.4# ls /tmp | grep navdemo

bash-4.4# ls
Demo_Clean.sh		      Workstation_Navigator_Demo.yml  ansible.cfg
Demo_Complete.sh	      ansible-navigator.log	      inventory
Localhost_Navigator_Demo.yml  ansible-navigator.yml	      playbook.yml

bash-4.4# ./Demo_Complete.sh

Repeat the above steps for the Workstation_Navigator_Demo.yml. What should be seen is that using the Demo_Complete.sh script in the container does nothing and you must run it locally on the workstation system in order for the contents to exist in the correct directory on the correct system. Even though workstation is used for the ansible-navigator command to launch the same EEI as an EE, the playbook targets localhost for one playbook and workstation for the other playbook. 

These differences must be taken into account when refactoring older playbooks as often playbooks would be written in a way to collect items “locally” on the control node (node running the Ansible playbook command) and now, if it is set to write content to localhost, it is being written inside the running container which is on ephemeral storage and is deleted when the ansible-navigator command has completed. There are various methods to modify and update the playbooks or it is also possible to update the ansible-navigator.yml file so that persistent storage is presented in the running container and when things are written to “localhost” they are saved. More information and demos can be found (https://github.com/tmichett/do374/tree/main/Demos/Misc) with the Delegation or the Extra_Mounts directories.

[Chetan_Tiwary_]

Thanks much @Travis for the detailed write-up - it is a valuable resource for all of us here. 

[shashi01]

@Travis 

Truly impressive. Thanks a lot.

[Trevor]

This is gold!!!    And I'm still in favor of that article 
in Enable Sysadmin

[ConstantinB]

Thanks @Travis !

Probably it worth to add few things to consider when running the playbooks with ansible-navigator:

- All the files reffered in the playbooks need to be accessible from EE container.

- The collections installed on the control host via "ansible-galaxy collection install" without the "-p" might be not listed in the ansible-navigator.

[Travis]

@ConstantinB -

This is true. That is why it is recommended that playbooks and files needed are contained within the project directory with the playbooks. That way it is mounted within the execution environment that ansible-navigator launches. Keep in mind, the ansible-navigator.yml file controls ansible-navigator configurations and the ansible.cfg file contains the Ansible configurations

So for the files referenced by playbooks, they should exist within the main project directory so that they get pulled into the execution environment. It is possible to mount additional directories into the execution environment by modifying the ansible-navigator.yml file.

As for collections, you are correct ansible-galaxy collection install without the -P will install the collections at the system-level for the user and not at the project-level. If the user is a regular user, it will just install them locally for the user in their home directory. If the user is the root user, it will install them as part of the system-wide resources for all users. Again, this is something that would be considered "Best Practices" when developing playbooks. Anytime roles or collections are installed, they should be installed in a roles or collections directory (sub-directory relative to the playbooks) and the ansible.cfg file should have a reference to the roles/collections paths. Real-world, you would have these sub-directories already there and instead of arbitrary collections, you would have a requirements.yml file in each directory with a list of collections and roles to install. This sets everything up for success for using Ansible Automation Controller as it will automatically install the collections and roles based on the requirements.

Going further, if these are roles and collections you are planning on using frequently, it would be best to create a custom execution environment image that contains these roles and collections. In this instance, it wouldn't even be necessary to install the roles/collections before running the playbook with Ansible Navigator.

IMPORTANT

The ansible-navigator command launches an execution environment which is a container. This is where the ansible-playbook command as well as other Ansible resources exist including the playbook directory that gets mounted and presented in the container as well as all collections and Ansible modules that are part of the Execution Environment Image. So even though for the plain "ansible" command you might have needed to install collections, it is possible to have them already installed on an Execution Environment Image (EEI) that ansible-navigator is running.

[ConstantinB]

Thanks @Travis for very detailed description!

BTW I've noted another difference between running the playbook with "ansible-playbook" and "ansible-navigator". 

If the "remote_user" key is not set in ansible.cfg used by EE,

when running with ansible-navigator, the remote user is root (even the documentation is saying it should be run in the context of the user running the playbook).  

On the screenshoot above the "ansible" user is unprivileged on the control host and is member of sudoers and has passwordless login configured on the targed "node1".

With ansible-playbook it works with both "remote_user" values.

Not sure if it's a bug  or it's me doing something wrong.

[bonnevil]

@ConstantinBIt's not a bug.  The container has its own idea of what users exist that is separate from the control node.  If you look, the execution environment even has its own /etc/passwd file.  So when you use ansible-navigator to run a playbook as some regular user, inside the container it runs the playbook as the root user and it'll assume that remote_user should also be root unless you say otherwise.  This follows the rule that you mention: the name of the remote_user that is used by default is the same name as the local user that ran the playbook; it's just confusing because while you started the EE as a regular user, the playbook was run inside the container as root.

A playbook you can use to test this with, using ansible-playbook and ansible-navigator run:

So, yeah.  If you don't want to use root as the remote user when using ansible-navigator, you have to set it one of several possible ways to something else.

[bonnevil]

@Travis  wrote:

When running the playbooks, you should also open another terminal window so that you can use a podman exec -it <Container_Name> /bin/bash command to open the container and look around. This will provide the most useful information. The playbook looks for a file called /tmp/navdemo on the system and expects to find DEMO COMPLETE in that file. It will wait at the Ansible task until that file exists with the correct content (similar to something else you might need in a real Ansible task).

If you just want to see the inside of the container and aren't trying to track what it's doing when it's running, there's also the ansible-navigator exec command.  That will start up the automation execution environment and drop you into a shell prompt in there (assuming the EE is set up like Red Hat automation execution environments). It will pull in the current working directory and its subdirectories from the control node just like ansible-navigator run does.

One thing I just learned is that you can use this to run old fashioned ad hoc commands from inside the EE, using the ansible command provided inside the EE. For example, this works:

ansible-navigator exec -- ansible all -u yourremoteuser --ask-pass -m ansible.builtin.ping

[Travis]

@bonnevil -

Excellent point Steve. You are correct in the additional terminal at the same time to look around. Never thought to mention that. I setup the steps and I've used this demo quite often when I taught the DO374 course and AAP 2.x was really new. I had a few students with older playbooks and I had to show them what "localhost" was really doing, so I put a dummy task in the playbook that "waited" so I could show them inside the container.

Love the nugget of information for the ad-hoc commands as I still show that as a troubleshooting and verification set of steps (way way way early on) in the course. I had actually worked on my own EE (outside of Navigator and AAP2) so I could run Ansible from a Windows computer. 

I updated the project last week to get off Fedora 33/34 and use the latest Fedora as a base image. The URL is https://github.com/tmichett/AnsibleContainer and it works pretty well. It will drop you into a running container. It will work with Docker or Podman and therefore it also works on Windows and MacOS if you have Podman Desktop or Docker Desktop installed.

The additional context you've added here is great and thanks for the valuable insights. I will assimilate this and add it to the collective.