Testing an Ansible Collection - Part 2 - Testing the Collection

There are different aspects to testing a collection you intend to publish to Ansible Galaxy.  Flake8 and ansible-lint are run, and meta/runtime.yml must be present in order to successfully import your collection.  Once again, test everything you can beforehand, because there is no deleting content once accepted by Galaxy.  All these tests are run against the Morpheus core collection.

Preparing for local testing:

virtualenv ~/collection-venv
source ~/collection-venv/bin/activate
pip install flake8 ansible-lint

Flake8:

flake8 plugins/inventory/morpheus_inventory.py
plugins/inventory/morpheus_inventory.py:75:80: E501 line too long (81 > 79 characters)
plugins/inventory/morpheus_inventory.py:83:80: E501 line too long (142 > 79 characters)
plugins/inventory/morpheus_inventory.py:100:80: E501 line too long (85 > 79 characters)
plugins/inventory/morpheus_inventory.py:113:80: E501 line too long (92 > 79 characters)
plugins/inventory/morpheus_inventory.py:116:80: E501 line too long (101 > 79 characters)
plugins/inventory/morpheus_inventory.py:120:80: E501 line too long (87 > 79 characters)
plugins/inventory/morpheus_inventory.py:123:80: E501 line too long (99 > 79 characters)
plugins/inventory/morpheus_inventory.py:130:80: E501 line too long (90 > 79 characters)

Luckily for me, Ansible Galaxy uses a maximum line length of 160, so running again:

flake8 --max-line-length 160 plugins/inventory/morpheus_inventory.py
# no output

Now for ansible-lint:

ansible-lint
# no output

I wanted to make sure that it actually ran, so I added a couple consummate v's:

ansible-lint -vv
DEBUG    Logging initialized to level 10
...
DEBUG    Added role: roles/clouds (role)
DEBUG    Added role: roles/groups (role)
DEBUG    Added role: roles/instancetypes (role)
DEBUG    Added role: roles/integrations (role)
...

Both of these test give you a non-zero exit level, so you can use them in testing really easily, which I did:

name: Build Collection
on: [workflow_dispatch]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Prerequisite install
        run: |
          sudo apt remove ansible
          sudo pip install ansible ansible-lint flake8
          
      - name: Lint collection
        run: |
          cd $GITHUB_WORKSPACE
          ansible-lint
          find . -name *.py | xargs -n 1 flake8 --max-line-length 160 

Next, try building the collection:

ansible-galaxy collection build
Created collection for morpheus.core at morpheus-core-0.2.2.tar.gz

You can test installation by using:

ansible-galaxy collection install morpheus-core-0.2.2.tar.gz

Testing against whatever python and Ansible versions you happen to have installed is all fine and good, but since many people use this collection, I wanted to broaden functional testing to include a real testing target (see part 1 for that) and testing against multiple versions of python and Ansible.  I was considering running multiple docker containers and installing multiple versions of Ansible when I stumbled on the matrix arguments of GitHub actions.  It's basically a for loop to test against a matrix of different versions of targets:

  test-collection-ansible-29:
    runs-on: ubuntu-latest
    needs: build-collection
    strategy:
      matrix:
        python-version: ['2.x', '3.6', '3.7', '3.8', '3.9']
        ansible-version: ['ansible<2.10']
        
# and

  test-collection-ansible-210-up:
    runs-on: ubuntu-latest
    needs: build-collection
    strategy:
      matrix:
        python-version: ['2.x', '3.6', '3.7', '3.8', '3.9']
        ansible-version: ['ansible<2.11', 'ansible<6', 'ansible-core<2.12', ansible-core]

I put these in two separate jobs because referencing the inventory plugin is different between Ansible 2.9 and everything above.  It runs against the test Morpheus and uses jQuery to test the output with the -e flag to force an error if the output isn't found.  It also uses secrets to store the credentials and URL for the test Morpheus instance.  The entire run is here.

Once all these are done, the collection should be good to go if you want to manually upload it to Galaxy.  I still do it manually out of paranoia, but I will write a part 3 if I end up automating that part of the process.