OpenStack Ansible EE operator

An operator to deploy and run an OpenStack Ansible Execution Environment container on Openshift

Build and deploy

It uses operator-sdk to build and run.

To build and push to a container registry

make docker-build docker-push IMG="<your image name>"

To deploy in to the cluster

make deploy IMG="<your image name>"

To undeploy it from the cluster

make undeploy

Use

Once the operator has been deployed succesfully to the openshift/kubernetes cluster, you can see it in action by creating a new "ansibleee" CR.

There are some examples on the examples directory.

The first one is openstack-ansibleee-playbook-local.yaml. This wil execute locally an example playbook, provided inline, which will print an "hello world" message using ansible debug module.

oc apply -f examples/openstack-ansibleee-playbook-local.yaml

There are other examples that also execute locally the playbook "test.yaml", but that serve as extraMounts demonstration: openstack-ansibleee-extravolumes.yaml and openstack-ansibleee-extravolumes_2_secret.yaml that need the secrets ceph-secret-example.yaml and ceph-secret-example2.yaml created:

oc apply -f ceph-secret-example.yaml
oc apply -f ceph-secret-example2.yaml
oc apply -f examples/openstack-ansibleee-extravolumes.yaml

There are also a number of examples that feature remote execution. By default, all of them expect a compute node to be available in 10.0.0.4, adjust the inventory accordingly for your environment.

The first remote example is openstack-ansibleee-playbook.yaml. This runs one of the standalone playbooks that is included in the default image.

To access an external node, you need to provide the ssh private key so ansible can connect to the node. This is being expected to be provided by a "ssh-key-secret" Secret with this format:

apiVersion: v1
kind: Secret
metadata:
  name: ssh-key-secret
  namespace: openstack
data:
  ssh-privatekey:  3390 bytes                                                                                       │
  ssh-publickey:   750 bytes

Once the key has been created, the CR should run the deploy-edpm-os-configure.yml playbook on the external node:

oc apply -f examples/openstack-ansibleee-playbook.yaml

The second remote example is openstack-ansibleee-role.yaml, which will run a certain number of tasks from specific standalone roles:

oc apply -f examples/openstack-ansibleee-role.yaml

And the last remote example is ansibleee-play.yaml, which will run a CR-defined playbook using an inventory stored in a ConfigMap.

oc apply -f examples/inventory-configmap.yaml
oc apply -f examples/openstack-ansibleee-play.yaml

Example Development Cycle

The following has been verified on openshift-local.

Create the CRD managed by the operator. This must be deleted and re-created any time the api changes.

oc create -f config/crd/bases/ansibleee.openstack.org_openstackansibleees.yaml

Build and run a local copy of the OpenStack Ansible Execution Environment operator.

make generate
make manifests
make build
./bin/manager

Once the operator is running, create the examle CR to run the test playbook.

oc create -f examples/openstack-ansibleee-playbook-local.yaml

The operator will create an OpenStackAnsibleEE resource which will create a job. That job will spawn a pod using an ansible runner that will execute the given playbook. When the playbook is done, the pod will move to a Completed state.

You can get the OpenStackAnsibleEE resource created with the oc get openstackansibleee (or osaee) command.

$ oc get osaee
NAME                       NETWORKATTACHMENTS   STATUS   MESSAGE
ansibleee-playbook-local                        True     Job completed

After knowing the name of the OpenStackAnsibleEE resource, you can search for its pod by filtering the openstackansibleee_cr label.

$ oc get pods -l openstackansibleee_cr=ansibleee-playbook-local
NAME                             READY   STATUS      RESTARTS   AGE
ansibleee-playbook-local-lbl6c   0/1     Completed   0          44s

To see the result of the playbook run, use oc logs.

oc logs -l openstackansibleee_cr=ansibleee-playbook-local
No config file found; using defaults

PLAY [Print hello world] *******************************************************

TASK [Gathering Facts] *********************************************************
Wednesday 12 July 2023  15:43:22 +0000 (0:00:00.022)       0:00:00.022 ********
ok: [localhost]

TASK [Using debug statement] ***************************************************
Wednesday 12 July 2023  15:43:23 +0000 (0:00:01.320)       0:00:01.342 ********
ok: [localhost] => {
    "msg": "Hello, world this is openstack-ansibleee-play.yaml"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
Wednesday 12 July 2023  15:43:24 +0000 (0:00:00.077)       0:00:01.420 ********
===============================================================================
Gathering Facts --------------------------------------------------------- 1.32s
Using debug statement --------------------------------------------------- 0.08s

Using openstack-ansibleee-operator with EDPM Ansible

When the openstack-ansibleee-operator spawns a job ansible execution environment crafted image can use playbooks and roles contained in its image.

An openstack-ansibleee-runner image is hosted at quay.io/openstack-k8s-operators/openstack-ansibleee-runner which contains edpm-ansible. The following commands may be used to inspect the content.

podman pull quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest
IMAGE_ID=$(podman images --filter reference=openstack-ansibleee-runner:latest --format "{{.Id}}")
podman run $IMAGE_ID ls -l

The container is built by a github action from a Dockerfile in the edpm-ansible repository.

Persistent logs

For enabling persistent logging, you need to mount /runner/artifacts into a persistent Volume through extraMounts field.

Example:
apiVersion: ansibleee.openstack.org/v1beta1
kind: OpenStackAnsibleEE
metadata:
  name: ansibleee-logs
  namespace: openstack
spec:
  playbook: "test.yaml"
  image: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest
  inventory: |
          all:
            hosts:
              localhost
  extraMounts:
    - extraVolType: Logs
      volumes:
      - name: ansible-logs
        persistentVolumeClaim:
          claimName: openstack-ansible-logs
      mounts:
      - name: ansible-logs
        mountPath: "/runner/artifacts"

Logging of additional inventories

Presence of additional inventory files, static or scripts, is reported with message:

additional inventory <MOUNTNAME> mounted

Conflicts caused by mounting multiple inventories in a single location are reported as:

inventory mount <MOUNTNAME> overrides existing inventory location

Providing multiple inventories

AnsibleEE runner is capable of accepting and combining multiple inventory sources including files and scripts.

Openstack-AnsibleEE-operator allows for provision of multiple inventories through the extraMounts field. Any file mounted, directly or indirecly, under the /runner/inventory/ path will be used to create a combinded inventory for the job.

Further information about AnsibleEE inventory processing is provided by core AnsibleEE docs.

Example:
apiVersion: ansibleee.openstack.org/v1beta1
kind: OpenStackAnsibleEE
metadata:
  name: ansibleee-logs
  namespace: openstack
spec:
  playbook: "test.yaml"
  image: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest
  inventory: |
          all:
            hosts:
              localhost
  extraMounts:
    - volumes:
      - name: ansible-secondary-inventory
        persistentVolumeClaim:
          claimName: openstack-ansible-secondary-inventory
      mounts:
      - name: ansible-secondary-inventory
        mountPath: "/runner/inventory/sub_inventory.sh"

Custom Resources

Sub Resources

Config

Config is a specification of where to mount a certain ConfigMap object

Field Description Scheme Required

name

Name is the name of the ConfigMap that we want to mount

string

true

mountpath

MountPoint is the directory of the container where the ConfigMap will be mounted

string

true

Back to Custom Resources

OpenStackAnsibleEE

OpenStackAnsibleEE is the Schema for the openstackansibleees API

Field Description Scheme Required

metadata

metav1.ObjectMeta

false

spec

OpenStackAnsibleEESpec

false

status

OpenStackAnsibleEEStatus

false

Back to Custom Resources

OpenStackAnsibleEEList

OpenStackAnsibleEEList contains a list of OpenStackAnsibleEE

Field Description Scheme Required

metadata

metav1.ListMeta

false

items

[]OpenStackAnsibleEE

true

Back to Custom Resources

OpenStackAnsibleEESpec

OpenStackAnsibleEESpec defines the desired state of OpenStackAnsibleEE

Field Description Scheme Required

playbookContents

PlaybookContents is an inline playbook contents that ansible will run on execution.

string

false

playbook

Playbook is the playbook that ansible will run on this execution, accepts path or FQN from collection

string

false

image

Image is the container image that will execute the ansible command

string

false

args

Args are the command plus the playbook executed by the image. If args is passed, Playbook is ignored.

[]string

false

name

Name is the name of the internal container inside the pod

string

false

envConfigMapName

EnvConfigMapName is the name of the k8s config map that contains the ansible env variables

string

false

env

Env is a list containing the environment variables to pass to the pod

[]corev1.EnvVar

false

restartPolicy

RestartPolicy is the policy applied to the Job on whether it needs to restart the Pod. It can be "OnFailure" or "Never". RestartPolicy default: Never

string

false

preserveJobs

PreserveJobs - do not delete jobs after they finished e.g. to check logs PreserveJobs default: true

bool

false

uid

UID is the userid that will be used to run the container.

int64

false

inventory

Inventory is the primary inventory that the ansible playbook will use to launch the job. Further inventories may be provided as ExtraMount in the /runner/inventory/ path.

string

false

extraMounts

ExtraMounts containing conf files, credentials and inventories

[]storage.VolMounts

false

backoffLimit

BackoffLimit allows to define the maximum number of retried executions (defaults to 6).

*int32

false

networkAttachments

NetworkAttachments is a list of NetworkAttachment resource names to expose the services to the given network

[]string

false

cmdLine

CmdLine is the command line passed to ansible-runner

string

false

initContainers

InitContainers allows the passing of an array of containers that will be executed before the ansibleee execution itself

[]corev1.Container

false

serviceAccountName

ServiceAccountName allows to specify what ServiceAccountName do we want the ansible execution run with. Without specifying, it will run with default serviceaccount

string

false

dnsConfig

DNSConfig allows to specify custom dnsservers and search domains

*corev1.PodDNSConfig

false

extraVars

Extra vars to be passed to ansible process during execution. This can be used to override default values in plays.

map[string]json.RawMessage

false

Back to Custom Resources

OpenStackAnsibleEEStatus

OpenStackAnsibleEEStatus defines the observed state of OpenStackAnsibleEE

Field Description Scheme Required

hash

Map of hashes to track e.g. job status

map[string]string

false

conditions

Conditions

condition.Conditions

false

networkAttachments

NetworkAttachments status of the deployment pods

map[string][]string

false

JobStatus

JobStatus status of the executed job (Pending/Running/Succeeded/Failed)

string

false

observedGeneration

ObservedGeneration - the most recent generation observed for this service. If the observed generation is less than the spec generation, then the controller has not processed the latest changes injected by the opentack-operator in the top-level CR (e.g. the ContainerImage)

int64

false

Back to Custom Resources