Run Tests via Test Operator

Note

Before you proceed with this section of the documentation, please make sure that you have read the prerequisites.

This guide describes:

If you want to get your hands on the test-operator quickly, then follow these two sections: Running Test Operator Using the Operator Lifecycle Manager (OLM) and Executing Tests.

Note

If you prefer visual guides, you can check out Test Operator Tempest Guide video.

Running Test Operator Using the Operator Lifecycle Manager (OLM)

The first option of how to start the operator is by running the pre-built operator image stored in the openstack-operator-index using the OLM.

Follow these steps to install the operator in the openstack-operators project.

  1. Create Subscription

cat subscription.yaml
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-operator
  namespace: openstack-operators
spec:
  name: test-operator
  source: openstack-operator-index
  sourceNamespace: openstack-operators

  1. Apply subscription.yaml

oc apply -f subscription.yaml

  1. Wait for the test-operator-controller-manager pod to successfully spawn. Once you see the pod running, you can start communicating with the operator using the CRs understood by the test-operator (see CRs Used By the test-operator). For more information about how to run tests via the test-operator, refer to the Executing Tests section.

oc get pods -n openstack-operators
...
test-operator-controller-manager-6c9994847c-6jwn5                 2/2     Running     0              20s
...

Running Test Operator Locally Outside the Cluster

This is a quick and easy way to experiment with the operator during development of a new feature.

ENABLE_WEBHOOKS=false make install run

Note that after running the following command, you will need to switch to another terminal unless you run it in the background.

Uninstalling Operator

If you installed the operator by following the steps in the Running Test Operator Using the Operator Lifecycle Manager (OLM) section, then this section can come in handy. You might need to uninstall the operator when:

  • you encountered issues during the installation process or when

  • you want to be sure that you are using the latest version of the operator.

Please make sure that you follow the order of the steps:

  1. Remove all instances of the CRDs supported by the test-operator (Tempest, Tobiko, …)

oc delete tempest --all
oc delete tobiko --all
oc delete horizontests --all
oc delete ansibletests --all

  1. Remove the crd

oc delete crd/tempests.test.openstack.org
oc delete crd/tobikoes.test.openstack.org
oc delete crd/ansibletests.test.openstack.org
oc delete crd/horizontests.test.openstack.org

  1. Remove the subscription you created during the installation.

oc delete subscription/test-operator -n openstack-operators

  1. Remove the csv

oc delete clusterserviceversion.operators.coreos.com/test-operator.v0.0.1 -n openstack-operators

  1. Remove test-operator related installplan (replace XXXXX with value obtained with the first command oc get installplans)

oc get installplans -n openstack-operators | grep "test-operator"
oc delete installplan install-XXXXX -n openstack-operators

  1. Remove the operator. It is possible that if you executed the previous commands too quickly, then you will need to execute this command twice.

oc delete operator/test-operator.openstack-operators

  1. Check that there are no test-operator related resources hanging. This step is not required.

oc get olm -n openstack-operators

Note

It might happen that by changing the order of the uninstallation steps, you encounter a situation when you will not be able to delete the crd. In such a case, try to delete the finalizers: section in the output of the oc edit tempest/tempest-tests.

Executing Tests

Once you have the test operator running, then you can apply a custom resource accepted by the test-operator to start the testing. Currently, four types of custom resources are being accepted by the test-operator (see CRs Used By the test-operator section):

  1. Create a manifest for custom resource accepted by the test-operator (CRs Used By the test-operator section).

  2. Apply the manifest. Either go with the default one, the command below, or replace the path with a manifest created in the first step.

oc apply -f config/samples/test_v1beta1_tempest.yaml

  1. Verify that the pod executing the tests is running. It might take a couple of seconds for the test pod to spawn. Also, note that by default, the test-operator allows only one test pod to be running at the same time (read Parallel Execution). If you defined your own custom resource in the first step, then your test pod will be named according to the name value stored in the metadata section.

oc get pods | grep tempest

You should see a pod with a name like tempest-tests-xxxxx.

  1. Investigate the stdout of the test-pod:

oc logs <name of the pod>

Read Getting Logs section if you want to see logs and artifacts produced during the testing.

Getting Logs

The test-operator creates a persistent volume that is attached to a pod executing the tests. Once the pod completes test execution, the pv contains all the artifacts associated with the test run.

Note

Please keep in mind that all resources created by the test operator are bound to the CR. Once you remove the CR (e.g.:tempest/tempest-tests), then you also remove the PV containing the logs.

If you want to retrieve the logs from the pv, you can follow these steps:

  1. Spawn a pod with the pv attached to it.

---
apiVersion: v1
kind: Pod
metadata:
  name: test-operator-logs-pod
  namespace: "openstack"
spec:
  containers:
    - name: test-operator-logs-container
      image: quay.io/quay/busybox
      command: ["/bin/sh", "-c", "--"]
      args: ["while true; do sleep 30; done;"]
      volumeMounts:
        - name: logs-volume
          mountPath: /mnt
  volumes:
    - name: logs-volume
      persistentVolumeClaim:
        # Note: In case you created your own custom resource then you
        #       have to put here the value from metadata.name.
        claimName: tempest-tests

2. Get an access to the logs by connecting to the pod created in the first step:

oc rsh pod/test-operator-logs-pod
cd /mnt

OR get an access to the logs by copying the artifacts out of the pod created in the first step:

mkdir test-operator-artifacts
oc cp test-operator-logs-pod:/mnt ./test-operator-artifacts