CRs Used By the test-operator

This file contains definitions of all custom resources (CRs) that are accepted by the test-operator. Currently, there are four CRs accepted by the test-operator:

• Tempest Custom Resource

• Tobiko Custom Resource

• HorizonTest Custom Resource

• AnsibleTest Custom Resource

Tempest Custom Resource

---
apiVersion: test.openstack.org/v1beta1
kind: Tempest
metadata:
  name: tempest-tests
  namespace: openstack
spec:
  containerImage: ""
  # storageClass: local-storage
  # parallel: false
  # debug: false

  # configOverwrite
  # ---------------
  # An interface to overwrite default config files like e.g. logging.conf But can also
  # be used to add additional files. Those get added to the service config dir in
  # /etc/test_operator/<file>
  #
  # configOverwrite:
  #   file.txt: |
  #     content of the file

  # SSHKeySecretName
  # ----------------
  # SSHKeySecretName is the name of the k8s secret that contains an ssh key. The key is
  # mounted to ~/.ssh/id_ecdsa in the tempest pod. Note, the test-operator looks for
  # the private key in ssh-privatekey field of the secret.
  #
  # SSHKeySecretName: secret_name

  # Privileged
  # ----------
  # Use with caution! This parameter specifies whether test-operator should spawn test
  # pods with allowedPrivilegedEscalation: true and the default capabilities on
  # top of capabilities that are usually needed by the test pods (NET_ADMIN, NET_RAW).
  # This parameter is deemed insecure but it is needed for certain test-operator
  # functionalities to work properly (e.g.: extraRPMs in Tempest CR, or certain set
  # tobiko tests).
  #
  # privileged: false
  tempestRun:
    # NOTE: All parameters have default values (use only when you want to override
    #       the default behaviour)
    includeList: | # <-- Use | to preserve \n
      tempest.api.identity.v3.*
    concurrency: 8
    # excludeList: | # <-- Use | to preserve \n
    #   tempest.api.identity.v3.*
    # expectedFailuresList: | # <-- Use | to preserve \n
    #   tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token
    # workerFile: | # <-- Use | to preserve \n
    #   - worker:
    #     - tempest.api.*
    #     - neutron_tempest_tests
    #   - worker:
    #     - tempest.scenario.*
    # smoke: false
    # serial: false
    # parallel: true
    # externalPlugin:
    #   - repository: "https://opendev.org/openstack/barbican-tempest-plugin.git"
    #   - repository: "https://opendev.org/openstack/neutron-tempest-plugin.git"
    #     changeRepository: "https://review.opendev.org/openstack/neutron-tempest-plugin"
    #     changeRefspec: "refs/changes/97/896397/2"
    # extraImages:
    #   - URL: https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img
    #     name: cirros-0.6.2-test-operator
    #       flavor:
    #         name: cirros-0.6.2-test-operator-flavor
    #         RAM: 512
    #         disk: 20
    #         vcpus: 1

    # extraRPMs:
    # ----------
    # A list of URLs that point to RPMs that should be installed before
    # the execution of tempest. WARNING! This parameter has no efect when used
    # in combination with externalPlugin parameter.
    # extraRPMs:
    #   - https://cbs.centos.org/kojifiles/packages/python-sshtunnel/0.4.0/12.el9s/noarch/python3-sshtunnel-0.4.0-12.el9s.noarch.rpm
    #   - https://cbs.centos.org/kojifiles/packages/python-whitebox-tests-tempest/0.0.3/0.1.766ff04git.el9s/noarch/python3-whitebox-tests-tempest-0.0.3-0.1.766ff04git.el9s.noarch.rpm

  tempestconfRun:
    # NOTE: All parameters have default values (use only when you want to override
    #       the default behaviour)
    # create: true
    # collectTiming: false
    # insecure: false
    # noDefaultDeployer: false
    # debug: false
    # verbose: false
    # nonAdmin: false
    # retryImage: false
    # convertToRaw: false
    # out: ./etc/tempest.conf
    # flavorMinMem: 128
    # flavorMinDisk: 1
    # timeout: 600
    # imageDiskFormat: qcow2
    # image: https://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img

    # The following text will be mounted to the tempest pod
    # as /etc/test_operator/deployer_input.yaml
    # deployerInput: |
    #   [section]
    #   value1 = exmaple_value2
    #   value2 = example_value2

    # The following text will be mounted to the tempest pod
    # as /etc/test_operator/accounts.yaml
    # testAccounts: |
    #  - username: 'multi_role_user'
    #    tenant_name: 'test_tenant_42'
    #    password: 'test_password'
    #     roles:
    #       - 'fun_role'
    #       - 'not_an_admin'
    #       - 'an_admin'

    # The following text will be mounted to the tempest pod
    # as /etc/test_operator/profile.yaml
    # profile: |
    #   collect_timing: false
    #   create: false
    #   create_accounts_file: null

    # createAccountsFile: /path/to/accounts.yaml
    # generateProfile: /path/to/profile.yaml
    # networkID:
    # append: |  # <-- Use | to preserve \n
    #   section1.name1 value1
    #   section1.name1 value2
    # remove: |  # <-- Use | to preserve \n
    #   section1.name1 value1
    #   section1.name1 value2
    # overrides: | # <-- Use | to preserve \n
    #  overrides_section1.name1 value1
    #  overrides_section1.name1 value2

  # Workflow
  # --------
  # Workflow section can be utilized to spawn multiple test pods at the same time.
  # The commented out example spawns two test pods that are executed sequentially.
  # Each step inherits all configuration that is specified outside of the workflow
  # field. For each step you can overwrite values specified in the tempestRun and
  # tempestconfRun sections.
  #
  # workflow:
  #   - stepName: firstStep
  #     tempestRun:
  #       includeList: |
  #         tempest.api.*
  #   - stepName: secondStep
  #     tempestRun:
  #       includeList: |
  #         neutron_tempest_plugin.*

Tobiko Custom Resource

---
apiVersion: test.openstack.org/v1beta1
kind: Tobiko
metadata:
  name: tobiko-tests
  namespace: openstack
spec:
  containerImage: ""

  # Privileged
  # ----------
  #
  # Use with caution! This parameter specifies whether test-operator should spawn test
  # pods with allowedPrivilegedEscalation: true and the default capabilities on
  # top of capabilities that are usually needed by the test pods (NET_ADMIN, NET_RAW).
  # This parameter is deemed insecure but it is needed for certain test-operator
  # functionalities to work properly (e.g.: extraRPMs in Tempest CR, or certain set
  # of tobiko tests).
  #
  # privileged: false

  # storageClass: local-storage
  # parallel: false
  # debug: false
  # privateKey: |
  #   <private-key-value>
  # publicKey: |
  #   <public-key-value>
  # numProcesses: <integer>
  # preventCreate: <true/false>
  # pytestAddopts: >- # example:
  #   -m <markers>
  #   --maxfail <max-failed-tests>
  #   --skipregex <regex>
  testenv: py3
  version: master
  config: |
    [DEFAULT]
    log_file = value

    [testcase]
    timeout = value
    test_runner_timeout = value

    [ubuntu]
    interface_name = value

    [keystone]
    interface_name = value
  workflow:
    - stepName: unit-tests
      testenv: py3

    - stepName: functional-tests
      testenv: functional

HorizonTest Custom Resource

apiVersion: test.openstack.org/v1beta1
kind: HorizonTest
metadata:
  labels:
    app.kubernetes.io/name: horizontest
    app.kubernetes.io/instance: horizontest-sample
    app.kubernetes.io/part-of: test-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: test-operator
  name: horizontest-sample
spec:
  containerImage: ""
  # debug: false
  storageClass: "local-storage"

  # OpenStack admin credentials
  adminUsername: "admin"
  adminPassword: "12345678"

  # The URL of the Horizon dashboard
  dashboardUrl: "https://horizon-openstack.apps.ocp.openstack.lab/"

  # The OpenStack authentication URL
  authUrl: "https://keystone-public-openstack.apps.ocp.openstack.lab"

  # The Horizon repository URL
  repoUrl: "https://review.opendev.org/openstack/horizon"

  # The branch of the Horizon repository to checkout
  horizonRepoBranch: "master"

  # The URL to download the Cirros image (optional)
  imageUrl: "http://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img"

  # The name of the OpenStack project for Horizon tests (optional)
  projectName: "horizontest"

  # The username under which Horizon tests will run (optional)
  user: "horizontest"

  # The password for the user running the Horizon tests (optional)
  password: "horizontest"

  # The name of the OpenStack flavor to create for Horizon tests (optional)
  flavorName: "m1.tiny"

  # The name of the directory to store test logs (optional)
  logsDirectoryName: "horizon"

  # The directory path for Horizon tests (optional)
  horizonTestDir: "/var/lib/horizontest"

  # Whether to run tests in parallel (optional)
  parallel: false

  # The maximum number of retry executions (optional)
  backoffLimit: 0

  # Privileged
  # ----------
  #
  # Use with caution! This parameter specifies whether test-operator should spawn test
  # pods with allowedPrivilegedEscalation: true and the default capabilities on
  # top of capabilities that are usually needed by the test pods (NET_ADMIN, NET_RAW).
  # This parameter is deemed insecure but it is needed for certain test-operator
  # functionalities to work properly (e.g.: extraRPMs in Tempest CR, or certain set
  # of tobiko tests).
  #
  # privileged: false

AnsibleTest Custom Resource

---
apiVersion: test.openstack.org/v1beta1
kind: AnsibleTest
metadata:
  name: performance-tests
  namespace: openstack
spec:
  extraMounts:
    - name: some-configmap
      subPath: this.conf
      mountPath: /var/conf
  debug: true
  storageClass: "local-storage"
  workloadSSHKeySecretName: 'open-ssh-keys'
  ansiblePlaybookPath: playbooks/my_playbook.yaml
  ansibleGitRepo: https://github.com/myansible/project
  # containerImage:
  ansibleInventory: |
    localhost ansible_connection=local ansible_python_interpreter=python3
  ansibleVarFiles: |
    ---
    # Use exist cloud resources
    somevar: somevalue

  # Privileged
  # ----------
  #
  # Use with caution! This parameter specifies whether test-operator should spawn test
  # pods with allowedPrivilegedEscalation: true and the default capabilities on
  # top of capabilities that are usually needed by the test pods (NET_ADMIN, NET_RAW).
  # This parameter is deemed insecure but it is needed for certain test-operator
  # functionalities to work properly (e.g.: extraRPMs in Tempest CR, or certain set
  # of tobiko tests).
  #
  # privileged: false
  workflow:
    - stepName: beststep
      ansibleExtraVars: ' -e manual_run=false '
    - stepName: laststep
      ansibleExtraVars: ' -e manual_run=false '

Parallel Execution

By default, test-operator runs pods serially. The reason for this is to prevent collisions between tests (e.g., two tests from two different frameworks modifying the same resource). So, for example, if you run the following two commands:

oc apply -f crd1.yaml
oc apply -f crd2.yaml

where crd1.yaml and crd2.yaml are, let’s say, two Tobiko CRs, then you will first see a test pod executing tests defined by crd1.yaml. After the test pod finishes, you will see a second test pod that will be executing tests specified in crd2.yaml.

If you want to run multiple test pods at the same time, then you need to set parallel: true in the Spec section in both crd1.yaml and crd2.yaml:

---
apiVersion: test.openstack.org/v1beta1
kind: Tobiko
metadata:
  name: crd1
  namespace: openstack
spec:
  parallel: true # <-- HERE
  testenv: py3
---
apiVersion: test.openstack.org/v1beta1
kind: Tobiko
metadata:
  name: crd1
  namespace: openstack
spec:
  parallel: true # <-- HERE
  testenv: py3

Workflow Section

The workflow section enables the spawning of multiple test pods at the same time. For example, in the Tempest CR shown below, two test pods are spawned, each corresponding to a different step. Each step inherits a configuration that is specified outside the workflow section. In individual steps, you can overwrite values specified in the tempestRun and tempestconfRun sections.

---
apiVersion: test.openstack.org/v1beta1
kind: Tempest
metadata:
  name: tempest-tests
  namespace: openstack
spec:
  containerImage: ""
  # parallel: true # <-- Uncomment for parallel execution
  tempestRun:
  includeList: |
    tempest.api.identity.v3.*
  concurrency: 8
  tempestconfRun:
  workflow:
    - stepName: first-step
      tempestRun:
        includeList: |
          tempest.api.*
    - stepName: second-step
      tempestRun:
        includeList: |
          neutron_tempest_plugin.*

By default, test pods are executed sequentially. To enable parallel execution of test pods, you need to set parallel: true in the spec section.

CRs that can use the workflow section:

• Tempest Custom Resource

• Tobiko Custom Resource

• AnsibleTest Custom Resource