Acceptance tests for Jenkins

End to end test suite for Jenkins automation server and its plugins.

The scenarios are described in the form of tests controlling Jenkins under test (JUT) through UI / REST APIs. Clean instances are started for individual tests to isolate the tests. The harness provides convenient docker support so integration tests can be written easily.

Contributing to acceptance tests

Follow the contributing guidelines if you want to propose new tests for this project.

Getting Started

The simplest way to start the harness is calling BROWSER=firefox JENKINS_VERSION=2.73 mvn test. The complete test suite takes hours to run due to the number of covered components/use-cases, the cost of Jenkins setup, and selenium interactions. That can be avoided by selecting a subset of tests to be run - smoke tests for instance.

Here is a walkthrough for running ATH tests on changes made to a local version of Jenkins.

Running tests

The harness provides a variety of ways to configure the execution including:

• Selecting web browser
• Specifying test(s) to run
• Managing the versions of Jenkins and plugins
• Using a http proxy
• Prelaunching Jenkins
• Selecting how to launch Jenkins
• Obtaining a report of plugins that were exercised
• Running tests in container
• Debugging tests in container
• Capture a support bundle
• Selecting tests based on plugins they cover (TODO)
• Controlling what gets tested on ci.jenkins.io

Creating tests

Given how long it takes for the suite to run, test authors are advised to focus on the most popular plugins and use-cases to maximize the value of the test suite. Tests that can or already are written as a part of core/plugin tests should be avoided here as well as tests unlikely to catch future regressions (reproducers for individual bugs, boundary condition testing, etc.). Individual maintainers are expected to update their tests reflecting core/plugin changes as well as ensuring the tests does not produce false positives. Tests identified to violate this guideline might be removed without author's notice for the sake of suite reliability.

• Selenium test in plugin repository
• Docker fixtures
• Page objects
  • Mix-ins
• Guice is our glue
• Writing tests
  • Video tutorial by Kohsuke on how to write tests
  • Writing JUnit test
• Testing agents
• Hamcrest matchers
• EC2 provider configuration
• Investigation

Areas where acceptance-tests-harness is more suitable than jenkins-test-harness are:

• Installing plugins for cross-plugin integration
• Running tests in a realistic classloader environment
• Verifying UI behaviour in an actual web browser

Controlling what gets tested on ci.jenkins.io

Every build (branch or pull request) on ci.jenkins.io runs the acceptance test suite against a matrix of:

• Jenkins version lines: lts and latest (weekly)
• JDKs: 21 for the lts line, 21 and 25 for the latest line

By default, the full matrix above is tested. On pull requests, this default can be overridden by adding one or more of the following labels, which is useful to save build resources when only a subset of the matrix is relevant to the change:

Label	Effect
weekly-test	Restricts testing to the latest (weekly) Jenkins version line.
lts-test	Restricts testing to the lts Jenkins version line.
java-$version (e.g. java-25)	Restricts testing to the given JDK version, across whichever Jenkins version line(s) are selected.

Labels can be combined. For example, adding both weekly-test and java-25 will only test the latest line on JDK 25. Adding only weekly-test will test the latest line on all of its default JDKs (21 and 25).

If none of these labels are present, the default full matrix described above is tested.

Marker files

Contributors without permission to add labels to the pull request (e.g. from a fork) can achieve the same effect by committing an empty marker file at the repository root, named after the label: weekly-test, lts-test, or java-$version (e.g. java-25; multiple java-* marker files can be added at once, same as multiple java-* labels). The build will mark itself unstable as a reminder to remove the marker file(s) (git rm <file>) before the pull request is merged.

Renovate

Renovate automatically adds the weekly-test label to the pull requests it opens, so that dependency bumps are, by default, only tested against the latest (weekly) Jenkins version line. The exception is the pull request bumping the Jenkins LTS baseline (the <!--RENOVATE-LTS--> marked jenkins.version property in pom.xml), which instead gets the lts-test label so it is tested against the lts line.