UI Functional Testing Journey

Posted on:

source

At Message Media I built a Functional Test pipeline within our CI/CD process. The idea was to drastically reduce the amount of regression testing we were doing and enable the QAs to do what they do best; -create work for me- focus on finding bugs.

As our tests grew larger we started to run into issues with stability. Writing the tests were time consuming and the format did not match the coding style of the application.

So what were you using?

So why the change?

Selenium and the W3C WebDriver API plainly sucks, we were constantly running into issues with:

  • Stability
  • Random failures
  • Writing the tests
  • Mismatched coding styles

It’s also

  • Bulky requiring large installs
  • Has Java as a requirement
  • A general pain in the ass

There’s got to be an easier solution out there, right?

Our Requirements

  • Uses the Cucumber Gherkin Syntax
  • Cross platform as we have team members using Mac / Windows / Linux
  • Testing must be consistent
  • Aim for under ~10 minute full run (developers are bad at waiting)

Nightwatch

From initial research Nightwatch with nightwatch-cucumber looked like a promising candidate. It offered the gherkin syntax with a better Javascript style. It also has the added bonus of being able to debug with VS Code

When putting it into practice, the following issues occurred:

  • The Nightwatch Cucumber plugin does not support scenarios
  • It still uses Selenium (through Selenium Server)
  • Upon further reading we would have to spend a lot of time and energy writing fault tolerant tests

Test Cafe

Looked more promising as it doesn’t use Selenium and also has a Gherkin wrapper to achieve the Cucumber requirement.

Once again we found issues including:

  • The Gherkin plugin did not allow us to pass parameters to Test Cafe
  • Cookies were reset between the Gherkin steps not allowing complete scenarios to run
  • Lacked documentation and was hard to navigate
  • Difficult to write tests
  • Forces you to use a PageModel syntax

Cypress

The holy grail with a great Cucumber Preprocessor plugin, amazing documentation and built in fault tolerance to help with random failures.

Good parts

Bad parts

  • No parallel testing yet
  • No cross browser support yet
  • You have to be very specific when writing your Cukes (Given / When / Then)

We have since converted our Front end Functional Testing process to Cypress, which surprisingly only took a few days and in the process made me fall in love with their toolchain.

I hope this article helps with your decision process on what testing tool to use.