At Message Media I built a Functional Test pipeline within our CI/CD process. The idea was to drastically reduce the amount of regression testing we were doing and enable the QAs to do what they do best; -create work for me- focus on finding bugs.
As our tests grew larger we started to run into issues with stability. Writing the tests were time consuming and the format did not match the coding style of the application.
So what were you using?
So why the change?
Selenium and the W3C WebDriver API plainly sucks, we were constantly running into issues with:
- Random failures
- Writing the tests
- Mismatched coding styles
- Bulky requiring large installs
- Has Java as a requirement
- A general pain in the ass
There’s got to be an easier solution out there, right?
- Uses the Cucumber Gherkin Syntax
- Cross platform as we have team members using Mac / Windows / Linux
- Testing must be consistent
- Aim for under ~10 minute full run (developers are bad at waiting)
When putting it into practice, the following issues occurred:
- The Nightwatch Cucumber plugin does not support scenarios
- It still uses Selenium (through Selenium Server)
- Upon further reading we would have to spend a lot of time and energy writing fault tolerant tests
Once again we found issues including:
- The Gherkin plugin did not allow us to pass parameters to Test Cafe
- Cookies were reset between the Gherkin steps not allowing complete scenarios to run
- Lacked documentation and was hard to navigate
- Difficult to write tests
- Forces you to use a PageModel syntax
- Amazing documentation
- Easy to use live development (watch) mode
- Does not use selenium
- Uses mocha under the hood for simpler assertion writing
- Supports TDD Gherkin syntax with a plugin
- Built in error handling
- No parallel testing yet
- No cross browser support yet
- You have to be very specific when writing your Cukes (Given / When / Then)
We have since converted our Front end Functional Testing process to Cypress, which surprisingly only took a few days and in the process made me fall in love with their toolchain.
I hope this article helps with your decision process on what testing tool to use.