From 7003dd2d2d998bd27e3e1d48ff18a76d7c2beb70 Mon Sep 17 00:00:00 2001 From: Florian Giroud <6267288+fgiroud@users.noreply.github.com> Date: Sun, 7 Feb 2021 19:07:54 +0100 Subject: [PATCH] doc: Enriched UX testing documentation, #3573 (#3583) --- .../technical-reference/functional-tests.md | 188 ++++++++++++++---- .../tests/cypress/cypress/support/commands.js | 43 ++++ 2 files changed, 187 insertions(+), 44 deletions(-) diff --git a/docs/docs/technical-reference/functional-tests.md b/docs/docs/technical-reference/functional-tests.md index c055bc0fb..fee0b64c4 100644 --- a/docs/docs/technical-reference/functional-tests.md +++ b/docs/docs/technical-reference/functional-tests.md @@ -6,53 +6,22 @@ sidebar_label: Functional tests import useBaseUrl from '@docusaurus/useBaseUrl'; -You will need: +## Introduction -- [Node.js 10 or 12 and above](https://nodejs.org) -- [Yarn or NPM](https://yarnpkg.com/) -- A Unix/Linux shell environment or the Windows command line +OpenRefine interface is tested with the [Cypress framework](https://www.cypress.io/). +With Cypress, tests are performing assertions using a real browser, the same way a real user would use the software. -## Installation +Cypress tests can be ran -To install Cypress and dependencies, run : +- using the Cypress test runner (development mode) +- using a command line (CI/CD mode) -``` -cd ./main/tests/cypress -yarn install -``` - -Cypress tests can be started in two modes: - - -### Development / Debugging mode - -Dev mode assumes that OpenRefine is up and running on the local machine, the tests themselves do not launch OpenRefine, nor restarts it. - -Run : - -```shell -yarn --cwd ./main/tests/cypress run cypress open -``` - -It will open the Cypress test runner, where you can choose, replay, visualize tests. -This is the recommended way to run tests when adding or fixing tests. -The runners assumes - -### Command-line mode - -Command line mode will starts OpenRefine with a temporary folder for data - -```shell -./refine ui_test chrome -``` - -It will run all tests in the command-line, without windows, displaying results in the standard output -This is the way to run tests in CI/CD +If you are writing tests, the cypress test runner is good enough, and the command-line is mainly used by the CI/CD platform (Github actions) ## Cypress brief overview Cypress operates insides a browser, it's internally using NodeJS. -That's a key difference with tools such as selenium. +That's a key difference with tools such as selenium. **From the Cypress documentation:** @@ -67,12 +36,132 @@ The general workflow of a Cypress test is to - Trigger user actions - Assert that the DOM contains expected texts and elements using selectors -## Browsers +## Getting started + +If that's the first time you use Cypress, it is recommended for you to get familiar with the tool. + +- [Cypress overview](https://docs.cypress.io/guides/overview/why-cypress.html) +- [Cypress examples of tests and syntax](https://example.cypress.io/) + +### 1. Install Cypress + +You will need: + +- [Node.js 10 or 12 and above](https://nodejs.org) +- [Yarn or NPM](https://yarnpkg.com/) +- A Unix/Linux shell environment or the Windows command line + +To install Cypress and dependencies, run : + +``` +cd ./main/tests/cypress +yarn install +``` + +### 2. Start the test runner + +The test runner assumes that OpenRefine is up and running on the local machine, the tests themselves do not launch OpenRefine, nor restarts it. + +Start OpenRefine with + +```shell +./refine +``` + +Then start Cypress + +```shell +yarn --cwd ./main/tests/cypress run cypress open +``` + +### 3. Run the existing tests + +Once the test runner is up, you can choose to run one or several tests by selecting them from the interface. +Click on one of them and the test will start. + +### 4. Add your first test + +- Add a `test.spec.js` into the cypress/integration folder. +- The test is instantly available in the list +- Click on the test +- Start to add some code + +## Tests technical documentation + +### A typical test + +A typical OpenRefine test starts with the following code + +```javascript +it('Ensure cells are blanked down', function () { + cy.loadAndVisitProject('food.mini') + cy.get('.viewpanel-sorting a').contains('Sort').click() + cy.get('.viewpanel').should('to.contain', 'Something') +}) +``` + +The first noticeable thing about a test is the description (`Ensure cells are blanked down`), which describes what the test is doing. +Lines usually starts with `cy.something...`, which is the main way to interact with the Cypress framework. + +A few examples: + +- `cy.get('a.my-class')` will retrieve the `` element +- `cy.click()` will click on the element +- eventually, `cy.should()` will perform an assertion, for example that the element contains an expected text with `cy.should('to.contains', 'my text')` + +On top of that, OpenRefine contributors have added some functions for common OpenRefine interactions. +For example + +- `cy.loadAndVisitProject` will create a fresh project in OpenRefine +- `cy.assertCellEquals` will ensure that a cell contains a given value + +See below on the dedicated section 'Testing utilities' + +### Testing guidelines + +- `cy.wait` should be used in the last resort scenario. It's considered a bad practice, though sometimes there is no other choice +- Tests should remain isolated from each other. It's best to try one feature at the time +- A test should always start with a fresh project +- The name of the files should mirror the OpenRefine UI organization + +### Testing utilities + +OpenRefine contributors have added some utility methods on the top of the Cypress framework. +Those methods perform some common actions or assertions on OpenRefine, to avoid code duplication. + +Utilities can be found in `cypress/support/commands.js`. + +The most important utility method is `loadAndVisitProject`. +This method will create a fresh OpenRefine project based on a dataset given as a parameter. +The fixture parameter can be + +- An arbitrary array, the first row is for the column names, other rows are for the values + Use an arbitrary array **only** if the test requires some specific grid values + **Example:** + + ```javascript + const fixture = [ + ['Column A', 'Column B', 'Column C'], + ['0A', '0B', '0C'], + ['1A', '1B', '1C'], + ['2A', '2B', '2C'], + ] + cy.loadAndVisitProject(fixture) + ``` + +- A referenced dataset: `food.small` or `food.mini` + Most of the time, tests does not require any specific grid values + Use food.mini as much as possible, it loads 2 rows a very few columns in the grid + Use food.small if the test requires a few hundreds of rows in the grid + + Those datasets live in `cypress/fixtures` + +### Browsers In terms of browsers, Cypress is using what is installed on your operating system. See the [Cypress documentation](https://docs.cypress.io/guides/guides/launching-browsers.html#Browsers) for a list of supported browsers -## Folder organization +### Folder organization Tests are located in main/tests/cypress. The test should not use any file outside the cypress folder. @@ -83,7 +172,7 @@ The test should not use any file outside the cypress folder. - `/screenshots` and `/videos` contains the recording of the tests, Git ignored - `/support` is a custom library of assertion and common user actions, to avoid code duplication in the tests themselves -## Configuration +### Configuration Cypress execution can be configured with environment variables, they can be declared at the OS level, or when running the test @@ -93,11 +182,11 @@ Available variables are Cypress contains and [exaustive documentation](https://docs.cypress.io/guides/guides/environment-variables.html#Setting) about configuration, but here are two simple ways to configure the execution of the tests: -### Overriding with a cypress.env.json file +#### Overriding with a cypress.env.json file This file is ignored by Git, and you can use it to configure Cypress locally -### Command-line +#### Command-line You can pass variables at the command-line level @@ -105,6 +194,17 @@ You can pass variables at the command-line level yarn --cwd ./main/tests/cypress run cypress open --env OPENREFINE_URL="http://localhost:1234" ``` +## CI/CD + +In CI/CD, tests are runned headless, with the following command-line + +```shell +./refine ui_test chrome +``` + +Results are displayed in the standard output + ## Resources [Cypress command line options](https://docs.cypress.io/guides/guides/command-line.html#Installation) +[Lots of good Cypress examples](https://example.cypress.io/) diff --git a/main/tests/cypress/cypress/support/commands.js b/main/tests/cypress/cypress/support/commands.js index 153b36889..50e572c2b 100644 --- a/main/tests/cypress/cypress/support/commands.js +++ b/main/tests/cypress/cypress/support/commands.js @@ -35,6 +35,9 @@ Cypress.Commands.add('editCell', (rowIndex, columnName, value) => { cy.get('.menu-container button[bind="okButton"]').click(); }); +/** + * Ensure a textarea have a value that id equal to the JSON given as parameter + */ Cypress.Commands.add('assertTextareaHaveJsonValue', (selector, json) => { cy.get(selector).then((el) => { // expected json needs to be parsed / restringified, to avoid inconsitencies about spaces and tabs @@ -42,6 +45,10 @@ Cypress.Commands.add('assertTextareaHaveJsonValue', (selector, json) => { cy.expect(JSON.stringify(present)).to.equal(JSON.stringify(json)); }); }); + +/** + * Open OpenRefine + */ Cypress.Commands.add('visitOpenRefine', (options) => { cy.visit(Cypress.env('OPENREFINE_URL'), options); }); @@ -78,6 +85,9 @@ Cypress.Commands.add('doCreateProjectThroughUserInterface', () => { }); }); +/** + * Cast a whole column to the given type, using Edit Cell / Common transform / To {type} + */ Cypress.Commands.add('castColumnTo', (selector, target) => { cy.get( '.data-table th:contains("' + selector + '") .column-header-menu' @@ -90,6 +100,9 @@ Cypress.Commands.add('castColumnTo', (selector, target) => { cy.get('body > .menu-container').eq(2).contains(targetAction).click(); }); +/** + * Return the td element for a given row index and column name + */ Cypress.Commands.add('getCell', (rowIndex, columnName) => { const cssRowIndex = rowIndex + 1; // first get the header, to know the cell index @@ -102,6 +115,9 @@ Cypress.Commands.add('getCell', (rowIndex, columnName) => { }); }); +/** + * Make an assertion about the content of a cell, for a given row index and column name + */ Cypress.Commands.add('assertCellEquals', (rowIndex, columnName, value) => { const cssRowIndex = rowIndex + 1; // first get the header, to know the cell index @@ -121,25 +137,40 @@ Cypress.Commands.add('assertCellEquals', (rowIndex, columnName, value) => { }); }); +/** + * Navigate to one of the entries of the main left menu of OpenRefine (Create Project, Open Project, Import Project, Language Settings) + */ Cypress.Commands.add('navigateTo', (target) => { cy.get('#action-area-tabs li').contains(target).click(); }); +/** + * Wait for OpenRefine to finish an Ajax load + */ Cypress.Commands.add('waitForOrOperation', () => { cy.get('body[ajax_in_progress="true"]'); cy.get('body[ajax_in_progress="false"]'); }); +/** + * Delete a column from the grid + */ Cypress.Commands.add('deleteColumn', (columnName) => { cy.get('.data-table th[title="' + columnName + '"]').should('exist'); cy.columnActionClick(columnName, ['Edit column', 'Remove this column']); cy.get('.data-table th[title="' + columnName + '"]').should('not.exist'); }); +/** + * Wait until a dialog panel appear + */ Cypress.Commands.add('waitForDialogPanel', () => { cy.get('body > .dialog-container > .dialog-frame').should('be.visible'); }); +/** + * Click on the OK button of a dialog panel + */ Cypress.Commands.add('confirmDialogPanel', () => { cy.get( 'body > .dialog-container > .dialog-frame .dialog-footer button[bind="okButton"]' @@ -147,6 +178,9 @@ Cypress.Commands.add('confirmDialogPanel', () => { cy.get('body > .dialog-container > .dialog-frame').should('not.exist'); }); +/** + * Will click on a menu entry for a given column name + */ Cypress.Commands.add('columnActionClick', (columnName, actions) => { cy.get( '.data-table th:contains("' + columnName + '") .column-header-menu' @@ -158,11 +192,20 @@ Cypress.Commands.add('columnActionClick', (columnName, actions) => { cy.get('body[ajax_in_progress="false"]'); }); +/** + * Go to a project, given it's id + */ Cypress.Commands.add('visitProject', (projectId) => { cy.visit(Cypress.env('OPENREFINE_URL') + '/project?project=' + projectId); cy.get('#project-title').should('exist'); }); +/** + * Load a new project in OpenRefine, and open the project + * The fixture can be + * * an arbitrary array that will be loaded in the grid. The first row is for the columns names + * * a file referenced in fixtures.js (food.mini | food.small) + */ Cypress.Commands.add( 'loadAndVisitProject', (fixture, projectName = Date.now()) => {