Skip to content

Release QA testing

This page describes manual QA testing routines that we perform before a release.

Administrator-facing QA

Pre-requisites for testing Installation & Upgrades

For setups from scratch

  • Create and push a tag on GitHub with a draft release version.

    git checkout $VERSION
git pull
git tag "draft-$VERSION"
git push origin "draft-$VERSION"
    • GH will automatically create a draft release containing the static assets for this tag. This will take a few minutes.

  • Use the following command to download the static_files.zip asset from the draft release for the “Download release static files and extract into the correct directory” step during installation.

    gh release download --repo mathesar-foundation/mathesar "draft-$VERSION" -p 'static_files.zip'
    • You could also download this file via the GH web UI.
    • Since draft releases aren’t final, the urls generated by GH for the assets are dynamic. The static urls are formed when the release is published.

For installations:

  • Clear out remains of any previous installations.

For upgrades:

  • Clear out remains of any previous installations.
  • Install the current published version of Mathesar.

For both:

  • Locally download and tag the new version images as latest:

    docker image pull mathesar/mathesar-caddy:<version_number>
docker image tag mathesar/mathesar-caddy:<version_number> mathesar/mathesar-caddy:latest

docker image pull mathesar/mathesar-prod:<version_number>
docker image tag mathesar/mathesar-prod:<version_number> mathesar/mathesar-prod:latest

Tasks

  • Test installing the release from scratch by following our documentation
  • Test all different installation methods
    • Docker compose
      • All defaults, local
      • Exposed on domain, DB managed by Mathesar
      • Exposed on domain, DB preexisting for users, managed DB for Django
      • Exposed on domain, DB preexisting for all data
    • Docker integrated image
      • All defaults, local
      • Exposed on domain, DB managed by Mathesar
      • Exposed on domain, DB preexisting for users, managed DB for Django
      • Exposed on domain, DB preexisting for all data
    • Build from scratch
      • All defaults, local
      • Exposed on domain, DB preexisting for all data
  • Test that previously installed versions of Mathesar can be upgraded to this release.
    • Docker compose – All above variants for docker compose
      • Same docker compose file, same .env
      • New docker compose file, same .env
      • New docker compose file, bring .env into the new file
    • Build from scratch – All above variants for build from scratch

Cleanup

  • Destroy any infra setup on GCP that was created for QA purposes.

  • Delete the draft release and tag that were created when testing “Setups from scratch”.

    gh release delete "draft-$VERSION" -y
git tag -d "draft-$VERSION"
git push -d origin "draft-$VERSION"

User-facing QA

Tips:

  • Don’t just aim to complete the tests with the minimum possible steps — try to test the edges around each of the features, seeing if you can hammer at Mathesar to break something or uncover bugs.
  • If you find a bug, try to reproduce it against the latest release (e.g. on the demo site). If you can reproduce it on the latest release, then it’s not a regression. It’s still worth reporting, but it won’t be as high of a priority. If you can’t reproduce it on the latest release, then it is a regression. It’s important to specify this.
  • The PRs for issues should be based off of the release branch, and merged into the release branch.

Tasks:

  • Test create/update/delete for:
    • Mathesar users
    • Connection
    • Schema
    • Table
    • Column
    • Constraint
  • Import
    • Test basic import.
    • Try different options for “Data Source”
    • Try different options for “Column Data Types”
  • Table Page
    • Test filtering, sorting, grouping
    • Test pagination
    • Test updating cell values for all data types
    • Test showing/hiding/resizing table inspector and collapsing/expanding sections
    • Test keyboard shortcuts to move active cell, enter edit mode, save value
    • Ensure the context menu looks correct for data cells, column header cells, and row header cells
    • Test table sharing
    • Test cell selection via: dragging from data cell to data cell, dragging from column header cell to column header cell, dragging from row header cell to row header cell
    • Test custom record summary template
    • Test updating column data type
    • Test updating column display settings
    • Test setting column default value
    • Test column re-ordering via drag & drop
    • Test “Create Link” dialog
    • Test “Extract Column Into a New Table”
    • Test “Move Column To Linked Table”
    • Test “Share” functionality — and test that the share page works
  • Record Selector
    • Test filtering on multiple columns
    • Test picking an existing record
    • Test creating a new record from within the record selector
    • Test selecting a record from within a nested record selector
    • Test horizontal and vertical scrolling for narrow and short viewports
  • Record Page
    • Test updating direct fields
    • Test setting field values to null
    • Test navigating to linked records through linked record input
    • Test loading a record page with linked records
  • Data Explorer
    • Test creating and saving an exploration from Data Explorer view
    • Try joining columns from multiple tables without summarization
    • Try joining columns from multiple tables with summarization
    • Try renaming column in column properties
    • Test pagination
    • Test ‘Filter’ transformation
    • Test ‘Sort’ transformation
    • Test ‘Hide Columns’ transformation
    • Test ‘Summarization’ transformation
    • Try summarizing with different aggregations
    • Test opening an existing exploration
    • Test editing and saving an existing exploration
    • Create an exploration from Table page -> Inspector -> Actions -> Explore Data
    • Test “Share” functionality — and test that the share page works