Skip to content

Release QA testing

This page describes manual QA testing routines that we perform before a release.

Administrator-facing QA

Pre-requisites for testing Installation & Upgrades

For direct install on Linux, macOS, and WSL

  • Ensure that there is a release on GitHub and the release assets can be downloaded via public url.
  • This should have been done when cutting the release branch.

For upgrades:

  • Run the new install script pointing to the same directory as the previous installation.

For installations:

  • Clear out remains of any previous installations.

For upgrades:

  • Clear out remains of any previous installations.
  • Install the current published version of Mathesar.

For both:

  • Locally download and tag the new version images as latest:

    docker image pull mathesar/mathesar-caddy:<version_number>
    docker image tag mathesar/mathesar-caddy:<version_number> mathesar/mathesar-caddy:latest
    
    docker image pull mathesar/mathesar:<version_number>
    docker image tag mathesar/mathesar:<version_number> mathesar/mathesar:latest
    

Understanding Database connections

Database types

  • Mathesar Database - The Django database used by Mathesar
  • Data database - The user database(s) being explored in Mathesar’s UI

Connection types

  • Local - On the host machine
  • External - Outside of the host machine
  • Integrated - Inside of Mathesar’s docker image
  • Docker Postgres service - Postgres service running in a docker compose network

Tasks

  • Test direct install on Linux and macOS by following our documentation
  • Test all different installation methods
    • Docker compose
      • Local using all default settings
      • Exposed on domain with different database configurations:
      • Docker Postgres service Mathesar, Docker Postgres service Data
      • Docker Postgres service Mathesar, External Data
      • External Mathesar, External Data
    • Docker “Try Mathesar” command with different database configurations:
      • Integrated Mathesar, Integrated Data
      • Integrated Mathesar, External Data
    • Direct install on Linux
      • Exposed on domain, DB preexisting for all data
      • Local using all default settings
      • Exposed on domain with Local Mathesar, Local Data
  • Test that previously installed versions of Mathesar can be upgraded to this release.
    • Docker compose – All above variants for docker compose
      • Same docker compose file, same .env
      • New docker compose file, same .env
      • New docker compose file, bring .env into the new file
    • Direct install on linux (Previously known as Install from scratch) – All above variants

User-facing QA

Tips:

  • Don’t just aim to complete the tests with the minimum possible steps — try to test the edges around each of the features, seeing if you can hammer at Mathesar to break something or uncover bugs.
  • If you find a bug, try to reproduce it against the latest release (e.g. on the demo site). If you can reproduce it on the latest release, then it’s not a regression. It’s still worth reporting, but it won’t be as high of a priority. If you can’t reproduce it on the latest release, then it is a regression. It’s important to specify this.
  • The PRs for issues should be based off of the release branch, and merged into the release branch.

Tasks:

  • Test create/update/delete for:
    • Mathesar users
    • Connection
    • Schema
    • Table
    • Column
    • Constraint
  • Import
    • Test basic import.
    • Try different options for “Data Source”
    • Try different options for “Column Data Types”
  • Table Page
    • Test filtering, sorting, grouping
    • Test pagination
    • Test updating cell values for all data types
    • Test showing/hiding/resizing table inspector and collapsing/expanding sections
    • Test keyboard shortcuts to move active cell, enter edit mode, save value
    • Ensure the context menu looks correct for data cells, column header cells, and row header cells
    • Test table sharing
    • Test cell selection via: dragging from data cell to data cell, dragging from column header cell to column header cell, dragging from row header cell to row header cell
    • Test custom record summary template
    • Test updating column data type
    • Test updating column display settings
    • Test setting column default value
    • Test column re-ordering via drag & drop
    • Test “Create Link” dialog
    • Test “Extract Column Into a New Table”
    • Test “Move Column To Linked Table”
    • Test “Share” functionality — and test that the share page works
  • Record Selector
    • Test filtering on multiple columns
    • Test picking an existing record
    • Test creating a new record from within the record selector
    • Test selecting a record from within a nested record selector
    • Test horizontal and vertical scrolling for narrow and short viewports
  • Record Page
    • Test updating direct fields
    • Test setting field values to null
    • Test navigating to linked records through linked record input
    • Test loading a record page with linked records
  • Data Explorer
    • Test creating and saving an exploration from Data Explorer view
    • Try joining columns from multiple tables without summarization
    • Try joining columns from multiple tables with summarization
    • Try renaming column in column properties
    • Test pagination
    • Test ‘Filter’ transformation
    • Test ‘Sort’ transformation
    • Test ‘Hide Columns’ transformation
    • Test ‘Summarization’ transformation
    • Try summarizing with different aggregations
    • Test opening an existing exploration
    • Test editing and saving an existing exploration
    • Create an exploration from Table page -> Inspector -> Actions -> Explore Data
    • Test “Share” functionality — and test that the share page works

Cleanup

After QA

Cleanup before another round of QA and before publishing the release

  • Destroy any infra setup on GCP that was created for QA purposes.
  • Delete the pre-release and tag that were created when testing “Direct installation on Linux, macOS, and WSL”.
    • Recommended: Do this via the GitHub UI to avoid any potential mistakes.
    • Delete the pre-release.
    • Delete the tag associated with it.
    • Commands, if you’re confident:
      VERSION=1.2.3 # ⚠️ CUSTOMIZE THIS
      
      gh release delete "$VERSION" -y
      git tag -d "$VERSION"
      git push -d origin "$VERSION"