July 2022 meeting notes

2022-07-27 Cycle 3 check-in

We checked in on everybody’s workload and the progress of features involved in the demo.

Brent

• Will run out of tasks before next meeting
  • Not worth it to work on frontend tasks, so plan is to work on bonus backend features (Data Modeling Suggestions)
• Now assigned to help with any backend work for the Data Import
  • Only happy path work will be prioritized in Cycle 3
• Will discuss Schema Homepage with Ghislaine before starting work on feature
• Will discuss product plan for Data Modeling Suggestions with Kriti & Ghislaine before starting work on feature

Dhruvi

• Data Import is progressing
• Brent now assigned as a design approver to help with backend concerns
• Once Pavish is back, will need to discuss the format of deliverable needed for implementation

Dom

• Will be done with Data Explorer and Filtering work very soon
• Next priority: Check in with Sean if frontend work is available
• Otherwise: Start on Dynamic Defaults backend work
  • Discuss with Kriti & Ghislaine before starting work

Ghislaine

• On track with existing work
• Table Inspector planned to be done for Rajat to start work on next week
• No additional work needed

Mukesh

• Backend work on Column Extraction will be open for review soon
• Record Summary frontend and backend work pending, will be good until next week’s meeting

Pavish

Skipped since Pavish is on vacation.

Rajat

Skipped since Rajat doesn’t start until Monday.

Sean

• First priority is to get frontend tasks ready for Dom, Rajat, Mukesh
• Current work is on track and no additional work is needed

2022-07-22 Cycle 3 workflow

Topic: Cycle 3 workflow and loose ends discussion Attendees: Kriti, Brent, Ghislaine, Mukesh, Pavish, Sean

Communication

The Cycle 3 team needs to:

• know what everyone’s currently working on
• raise blockers and dependencies
• keep track of progress on current cycle goals
• find work
• know all relevant product, design, and technical specs related to what they’re implemented
• have asychronous long-form conversations to resolve non-urgent issues
• have synchronous conversations to resolve blockers
• have synchronous conversations to talk through problems
• document decisions from synchronous conversations for later reference

The Mathesar community needs to know:

• what the current goals of the project are
• how they can help
• how to collaborate with core team members

Considerations based on what was reported during the retrospective:

• The “big picture” goals and storyboards are useful
• Focusing on one thing at a time works well
• Email is controversial, everyone but Pavish had an opinion.
• A shared schema/dataset is useful, we should formalize a workflow around it.
• Small group conversations are useful, but they can get inefficient if information only gets transferred in 1:1s

Communication decisions

See the Moving forward column for how we plan to communicate about various things.

Type	Purpose	What we’ve tried	Moving forward
Team	Current Work Status	Matrix standups, email standups	Daily Matrix standups
Team	Blockers & Dependencies	direct comms, standups	Direct comms
Team	Current cycle goals	weekly planning GitHub discussions, delegated to sub-teams	Weekly meeting check-in with spreadsheet
Team	Finding new work	weekly planning GitHub discussions, delegated to sub-teams	Weekly meeting check-in with spreadsheet
Team	Implementation details	GitHub issues, direct comms, storyboard	Delegated to feature teams
Team	Async long form - product	GitHub discussions, email, Matrix	mathesar-developers list
Team	Async long form - workflow	GitHub discussions, email, Matrix	core-team or community-team list
Team	Resolve blockers	scheduled calls, direct comms, Matrix	Delegated to feature teams
Team	Talk through problems	ad-hoc	Delegated to feature teams
Team	Document decisions	wiki, GitHub discussions, email	Wiki (meeting notes, specs)
Community	Current project goals	developer mailing list, GitHub discussions	mathesar-developers mailing list
Community	How to help	GitHub issues, wiki	keep current workflow
Community	How to collaborate	wiki, Matrix, GitHub issues	keep current workflow

Notes

• Everyone except Mukesh hates GitHub Discussions
• Document decisions, but not so much that it slows work down.
  • Meeting notes should go on the wiki if you took them
  • Specs should go on the wiki

Organizing work

Design process

• Design owner meets with design approvers to
  • understand the requirements
  • decide on format of the deliverable
• Design owner produces deliverable
• All approvers sign off on the deliverable
• Deliverables should be on the wiki wherever possible
  • proto-specs and early deliverables too
• Frontend person starts work after deliverable is approved
• We will not be using the design prototype, Ghislaine will create prototypes and style the product directly
  • Backend will replace using prototype with using other design deliverables like spec, etc.
• Ghislaine will work on styling the product when nothing is blocked

Front end blockages

Design and backend work blocking frontend work was a concern raised in the retrospective. How do we resolve this in Cycle 3?

Not a concern for Cycle 3 since design and backend is currently ahead and frontend engineers are involved in design. We might need to think about it for Cycle 4.

May also be less of a concern as the product matures and we can have “thinner” vertical slices with fewer dependencies.

Dataset organization

During the retrospective, we talked about having a separate repo for sample schemas plus a process for changing it.

It would be useful to have for Cycle 3. Brent will set it up

• generate checkouts data as well (should be separate SQL file so it can be loaded optionally)
• We won’t be extending the library dataset, it will be frozen to what we need for the data.
• The repo can have additional data sets.
  • One column for each data type
  • Lots of self-referential data?
• We’ll actually just use the mathesar repo since it’s useful to load/unload data
  • Actually, we’ll decide to use a separate repo and anything that needs to go into the Mathesar repo for tests etc. can be moved to it
• Brent will make a GitHub issue to track this work

Eliminating unnecessary work

During the retrospective, we seemed to all agree that we need:

• a tighter scope
• moving faster

We did a good job of this during the paramaterized queries discussion. How can we continue that trend?

Weekly discussions will help – ask “how can we move faster or tighten scope?” at each weekly meeting.

Loose ends

Other things that came up in the retrospective.

2022 roadmap

Any questions or comments about the roadmap?

• Add performance & UX issues to Cycle 4
  • Perfomance might affect design (e.g. take things out)
  • API goals should be 50ms or less (local DB client speed)
    • DML performance matters more than DDL performance
• Start Cycle 5 work in Cycle 4 (already the plan)

Demo script

• Many people wanted a fully-fleshed out demo script
• Kriti will work on this during Cycle 3

Document to organize insights from storyboards

Do we want to do anything formal for this?

Depends on what people need from the storyboard - unified insights doesn’t seem that useful. This will be left up to individuals.

PDF of storyboard with pages would be helpful if someone knows how to do it.

Maybe move storyboard to spreadsheet with links to images if needed?

We got way too excited about refactoring the storyboards and decided to not make any changes in the end. If there’s a problem with discoverability, we will reevaluate.

API contract testing

What problem are we looking to solve here and is it worth prioritizing?

E2E tests shouldn’t test against the real API. Contract testing seems like a good idea in the long run.

Not relevant for Cycle 3.

2022-07-22 dependencies meeting

Topic: Decide scope of Yurii’s dependency project in the product and UI design Attendees: Kriti, Brent, Mukesh, Ghislaine

Two types of dependencies:

• DB objects (tables, functions, etc.)
• Mathesar objects

We need to portray CASCADE deletes (no-max-depth deletion of dependencies) differently from deleting Mathesar objects.

Brainstorming about dependencies in the UI

Brent’s wish-list

• Show “what would be deleted”
• Be able to use our own dependency tree to do the cascade delete, so that we don’t end up deleting dependencies not shown on the UI
• User should understand the difference between data deletions and data view or structure deletions
• User should understand delinking deletions
• UI should be able to handle arbitrary depth
• Categorize different dependencies.
  • data vs. representation
  • schemata
  • tables
  • constraints
  • functions
  • indexes
  • columns (maybe we could just drop fkey columns pointing to a table we want to delete)
  • sequences owned by table
  • etc.
• Dependencies of every type of object that can be directly deleted through Mathesar should be visible (even if we don’t have mathesar models for dependent objects)

Kriti’s wish-list

• We should track dependencies that are types of objects not handled by Mathesar’s UI (functions, indexes, whatever)
• We should show object types that we do have in the UI in as much detail as possible
• We should tell users when there are dependencies on object types that we don’t show in the UI (functions, etc.) and list them if we can
• We should also let the user know about non-DB object dependencies:
  • e.g. Mathesar queries, record summary templates
• We need to handle issues with objects that the user doesn’t have permissions for

Mukesh’s wish-list

• We should be using Django objects in place of database identifier
• Other dependencies should just contain the name(best case)
• dependencies should be categorized
• Filters for getting only certain objects

Ghislaine’s Wish-List

• Balance between risk of losing data and user inconvenience. We should avoid having the user take extra steps for minimal risk actions like columns, or summaries
• We need to handle bulk deletion scenarios. A user deleting multiple tables. Would we delete in steps and show dependencies for each? or a full list for all?
• Assuming that a constraint is deleted, user should be made aware of this when opening the table, not just at moment of deletion
  • Something like “information about table changes” when opening the table
  • Could this need a way to relink the column with all the FK?

Scope of the GSoC project

• UI scope should be limited to using dependencies during deletion and related operations, we will not have any other UI that uses dependencies.
• Brent, Yurii, and Mukesh will meet to discuss possible merging of dependencies and table relationships
• The scope will include tracking dependencies of all objects that the Mathesar API handles, i.e.
  • DBs
  • Schemas
  • Tables
  • Columns
  • Constraints
• The dependents tracked will include ALL objects, even if they’re not handled by the Mathesar API

Not in scope

• Separate page to show entire schema / DB depdendency graph
• Tracking dependencies of records
• Viewing the dependency graph of a single object without triggering a deletion

Type of deletion

• For the best UX, we should allow the users to delete multiple objects at once. It’s annoying when projects only allow deletion of one object at once.
• We do need to support automatic deletions of dependent objects as long as we have clarity on what’s getting deleted
  • If we don’t have clarity, we should make that very very clear.

Priorities for the GSoC project

• Backend parity with Postgres dependency tracking (so that we don’t have to resort to CASCADE)
• Backend for tracking Mathesar objects:
  • mapping tables and reverse dependencies
  • queries
  • record summaries
  • etc.

We can ship the project in a few different milestones.

Milestone 1: We show all the DB-only dependencies we can in the UI, allow users to use CASCADE if possible.

Milestone 2: We track mapping tables and reverse dependencies and show them in the UI

Milestone 3: We track Mathesar dependencies that are only in the service DB and show them in the UI (queries, record summaries)

Milestone 4: We track dependent DB functions (things not tracked in pg_depends)

Milestone 5: Parity with Postgres dependency tracking so that we can stop using CASCADE

2022-07-20 weekly meeting

Embedded Queries vs Parameterized Queries

• Summary: Figure out where the feature boundary between embedded and parameterized queries is and what’s actually essential for Cycle 3.
• Participants:: Kriti, Sean, Ghislaine, Pavish, Brent, Mukesh, Dhruvi

We’re all in agreement about the Record Page, which is a page that shows an individual record’s details. There’s a section in the page that shows a summary of related records and can allow records shown to be filtered.

The goal of this meeting is to figure out how much of this to implement in Cycle 3, since we have a lot of work to do and we want to do the minimum necessary to get the demo use case working.

Product and design concerns

Record Page design spec so far.

The product goal of embedded queries is to show a summary of related records on a record page. The data shown should match the links available via Data Explorer UI for consistency.

Sean’s long-term goals for the record page UX

• Let’s work from the Patrons record page as an example.
• A section of the Patrons record page displays related records which link to the patron.
• That section contains a list of queries.
• There are two types of queries within the list – “auto-generated queries” and “manually-generated queries”.
• Both types of queries can be re-ordered within the list
• Auto-generated queries:
  • display a table of the related records with all columns.
  • show on the record page by default according to the schema and thus will auto-update when the schema changes
  • should meet user’s needs in a majority of circumstances
  • can be hidden via an icon button (and un-hidden via a dropdown menu higher up on the page)
• Manually-generated queries
  • Can be renamed
  • Can be edited
  • Can be deleted
  • Can be created from scratch using the data explorer
• Reactions:
  • Other team members seem mostly in agreement with this UX vision
  • Kriti would like a way to eventually display the related records in a list form (e.g. genres associated with a book). But we’ll consider that use-case later.
  • We may eventually want to design functionality for the user to add new related records from within the record page. For example, I may want to record a new checkout for a patron directly on the patron’s page. We’ll consider that use-case later.

Backend concerns

• Need to be able to add/remove initial columns according to user desire from a stored query
• Don’t really need parameterized queries for the embedded queries. We can filter for the desired id using query string parameters when actually calling the query.
• Links to record:
  • call table endpoint, using filter for appropriate ID

Decisions

• We recognized that for our needs on the record page, we can handle the “parameterization” of queries using the “filters” functionality that we already have.
• The “auto-generated queries” on the record page will be displayed using our basic records API. No need to create a query resource at all! Just fetch the records and apply a filter to the FK column.
• For cycle 3 we’ll only focus on “auto-generated queries”. The “manually-generated queries” functionality will be designed/implemented later.
• The output of the auto-generated queries can be further filtered, beyond the auto-added FK filter condition. Any additional filter conditions that the user adds will not persist. But this ad-hoc filtering functionality also gets us a bit closer to the final goal of displaying checked-out books on the Patrons record page – all without using the data explorer (or Mathesar “queries”) at all!

Next steps

• The “Embedded Queries” feature is now folded into the “Record Page” feature. Because we’re not actually using “queries” for the functionality, we may use a name like “Table Widgets” (or similar) to refer to this functionality going forward.
• The “Parameterized Queries” feature is removed entirely as it seems like we may never need it. When the time comes for us to implement “manually-generated queries”, we will likely create basic query objects and then filter their results using the filtering functionality when displaying the query (not by altering the query definition).
• Brent is assigned to the Record Page backend since it doesn’t involve the Data Explorer anymore. The only thing we need to do is to make the “joinable tables” API work in both directions.
• Kriti will update the storyboard to remove the two features and update the flow according to the new implementation we decided on

2022-07-19 Cycle 3 planning

• Participants:: Kriti, Sean, Ghislaine, Pavish, Brent, Mukesh, Dominykas, Rajat, Dhruvi

We discussed the Cycle 3 storyboard, prioritized features, and assigned work to various team members. There were questions raised about the features “Embedded Queries” and “Parameterized Queries”, which we are going to schedule a separate meeting to iron out.

Team members who are working on Cycle 3 can view the work assignment spreadsheet here.

2022-07-18 Cycle 2 retrospective

• Participants:: Kriti, Sean, Ghislaine, Pavish, Brent, Mukesh, Dominykas

We discussed what went well and what could have gone better in Cycle 2, plus ideas for how to improve our workflow in Cycle 3. We used a whiteboarding tool during the meeting and did not take additional notes since the meeting focused on team workflow.

Core team members can view the whiteboard here.