Skip to content

September 2022 meeting notes

2022-09-28 weekly meeting

Team goals discussion

Attendees: Entire core team

We discussed the team goals document (only accessible to the core team), which outlines priorities for the rest of 2022 and criteria for what is critical for the live demo and first release.

The core team will be using this document as a guide from now on. We will need to come up with a new set of goals and priorities once we have completed our first release.

Live demo readiness check in & next steps

Attendees: Entire core team

What are next steps for getting ready for the live demo?

  • Kriti will close demo video milestone
  • Kriti will take a quick look at the live demo milestone and boot out tickets that don’t meet the criteria established above
    • This is a one time thing
    • Everyone on the team will be responsible for making sure their work fits in the live demo
  • We need a round of usability testing to flesh out the remaining Styling tickets (for column extraction and the exploration page)
    • Kriti will send out instructions for what to do via email
    • Looking for volunteers
      • Sean
      • Brent
      • Rajat
      • Ghislaine
      • Dom
      • Pavish
    • We don’t need so many people, Kriti will pick a subset of people and mention them in the email
      • Dom notes that he has relatively fresh eyes since he hasn’t used the frontend much
    • Goal for this round is “I don’t understand” / UX improvements
  • After the styling work has been implemented in the product, we will do another round of usability testing of the entire product focused on finding bugs
  • We can bring in CCI / external parties to do this
  • We’ll organize this later

Hacktoberfest

Attendees: Entire core team

Should we participate in Hacktoberfest?

No, we need to focus on the live demo. Maybe next year.

Backend problems discussion

Attendees: Kriti, Brent, Mukesh, Dom, Sean

Dom started this topic as “Importance of refactorability” with the following explanation:

Our workflow makes us extra-sensitive to refactorability, I argue; I’d like to reach a common understanding between (at least) backend devs regarding this topic.

We didn’t have enough time to have a thorough discussion, so we instead focused on documenting problems that the backend team has been facing. We’re going to set up a separate meeting to resolve the problems.

Problems:

  • Need a common understanding of how important refactorability is
    • We refactor often
    • Refactoring seems hard, accumulate technical debt fast
    • How can we make it less expensive to refactor?
  • Different people find different approaches easy or hard,
    • What’s hard for one person is easy for someone else
  • Our frameworks don’t make sense for our product
    • Refactoring within these frameworks is probably just another temporary solutions
  • Discoverability is a problem
    • Small functions
    • We don’t have a structure that we’ve agreed on, what’s our “atom?”
    • People misusing functions or not knowing it’s there
    • Helper functions in a module, but need to be used in a bunch of different modules
    • Discoverability is about good namespacing
      • It’s not that we have a lot of functions
        • It’s that they’re namespaced unintuitively
      • Boolean switches are a bad way to namespace
      • Our namespacing has been very lazy
        • Needlessly big files with varied content
          • E.g. most base.py files
        • We should aim for small base.py files
          • They’re the files that should tie a module together
  • Wait until we have a proper architecture before we decide on how to refactor
    • Counter argument
      • We need good refactorability to get to a good architecture
        • Currently we’re just too burdened
      • No stability in foreseeable future
  • Overengineering early is a problem, not knowing what things are going to be
    • What is overengineering in this context?
  • Brent doesn’t know how error infrastructure works,
    • but needs to know to find badly caught errors
    • Doesn’t work in some ways
    • Error infrastructure seems more complicated that needed
      • Deep inheritance trees
    • This is a problem with a bunch of different places in our code
    • It’s intimidating
      • But, once Mukesh showed me how to use it
        • I’ve been using it fine
  • “I might repeat this, pulling this out to a separate function”
    • Too early
    • Dom
      • I pull out stuff based on readability usually, not reuse
        • When you’re doing stuff,
        • and you encapsulate it into a named method,
        • that’s an opportunity to tell the reader what you’re doing.
      • So, for me, there’s no “too early”
  • Some people are more or less familiar with Django/Python idioms

Next steps: Kriti will schedule backend meeting.

2022-09-22 Styling planning

Attendees: Pavish, Sean, Rajat, Ghislaine

We discussed on styling deliverables and planning out frontend implementation.

Plan

  • Ghislaine will provide high-level information on all different types of layouts, so the frontend team can start implementing the lower level components
  • For each page in the figma designs, one or more of the frontend devs will take control of reviewing them thoroughly. This will be a general review.
  • Rajat will be incharge of the entire styling implementation. He will be working on most of it, and he can also delegate the tasks to the rest of the frontend team. The call rests on him.
  • Before implementation:
  • Rajat will ask any design questions on the figma file, which may affect the implementation process. Ghislaine will be incharge of answering them.
  • Rajat will create GH tickets for implementation work. Pavish or Sean will review the proposed architectural changes if any, or answer any questions on the GH issue, before approving it to start implementation.
  • The implementation tickets will be separated by new components needed, architectural changes, new features, and pure styling work.
  • During implementation:
  • Sean and Pavish will act in a supporting role, taking up any issue Rajat wants to delegate, answer any questions, or help designing the architecture.

2022-09-21 weekly meeting

Demo video plan

  • The new plan is to make a short product introduction video using the Figma mockups we’re creating for styling the product.
  • This will allow engineering to focus on the live demo and allow us to launch the marketing website sooner.

Live demo readiness plan

Three stages:

  • Stage 1: Continue to work on usability testing using the library storyboard and get everything to work and look good.
  • Stage 2: Usability testing on all other features we’ve implemented so far, get it all to work and look good.
  • Stage 3: Logistics for the live demo (e.g. how do we allow many different users to look at our live demo at the same time?)

We will probably need to cut scope for the live demo. We’ll talk through what’s essential at next week’s meeting. Kriti will send an agenda and things to think about beforehand.

Issue/PR triage

  • We’re removing CODEOWNERS files from repos to avoid unwanted notifications.
  • Sean will take over triaging issues and making sure they are assigned to someone for review.
    • Everyone is still responsible for assigning reviewers for their own PRs, Sean is just a last resort.
  • Rajat will be the backup person for triaging issues in case Sean is unavailable.

Current work check-ins

  • We’re abandoning the “everyone go around and talk about your work” part of this meeting. We’ll instead focus on blockers, dependencies, and questions that the entire team is needed for.
  • Brent: Users & Permissions is on pause because of live-demo readiness bugs, is this okay?
  • Yes, this is fine. Once Dom or Mukesh is freed from performance work, we should then have someone dedicated to Users & Permissions
  • Brent will need to schedule a call for deciding how to merge in Users & Permissions since it will break everyone’s workflows to need authentication
  • Rajat: Plan is to start DB page styling on Friday, are there other priorities?
  • Frontend team will meet to discuss how to break up styling work tomorrow
  • Ghislaine will also attend
  • Dom: There’s a failing test with moving columns that’s preventing merge of the metadata performance issue. Okay to merge it anyway and fix the moving columns issue later?
  • Yes, that is fine. We’ll make a ticket for it.
  • Mukesh has some time to debug this now.
  • Mukesh is making good progress on the performance issues, has reduced queries from 400 to 200. Remaining queries are related to Dom’s work.
  • Code is not very readable but since it’s related to Django-SQLAlchemy syncing, and we plan to remove that whole part of our architecture fairly soon, readability is not a concern.

2022-09-16 demo readiness

Attendees: Brent, Ghislaine, Kriti, Pavish, Sean (partial)

We went through usability testing notes on Figma and decided which issues were problems. Relevant issues were later transferred into GitHub.

Table Page

  • no changes needed to URL

Table Inspector

  • Table constraints: move to separate location
    • Location for demo video
    • For live demo, separate constraints by type
  • Links section
    • should exist,
    • should show both directions
  • Columns tab
    • Make different interactions have different styling
    • Try making interactions as similar as possible
    • unique constraint should be positive in both places or negative in both places
    • Reason for two places: multicolumn uniqueness constraints

Screen 8

  • Improve navigation (general)
    • specifically, navigating to import place

Screen 20

  • Column name running out
    • We’d like to persist column width settings for demo video
  • Record selector discoverability
    • Magnifying glass not exactly obvious
    • improve finding record selector
  • Record link vs. record selector icons distinction isn’t obvious
  • Help content
    • hard to understand that this does searching and also adding
    • adding and selecting should be more distinct
  • White space
    • we agree this is a problem

Screen 22

  • Want to add type when creating column
  • Improve modal for linking table
  • improve table load (or get rid of that) when creating link
  • one-to-one record not working

Screen 26

  • Styling
    • assess and improve
  • queries
    • more help (“why are these there?”)
  • Brent TODO file bug for missing records (or wrong count)

Screen 27

  • Bug for NOW() (confusing behavior)

Screen 28

  • Icons aren’t obvious or clear in their use
  • sequential loading of related records is confusing

Screen 30

  • Dropdown options are confusing
  • preproc is confusing

Screen 32

  • Go with count for default aggregation

Screen 34

  • Help user understand single vs. multiple results
  • Help user understand why some not available

2022-09-14 demo readiness

Attendees: Brent, Ghislaine, Kriti, Pavish, Rajat, Sean

We did some live usability testing, but didn’t get into any action items.

Sean volunteered to split some tickets into “demo video” and “live demo”.

2022-09-14 weekly meeting

We talked through progress for Cycle 4. Nothing major to report.

Scope and expectations for cycle

  • Topic summary: keeping scope and expectations limited for current cycle (and any before release)

  • Triggered by https://github.com/centerofci/mathesar/issues/1632 but not specific to it

  • Not clear why it’s blocking the demo video
  • Action item: If putting something into Demo video, live demo, or initial release milestones, please indicate why it’s absolutely crucial for those milestones

2022-09-07 demo video readiness

Attendees: Kriti, Ghislaine, Pavish, Brent, Rajat (partial), Sean (partial)

  • Demo should start with the Database page.
  • Database page design should be updated to include user onboarding elements.
    • Each schema should include a count of tables and explorations
    • Public schema will have a default description that might not be helpful for users.
      • From a product perspective we’d like users to create a new schema
    • New schema action should be more prominent
    • We need to design for various states:
      • Empty public schema
      • Public schema with tables

We switched to taking notes directly on the storyboard at this point.

GitHub issue

We created a GitHub issues with todos to do before the next meeting.

Please see https://github.com/centerofci/mathesar/issues/1604

Live demo issues

We won’t be handling this just yet, but making notes so we don’t forge

  • error handling checking and improvements in the backend

Next steps

  • Work on https://github.com/centerofci/mathesar/issues/1604
  • Next meeting on Wednesday
  • We’ll continue going through the storyboard and creating todos until we do a first pass of the full storyboard
  • Then we’ll go through and do a second pass on the storyboard

2022-09-07 weekly meeting

Attendees: Kriti, Ghislaine, Pavish, Brent, Rajat, Sean, Mukesh, Dominykas

Weekly check-in and planning

  • Data Explorer - done, remaining polish will be folded into Demo Video Readiness
  • Table Inspector:
    • Design is almost done, waiting on data type inspector PR
    • Development almost done, PR in review
  • Column Extraction:
    • Sean will prioritize frontend
  • Record Summary:
    • Backend: Mukesh and Sean to discuss if there’s more work remaining. If there’s more work, it’s a priority for Mukesh
    • Frontend: Sean will prioritize
  • Record Selector: no change
  • Record Page:
    • Frontend PR merged, now 75% done
    • A bunch of polish and smaller issues left
  • Data Import:
    • Pavish will take over on getting the design spec merged
    • Pavish will work on the frontend
  • Schema Page:
    • Almost done, just need schema descriptions
    • Brent is working on the backend
  • Database Page:
    • same as schema page
  • Exploration Page:
    • Pavish is working on this
  • API performance
    • Mukesh is working on this
    • Prototype works well
  • DB performance
    • Dom is working on this, slow going since it involves entire codebase
  • Users & Permissions: no progress
  • Demo Video Readiness:
    • Kriti wrote up some notes
    • Meeting today to figure out next steps
    • We still have a bunch of features to build (column extraction, etc.)
  • Demo Video Production and Mathesar Website: no real updates to report

Data explorer navigation

Summary of decisions is here: https://github.com/centerofci/mathesar/issues/1601#issuecomment-1239754189

2022-09-02 Record Summaries in Data Explorer

  • Topic: Record summaries in Data Explorer
  • Participants: Brent, Mukesh, Pavish, Sean
  • Time: ~90 minutes

The discussion split into the following sub-topics: 1. If, and if so, how should Record Summaries be displayed in Data Explorer. 2. What should the default aggregation be, when the user clicks on the Summarize option after grouping a table, as per screen 31 and 32 of our storyboard figjam file. 3. Storyboard ordering for the Demo video. 4. Allowing operations across schemas in Mathesar.

1. If, and if so, how should Record Summaries be displayed in Data Explorer.

When a user adds the id column of a table in the Data Explorer, should we show the id directly as an integer or should we show a Record Summary? * Arguments supporting keeping id columns as integers and not showing Record Summary - Data Explorer is meant to allow the user form any query structure they want, and there are valid usecases where we may want to directly display the id, even if not the most common. - Users can add any column they want in the Data Explorer, which would make the purpose of Record Summaries moot within Data Explorer. - Internally, calculating Record Summaries requires running a query. In this case, we’d run a query within a query, which isn’t performant. * Arguments supporting showing Record Summaries as they shown are in tables - It would appear & function similar to tables maintaining uniformity across the product. - From the user standpoint, they can always expect the same behaviour, avoids confusion. - Users can configure templates for Record Summaries however they want, which can include additional text. We don’t want them repeating it everything they create an Exploration, with an additional transformation. * Decision - We show the Record Summary in Data Explorer. - However, we have an option to toggle it, in the column properties pane. - Since the user can add the same column multiple times, they can add the ‘id’ column which will show the Record Summary, and they can add it again as a new column and toggle summaries to be ‘off’ and that will show an integer. - We arrived at this after a lot of to-and-fro discussing several alternatives, we didn’t document them. - This will be hard to implement and we’d prefer not to do it for the live demo (and preferably the initial release even). * Next Steps - Since we cannot complete implementing Record Summaries for the demo video, we started Discussion #2 to figure out how we handle the summarization scenario from the storyboard.

2. What should the default aggregation be, when the user clicks on the Summarize option after grouping a table, as per screen 31 and 32 of our storyboard figjam file.

  • Option1: We select a single additional column and aggregate it to count.
    • Needs no additional work.
  • The fastest solution to the problem.

  • All of us in the call were in favour of it.

  • Option 2: If we have to show the columns aggregated to list, how would we do it?.

    • We identify the columns that are used in Record Summary, and select them in the query instead of the id columns.
  • When the user summarizes the checkouts table, they will see a list of patron first names, and last names (considering these are the columns used by Record Summary) in the Data Explorer results. The id column will not be part of the query’s initial columns.
  • This will provide them with similar information to Record Summaries.

  • This is the quickest way of implementing list aggregation for this user scenario, without needing Record Summaries.

  • Decision & Next steps

    • This is a product level decision.
  • Brent and Pavish will discuss this on the upcoming ‘Demo video readiness check-in call’ on Tuesday (2022-09-06).

3. Storyboard ordering for the Demo video.

The previous discussion yielded another one on the user flow in the demo video. We introduce Data Explorer in Storyboard screens 31 and 32 through summarizing a grouped table result. * Summarization is not the best way to introduce Data Explorer since it involves several configurations being displayed to the user, which may come across as Data Explorer being hard to use or unintuitive. * The primary usecase of the Data Explorer is to be able to join tables together. Our UX is designed to make this usecase the simplest operation. It would be better if we started with this. * We can include summarization & navigating via grouped table result after we first introduce Data Explorer with simpler flows. * All of us in the call were in agreement with this. * Next steps - Brent and Pavish will discuss this on the upcoming ‘Demo video readiness check-in call’ on Tuesday (2022-09-06).

4. Allowing operations across schemas in Mathesar.

We took some liberty discussing bugs when this came into question, which resulted from the discrepancy in scoping queries between the backend and the frontend. * Currently queries on the backend are not scoped within a Schema. They exist directly under the Database level. * The UX treats Schemas to be the parent of Queries. * This is a product level discussion. * Next steps - Sean will send a mail regarding this.