Skip to content

2022-10-19 weekly meeting

Cycle 5 progress check in

Participants: Entire core team

We’re making progress, lots of green in the spreadsheet.

Discussion

  • Bugs should be closed if we can’t reproduce them easily, see discussion on mailing list
  • Error “cursor already closed” moved to urgent priority since it seems likely to occur in the live demo
  • DB page & schema page styling PRs are open
  • No usability testing this week since styling PRs are not merged yet
  • Mathesar website update:
    • Second round of copy has been reviewed, final draft is pending
    • We have someone to help with demo video production, we’ll start that process in early November
      • The idea is to record the demo ASAP after Nov 22.
    • Kriti may recruit people to help with script review or other video production tasks

Aggregation/grouping behavior for columns from base_table

Summary: This discussion is to resolve questions on how to decide which columns to group or aggregate when a summarization transform is added, and this depends on the grouped column. We also intend to resolve how we’d handle columns from tables with one-to-one relationship with the base_table/table of the first column. The outcome will impact our UX. Participants: Pavish, Dominykas, Brent, Kriti, Sean, Ghislaine

We discussed this for a while, mainly to help us all understand the problem better. Conclusions and next steps:

  • Design should include selecting multiple grouping columns / which columns are grouped vs. aggregation
    • Design feedback: Checkboxes are not the best way to separate the columns into two groups
  • Backend will apply sensible grouping by default, the user can change it based on the design above
    • Pavish, Brent, Dom will discuss separately and determine implementation details
    • Algorithm: columns involved in Unique constraints and PK columns / one-to-one should automatically be in grouping columns

Automatic aggregation

Summary: This discussion is to understand how we plan to implement automatic list aggregation, such as whether it would use a join or a subquery. This would depend on the first added column. The outcome will impact our UX. Participants: Pavish, Dominykas, Brent, Kriti, Ghislaine

Not much discussion here. Since the aggregation depends on the first added column, we need to help the user choose one from the base table. Next steps:

  • Design: First column MUST be from the base table
    • A concern was raised that this might annoy the user, but they can change the base table if they want to start from a different column
    • We’re helping the user set up their query the right way by constraining options
  • Design: Ensure that columns from base table can’t be deleted
    • at least one must remain

Overriding SqlAlchemy metadata reflection

Summary: SqlAlchemy reflects unwanted metadata which leads to a lot of queries when reflecting a Table. This becomes a bottleneck when reflecting a lot of Tables and since SqlAlchemy does not reflect all the tables at once, these queries stack up and can sometimes lead to 100+ queries. We can override the reflection function to allow us to selectively choose the metadata we want to reflect. This could lead to a reduction of 20-30% of the queries. Participants: Mukesh, Dominykas, Brent, Kriti

Discussion:

  • This is not worth the trouble.
  • We don’t want to override framework behavior, and we don’t want to further invest in metadata at this time.
  • There was lots of disagreement on how slow the API actually is, but the conclusion was that we can throw hardware at the problem for the live demo.
  • We’re not prioritizing really fast performance for the first release.