2022-10-19 weekly meeting¶
Cycle 5 progress check in¶
Participants: Entire core team
We’re making progress, lots of green in the spreadsheet.
- Bugs should be closed if we can’t reproduce them easily, see discussion on mailing list
- Brent will handle closing AttributeError from columns endpoint
- Error “cursor already closed” moved to urgent priority since it seems likely to occur in the live demo
- DB page & schema page styling PRs are open
- No usability testing this week since styling PRs are not merged yet
- Mathesar website update:
- Second round of copy has been reviewed, final draft is pending
- We have someone to help with demo video production, we’ll start that process in early November
- The idea is to record the demo ASAP after Nov 22.
- Kriti may recruit people to help with script review or other video production tasks
Aggregation/grouping behavior for columns from base_table¶
Summary: This discussion is to resolve questions on how to decide which columns to group or aggregate when a summarization transform is added, and this depends on the grouped column. We also intend to resolve how we’d handle columns from tables with one-to-one relationship with the base_table/table of the first column. The outcome will impact our UX. Participants: Pavish, Dominykas, Brent, Kriti, Sean, Ghislaine
We discussed this for a while, mainly to help us all understand the problem better. Conclusions and next steps:
- Design should include selecting multiple grouping columns / which columns are grouped vs. aggregation
- Design feedback: Checkboxes are not the best way to separate the columns into two groups
- Backend will apply sensible grouping by default, the user can change it based on the design above
- Pavish, Brent, Dom will discuss separately and determine implementation details
- Algorithm: columns involved in Unique constraints and PK columns / one-to-one should automatically be in grouping columns
Summary: This discussion is to understand how we plan to implement automatic list aggregation, such as whether it would use a join or a subquery. This would depend on the first added column. The outcome will impact our UX. Participants: Pavish, Dominykas, Brent, Kriti, Ghislaine
Not much discussion here. Since the aggregation depends on the first added column, we need to help the user choose one from the base table. Next steps:
- Design: First column MUST be from the base table
- A concern was raised that this might annoy the user, but they can change the base table if they want to start from a different column
- We’re helping the user set up their query the right way by constraining options
- Design: Ensure that columns from base table can’t be deleted
- at least one must remain
Overriding SqlAlchemy metadata reflection¶
Summary: SqlAlchemy reflects unwanted metadata which leads to a lot of queries when reflecting a Table. This becomes a bottleneck when reflecting a lot of Tables and since SqlAlchemy does not reflect all the tables at once, these queries stack up and can sometimes lead to 100+ queries. We can override the reflection function to allow us to selectively choose the metadata we want to reflect. This could lead to a reduction of 20-30% of the queries. Participants: Mukesh, Dominykas, Brent, Kriti
- This is not worth the trouble.
- We don’t want to override framework behavior, and we don’t want to further invest in metadata at this time.
- There was lots of disagreement on how slow the API actually is, but the conclusion was that we can throw hardware at the problem for the live demo.
- We’re not prioritizing really fast performance for the first release.