Attendees: Brent, Dominykas, Mukesh
We realized that there’s a real feeling of time crunch that makes Dom reluctant to request code review, and all of us reluctant to spend enough time on doing the review for large PRs, or even moderately-sized ones. Turnaround is quite slow due to TZs, and even getting a fairly easy PR merged can take multiple days, which is a problem in our current situation. This is why Brent is going to push for sync review when when he feels his willingness to spend the time on it waning. It will be a time-saver in the long run that way.
Dom thinks we should stick with having the author decide whether PR will be reviewed or not. Brent agrees, but thinks we should be more pro-review that previously.
Mukesh advocates for TDD. He thinks that it tends to lead to smaller PRs, since you end up (usually) with checkpoints where you can merge. Might be worth considering for those advantages, but can also be constraining.
Idea: use TODOs to avoid getting sucked into larger projects, and also to avoid breaking flow to create issues.
Dom’s action items:
- Create TODO comments in code as working, collect into issues at a later time.
Brent’s action items:
- set up sync call when noticing demoralization while reviewing
- Also use TODO comments
Mukesh’s action items:
- Continue using TDD, help others understand where it’s most useful.
Attendees: Brent, Dominykas, Ghislaine, Kriti, Pavish, Rajat, Sean, Mukesh
We made progress on the following:
- Data Explorer (backend and frontend both over 90%)
- Navigation (design finalized, implementation almost complete)
- Table Inspector (design almost finalized, implementation starting soon)
- Record Summary (backend at 85%)
- Record Selector (frontend at 85%)
- Filtering (design finalized, implementation in progress)
- Data Import (design in progress)
- Styling (Ghislaine has started work)
- Data Modeling Suggestions (requirements worked out)
- Dynamic Defaults (requirements and backend plan worked out)
- Mukesh assigned to Column Extraction front end
- Dom will switch to frontend issues after Sean finishes Navigation
- Brent will focus on general backend support for Cycle 3 and take over any work from Dom or Mukesh as needed
- Dhruvi may need extra work, we’ll figure that out during the week since she’s not here today
- Data Explorer
- Brent and Dom will meet to discuss bug
- Table Inspector
- Ghislaine, Kriti, Sean, Rajat, Pavish will meet to resolve open questions
- Ghislaine, Sean, Rajat, Pavish to meet tomorrow once Ghislaine has her branch ready
- The goal is to discuss how to structure CSS in our codebase
- Column Extraction
- Spec should be ready by Friday to unblock Mukesh
We decided to extend Cycle 3 by 1 week to Aug 26. Since Record Page and Data Import are still at 0% and won’t be started until the end of the next week at the earliest, giving ourselves another week should get us much closer to completing our work.
This is a one-time extension, Cycle 3 will definitely end on Aug 26.
Topic: Backend planning for Dynamic Defaults feature
Attendees: Dominykas, Brent, Kriti
- Create a Django Column DD field that holds a DBFunction
- Have it define Postgres column’s DD
- Store dynamic default (DD) in Column Django model
- as a DBFunction
- Make sure frontend knows whether default is static or dynamic
- Don’t declare which DBFunctions are good as DDs
- for the sake of implementation speed
- Hardcode DBFunctions to use on the frontend
- for the sake of implementation speed
- Don’t reflect third-party defaults, only defaults created in Mathesar
- to be clear, we can reflect an SQL always
- it’s just that we can’t offer a nice UX for third-party DDs
- Once we want to provide “non-malicious tampering” guarantees (in the future)
- meaning guarantee that a third-party has not accidentally messed with it
- we can wrap the DD SQL expression in a special SQL function that
- has an argument holding the JSON for the DBFunction
- has an argument holding the SQL expression for the DBFunction
- Extend table columns endpoint’s “default” sub-dict
- to describe whether it’s a first- or third-party DD
For Cycle 3:
- Implementing the plan, but not tampering guarantees
- Making tickets for tampering guarantees
After Cycle 3
- Implement tampering guarantees
- Extend the system to handle more types of defaults and sequences
- Could be good first-time contributor tickets once infrastructure is set up
- Dynamic defaults likely won’t reflect in the same form as created
- Some are created as python functions, reflected as string
- Could be optimized in unclear ways
"value": "nextval('\"Library Management\".\"Publishers_id_seq\"'::regclass)",
- Guarantee wrapper
- showing non-dynamic, dynamic (Mathesar), dynamic (3rd-party) for API
We are considering implementing Dynamic Defaults as it increases the usability of adding new records.
For the purposes of the demo we need the following defaults
- Current date and time
- Current date and time + a specific duration (e.g. 14 days)
NOW() + '2 weeks'::INTERVAL
The dynamic default options will depend on the field’s data type.
We should consider implementing dynamic defaults using the same formulas that we plan to implement in the future
- User selects a column
- The inspector panel displays the properties for that column
- User toggles the “Default Value” option
- The user selects from a list of preset options
Formulas to use from our Formulas
- Current date and time: “Current date & time” formula
- Current date and time + duration: “Add Duration to Date” formula
- A specific date and time: static default (not dynamic)
Topic: Resolve product and design questions for Brent’s upcoming work
Attendees: Kriti, Ghislaine, Brent
“Suggest Candidate Columns to Create Separate Table” internship report
- Better to extract too few columns or too many?
- How fiddly should the feedback loop of deciding which columns to extract be?
- Brent’s proposed process:
- Suggest set of columns to extract
- Explain to user why those are useful
- User adds or removes columns
- Analyze in second round
- Alternate process (more manual):
- Have the user guide more of the column extraction
- Brent’s first proposed process is preferable from a UX perspective, having the user make repeated decisions is not ideal
- Second round is not necessary, let’s stick to one round.
- It’s better to be conservative with suggestions, better to suggest fewer columns than potentially unrelated data
- Eventually explore ProbComp integration
- Number / Date & Time (non-categorical data) should be avoided during suggestions, since it’s too risky and will involve additional statistical analysis to get right
- Not even if they’re directly correlated to categorical data columns (e.g. if there’s a person’s name and birthday, we only suggest name, not birthday)
- Brent will check for easy wins if there’s not too much effort involved
- Schemas will now have descriptions, stored in DB comments
- Mathesar won’t own the description, other clients can edit it
- UX should make it clear that this data can be edited outside of Mathesar (maybe mention that it’s stored as a DB comment)
- We should also do this for tables and columns at the same time, since it’s a very similar implementation
- We can easily get the count for both Mathesar-specific data like Explorations and for DB objects
- We can be more precise with DB objects
Topic: Resolve major questions remaining before finalizing the table inspector spec
Attendees: Ghislaine, Kriti, Sean, Rajat
We took a quick look at the current spec and content.
- Do we agree or have any concerns with the proposed selection modes?
- Selection triggers might not be the best way to trigger inspector modes.
- A tabbed panel that groups properties by level (table, column, record, cell) might be better.
- Column and Row information will be derived from selected cells.
- The selection status would change based on active tab
- Are we planning to provide keyboard and touch device interactions for selections in this cycle?
¶ Contextual Menus and Inspector Interactions
- Will the user be able to trigger the inspector panel via contextual menu actions?
- Yes, but will not be part of Cycle 3.
- We discussed the concept of adding an option like “Show Column Properties” in the inspector to open it with a chosen tab.
- Do we need to incorporate contextual menu actions into the spec?
We want to do this, but not in Cycle 3.
¶ Defining Actions and the Actions Panel
- Do we agree with the proposed use of actions to represent data transformation or modeling tasks?
- Yes, we agreed
- We need to make sure actions are consistent, e.g. Set to NULL is probably a property because it’s related to the cell value
- Do we agree or have any concerns with the proposed inspector states and triggers?
- Not selection triggered
- User can toggle
Out of Scope (for this spec / Cycle 3)
- Context menu changes for opening the inspector
- Keyboard and touch interactions
- Selecting disjoint cells and columns (will defer to keyboard interactions)
For the sake of brainstorming, I present a rough design which unifies the selection behavior for cells, rows, and columns. I’m not attached to this design, but rather putting it forward to demonstrate an alternate approach that may lead to even better design ideas collaboratively.
- Cells can be selected – but rows and columns cannot.
- Multiple cells can be selected via the same UX as LibreOffice Calc. Clicking on a row/column header selects all the cells within the row/column.
- As with LibreOffice Calc, the row/column headers visually indicate which rows and columns contain the current selection.
- When at least one cell is selected, an “Actions” sidebar appears on the right.
- The Actions sidebar can be hidden and re-shown by clicking on arrows. Its visibility state persists in local storage.
- If the actions sidebar is hidden, then all cells have a context menu which shows the contents of the actions sidebar. Opening the context menu from a selected cell maintains the current selection. Opening the context menu from an un-selected cell changes the selection to contain only that cell.
- The actions sidebar contains collapsible sections for “Cells”, “Rows”, and “Columns”. Each section heading also displays the number of entities currently selected, e.g. “Cells (3)”.
- The “Cells” section contains actions like “Set to NULL”, “Set to DEFAULT”, “Set to Now”, “Copy Down”, “Copy”, “Paste” and other actions depending on type.
- The “Rows” section contains actions like “Delete”, “Duplicate”
- The “Columns” section contains all the UI currently in the column header menu, plus the new UI for extracting columns.
- To aid in the discoverability and usability of selecting multiple cells in nonadjacent blocks, the Actions sidebar also contains a button labelled “Grow Selection” which displays above the Cell/Row/Column sections. When the user clicks “Grow Selection”, the button visually pulses indicating that a special (and persistent) mode has been enabled, and the user can select cells as they would with
Ctrl pressed in LibreOffice Calc. A help message also displays below the button when grow-selection mode is enabled.
- Was blocked on summarization until today, unblocked now
- Made progress on setting up library dataset, pretty much done
- Not sure if it works on Windows, but that’s not relevant for Cycle 3
- Reviewed Maria’s work for Data Modeling Suggestions
- Meeting with Kriti & Ghislaine tomorrow to discuss product work
- Schema homepage on backburner since it’s quick
- Data Import work is moving along
- Deliverable will be a spec
- Filtering almost done, need to meet with Pavish
- Dynamic Defaults
- Thinking about implementation constraints
- Will meet with Kriti and/or Ghislaine to discuss product concerns this week
- Navigation spec should be merged today
- Table Inspector
- Draft of spec ready
- Need to meet with Rajat and Sean about deliverable
- Column Extraction
- depends on table inspector, will need separate spec
- Schema Page
- not yet started, hopefully this week
- Column Extraction pretty much done, awaiting final review from Brent
- Record Summary backend work is close as well
- Record Summary frontend work is not yet started, will take up the rest of the week
- Data Explorer will occupy time until next meeting, including Filtering
- Grouping & summarization in Data Explorer are next
- Data Import has not been started
- Currently working on introductory tasks, making progress
- Next task: Table Inspector
- Read spec
- Meet with Ghislaine & Sean
- Need to resolve Navigation work assignment (see next agenda item)
- Record Selector work is moving along
- Record Page has not been started yet, would be good to do after Navigation
Difficult for Sean to coordinate with Pavish due to TZs. Navigation work seems to involve fiddly modifications of lots of code that Pavish originally wrote, thus it would be quicker with coordination.
- Sean and Pavish find a better meeting time (difficult)
- Pavish takes navigation, give some work to Sean in return
- Sean and Pavish will meet and discuss in a separate call
Thumbs up all around
- Design constraint:
- We want to make sure that the user understands that they cannot navigate away from the page during the import process.
- A modal is one way to do this, but it’s just a suggestion
- It will significantly simplify code on the frontend if we can stop portraying import as asynchronous like we’ve been doing
- We discussed rethinking the import flow to allow the user to having more stopping points in the import process since it’s synchronous
- they can see the files they’ve uploaded
- they can decide to import a specific file into a table
- they can control the inference process and whether/when it happens
- We decided not to rethink the flow since it complicates Cycle 3 work and is irrelevant to our use case.
- We will need to update the import flow to account for importing data into existing tables (Anish’s work), so we can think through all this then.