Do Not Disturb and SQLite data storage

Hello! This week's email about building Lotus is really exciting for me. First, I renamed Weekend Mode feature to Do Not Disturb and built it. But more importantly, I migrated from Redux-like state management to storing all the data in a SQLite database. There's lots to talk about, so let's dive right in!

Do Not Disturb

You may remember that I've already talked about Weekend Mode feature in one of the earlier newsletters. Now it's called Do Not Disturb to make it sound more generic, because I decided that it would be useful to ignore work-related notifications after 5pm as well and not just on weekends.

To set it up all you have to do is to add an organization or a repository to Do Not Disturb list and notifications from these will be silenced.

GitHub's API isn't as flexible as I hoped it would be, so this functionality is a combination of three separate searches:

Fetch all organizations you belong to and filter them using fuzzaldrin-plus
Fetch all repositories where you are listed as a collaborator and filter them using the same fuzzaldrin-plus library
Use GitHub's search API to search your own repositories

All of them are cached as well to avoid over-fetching and potentially hitting rate limits on the GitHub's side. After you add a repository to the list, this is how the settings look:

Icon on the left indicates whether this is an organization or a repository. Note that ink and pastel are my repositories and Lotus is kind enough to remove my username from the label.

When Do Not Disturb is activated, you'll also see a neat moon icon on the Inbox page with a tooltip that explains which notifications are hidden.

I'm quite happy with how this feature turned out, but there are still to-dos to make it complete:

Disable Do Not Disturb on demand to have an escape hatch to see silenced notifications temporarily
Handle rate limiting from GitHub API
Animations for showing the moon icon, tooltips and popovers
Customize the hour after which Do Not Disturb is activated, currently it's hard-coded to 5pm

SQLite data storage

Until this week, Lotus had a primitive solution to data storage - a big JSON blob saved in the file and synced back and forth between main and renderer (UI) processes. In the UI, state was managed in one huge useReducer React hook, which is basically Redux.

It let me prototype Lotus quickly and change how data is stored immediately. However, I realized that it won't serve me well long term and that JSON blob will only get bigger over time. That's why I decided to migrate to a proper database - SQLite.

SQLite is a variant of a SQL database where all of its data is stored in a single file, making it perfect for macOS apps. I use better-sqlite3 library to interact with SQLite database and I'm impressed with out-of-the-box performance and how easy it is to use.

I'm glad to say that after this week's work all Lotus data is stored in a local SQLite database! Data is no longer kept in two places and synced all the time. Now it's stored in a database and UI only pulls what it needs and when it needs to, making the app more efficient as well.

Because SQLite is still a SQL database, it means there has to be a schema for all the data I want to store. This presents an interesting challenge of database migrations. For example, Lotus v1.0 has tables notifications and users, but Lotus v1.1 requires a do_not_disturb table to exist. Now Lotus needs to detect this case and "upgrade" your database to a new schema.

Thankfully, I've already built a custom library for database migrations in my other project Linkjar, so I just extracted it from there and added into Lotus. Now all data migrations are automatic between Lotus versions, which is quite cool.

Now that the biggest technical roadblock is out of the way, I can focus on building the actual user-facing functionality. Next week I want to finish Do Not Disturb feature and start implementing the Screener. So excited for this one!

Building a dropdown for searching GitHub repositories in Do Not Disturb settings reminded me how creating an accessible UI component like this is still so complicated to do in the JavaScript world. After a lot of googling, turns out there are no actively maintained and well documented libraries to add such a dropdown, so I had to code it from scratch using Downshift and Popper.

Granted, these libraries made my life easier, but I still think it took way too much time than it should've. Same thing applies to quite a few of other areas of Lotus codebase, so I keep thinking about extracting stuff and releasing it as open source. Other potential candidates for extraction are:

Electron + Webpack + TypeScript boilerplate
SQLite database migrations
Setting up separate Webpack bundles for each app window
Creating windows with good defaults and restoring their last position
Draggable window titles when title bar is a custom one
Receiving shortcuts in the renderer process from app's menu

I'm sure there will be more as I move forward. While I'm not going to extract and release these now to ensure I'm not distracted from Lotus, I think it fits my goal of sharing knowledge as I build this whole thing.

I remember thinking "what can I write about" or "what is something I know that I could share" and having a hard time answering these questions. I even have the same thoughts when I write this newsletter every week! Well, it's so much easier to answer these as you're doing something. Turns out the recipe is simple - build, learn, share, repeat. Whether you actually stick to that routine is another topic, but at least the steps are there for you.

Thank you once again for sticking with me here! I hope you enjoy seeing small incremental steps in building Lotus and watching it become a "grown-up" app.

Until next week!

Vadim.

P.S. You can now read past newsletters on the getlotus.app website.