TL;DR: I lost several weeks' worth of new users due to a bug in the VS Code extension. Implement real time error monitoring early, if you can. Also, let's chat on Discord.
The first week of July, a random update to VS Code changed the internal authentication system, causing the existing versions of GitByBit to go into an infinite loop right at the start. Basically, none of the users could use the course, until this issue was addressed. Oblivious to the issue, I was working on the upcoming Pull Request practice (which is super cool and was released last week, btw).
At some point, someone finally reported the issue on GitHub, so I was able to fix it and release a new version. But I pretty much lost several weeks' worth of new users, because they were not able to use the course.
When you realize that your project didn't work for the last several weeks.
Most of my other projects are normal web apps, served from my own server. Built with popular frameworks like Laravel or React, you could easily integrate the app with some real time error reporting system, like Sentry or Rollbar. So, once an error happens, I get notified immediately and then can upload the fix to my server and all the users will get the fix right away. Debugging the problem is also easier, because I have access to the server logs and can see what exactly went wrong.
However, GitByBit is not your normal web app. There's a VS Code extension and the web version, each having separate front and back ends. All of these are built with different technologies, so error reporting is not as easy as putting a Rollbar API key somewhere in the config. So far, I had all these places sending telemetry back to my main server, where it's stored in logs (and cleaned up every now and then). But so far, there was no monitoring system in place to alert me about errors in real time. So, I had to open and read this to find out about errors:
Not very readable, unless you speak Klingon.
One can monitor logs like this every so often, but unless it's right after release time, it's not very practical. Most of the time, nothing bad happens, so you stop checking the logs regularly. You might check them once a week or so, but then you get busy with other things and forget about it.
Actually, I do... But you also need to monitor the logs, not just collect them.
You assume that if something goes wrong, you'll hear about it from users or see it in the logs eventually. There is some truth to that, but it can take a while before you notice the problem, and by then, you might have lost a lot of users who couldn't use the course because of the bug. This is exactly what happened to the project in July.
Hey, Alex, aren't you a seasoned developer? Why haven't you implemented proper error reporting in the first place? Why did you wait until now to do it?
Nice question! This is the challenge of being a solo developer: there's so much to do, and you only have two hands (hopefully). You have to PRIORITIZE! Besides:
- I already had basic logs in place, so I could see what was going wrong if I looked at them.
- I've been busy with all the content work, so taking weeks to implement the backend infrastructure didn't seem like a huge priority.
- This is early access, so people kind of expect some issues and bugs. I was hoping that if something goes wrong, users will report it, and I can fix it quickly.
But now that the course is getting more popular, I guess it's time to have a better way to monitor errors and issues in real time.
PostHog to the rescue
I've been trying to de-google for several years now, but I still mostly use Google Analytics on my projects for tracking user interactions. Recently, I found out about PostHog and decided to give it a try. It combines several features that I need: event tracking, session recording, and error reporting. It also has a generous free tier, so I can use it in my early access project without worrying about costs for now. Also, they have a cute hedgehog mascot called Max, which I simply adore.
If you don't like Max, you are a bad person.
VS Code developer guidelines prohibit using your own analytics scripts inside the extension, mainly for performance and security reasons. However, the telemetry that you collect via the VS Code API is fair game, so you can receive it on your server and then re-send it to PostHog or any other analytics service. This way, you can still track user interactions and errors in your VS Code extension without violating the guidelines.
Errors are now collected in a readable format, so I can see what went wrong and where. Duplicate errors are grouped together, so I can see how many users were affected.
Each error contains a fair amount of detail that could help you debug the underlying issue. Once an error is fixed, you can mark it as resolved, so it won't clutter the list.
This solved the readability and tracking issue, but I still needed a way to get notified about new errors in real time. PostHog has a built-in alerting system that can send you notifications via Discord, Slack, or other channels when a new error occurs. This way, I can be immediately notified about any issues and address them quickly.
Now, if I hear a constant flow of BOOP!!! coming from Discord, I know that the course is broken. I can then quickly check the PostHog dashboard and see what's wrong.
Community server on Discord
Speaking of Discord, I'm testing out a community server. If you want to chat with me or other GitByBit users, feel free to join. It's a great place to ask questions, share your progress, and get help with the course. I will also be there to answer your questions and provide sporadic support.
Blog
GitByBit dev diary, thoughts and ideas about Git, GitHub, and the world of software development by the industry veteran Alexander Shvets.