Over the past month, I have been quite busy working on the Notification service, which encompasses various types of notifications such as direct mentions, shares, etc., along with real-time updates. In this release, I have gained a lot of experience that I would like to share with you.
Firstly, let's discuss some of the experiences I gained. In the Tacitbase product, we had already implemented the notification service six months ago. At that time, we lacked proper implementation and the selection of the right architecture. Therefore, we opted for Kafka for publishing notification events. We created topics based on users and private resources, and published relevant events on particular topics so that subscribers would be notified of events they had subscribed to.
In this approach, we managed read and unread notifications in Kafka using partitions, which turned out to be a huge mistake. Here, we lacked proper documentation reading and didn't utilize tools effectively. After one month of releasing this service, it was too slow to deliver notifications and real-time updates. We encountered numerous problems in the notification service, and Shekhar pointed out that it was our mistake for not properly architecting this service. So, we made the decision to revamp the service and architect it properly this time.
Firstly, we conducted some calculations, finalized requirements, and decided that Kafka was not necessary at this moment since our user base wasn't as large as needed to process events in millions or billions. So, we started from scratch. We already knew a lot of things such as Pub-Sub, WebSocket, and OPA (for authorized users). We understood the flow of the service.
Therefore, we opted for Redis Pub-Sub for publishing real-time events. However, we planned for the future – in the coming days when clients increased, it must be easy to shift from Redis pub-sub to Kafka if needed. The difficulty here was that we needed to write a connector for the pub-sub client so that the shift could be made easily.
So, I learned a lot of things in this release. We will briefly discuss how I solved these problems, the service architecture, event process queue, WebSocket connection, pub-sub clients, cache mechanism, and goroutines, channels.
Stay tuned for the next blog. Here are some key points:
Choose tools according to your needs instead of trends.
Focus on problem statements
Interesting article.
Looking forward to next part. The illustrations is really pretty.