Supporting a database other than boltdb

[rajveerb]

I was hoping to know if there is a possibility of adding the option of using a database other than boltdb. The possibility of adding a database from cloud providers like GCP, etc.

[umputun]

Not sure what DB you have in mind, but either way you can implement your own storage and connect it via RPC. An example of a custom store can be found here

For remote intgration (i.e store-over-RPC) see engine/remote.go

[rajveerb]

I was thinking about using Postgres or Firebase. I am worried about boltdb not scaling beyond a point.

[umputun]

well, I still have yet to see any real use case with remark42 running into boltdb scaling issue. Prior to choosing boltdb as a primary store for this system I've used it for my work projects heavily. Those projects involve tens (sometimes hundreds) of million records. Even for such a size, I have not seen any significant performance hit.

The biggest remark42 instance I manage personally is about 200M in size. This is not a huge size but still the real deal. The typical queries (heavily cached by remark42 itself) usually perform under 1ms, and the cold requests may take up to 10ms.

see also related discussion #955

[rajveerb]

Interesting, if I decide to stick with boltdb, how difficult would you say horizontal scaling is. Since, its relying on a filesystem in a VM, I think.

On a separate note, where would you suggest I start from to understand the source code?

[umputun]

On a separate note, where would you suggest I start from to understand the source code?

I gave you links to the relevant parts above. If you meant the general understanding - the code is not that complex and easily readably, just give it a try. Open it in your favorite ide/editor and check what is in and how it does things.

if I decide to stick with boltdb, how difficult would you say horizontal scaling is.

If by horizontal scaling you really mean adding more remark42 nodes, this is not just boltdb, but others things as well, like external caching, auth-related things, and so on. Either way - all data is exportable to a simple json file and can be imported later on to any other store you need. This process is documented in the readme.