Reesd is an easy to use redundant storage service accessed through the regular
scp command-line tool. Reesd is currently in Alpha and in this blog post I explain why I started to build it and what I plan to do in the not too distant future.
I have started to build Reesd when I wanted to use continuous archiving of PostgreSQL WAL segments. In short, the WAL is PostgreSQL’s write ahead log, i.e. all the changes that must be made to the database. For instance the log can be used to restore a database after a crash or to continuously update a stand-by server. In PostgreSQL configuration, it is possible to specify a command to “archive” the WAL segments, that is save them for backup and/or replication. The obvious solution, in my mind, was to use
scp is available on the machine where PostgreSQL is running, and use SSH. SSH in turn is used a lot when you want to administer securely multiple machines and I like the idea of services accessible through SSH. The drawback was that the remote machine (receiving the archived WAL files) could lose all its data.
Again a possible solution seemed obvious: create a proxy talking the SCP protocol. The PostgreSQL’s archive command would use
scp against the proxy, and the proxy would stream the uploaded content to multiple machines.
Often when I have that kind of idea, a tool with a very clear usage and a simple “specification”, I just want to start writing it. Indeed, while I try to avoid being sidetracked too much, my mind makes a compelling case for implementing the tool. At that time, I was writing a Docker image registry (and you could probably say there was also some sidetracking going on here), primarily for myself but I was hoping to offer some commercial private Docker image hosting. For that commerical offering I needed a database, and I wanted to make sure I could save the WAL segments. So the compelling case was that I could use the redundant file store for the Docker images, but also all the user account management, billing, payment processing, and web frontend code.
Actually I think an
scp-based service can offer surprising use cases that I don’t imagine right now. For instance I recalled later that Vim supports editing files through SCP (and Vim also supports transparent GPG or OpenSSL encryption of files). Some nice features could be built on top of it too, similar to Git post-receive hooks (possibly exposed via webhooks).
In itself, learning about the SCP protocol was interesting too. There is no specification of the SCP protocol though. It is whatever OpenSSH’s
scp is doing, and
scp itself followed
rcp. I found a blog post or two, and an implementation in Python. In addition to OpenSSH, that was enough to get a prototype proxying
scp calls in a day or two.
I then modified the code to add streaming (so a file being uploaded to Reesd doesn’t need to be received completely on the proxy before it is sent to the backends), and modifed a library to compute SHA1 sums to work incrementally, as the files are streamed. At that moment I had used enough time that calling Reesd a weekend project any longer would become inaccurate.
I have put a lot of work in Reesd since then. I hope to write some blog posts about some of the things I have done in more detail; in the mean time here are some bullet points:
authorized_keyfile. Indeed since every user shares the
scpUnix account, that file will become too big.
Most of those points are not as complete as necessary. In particular Reesd Beta will appear when the whole platform is deployed across multiple datacenters, and with PostgreSQL synchronous replication (the file stores themselves are already redundant). In other words it will be deployed as for production but
I can’t wait to run Reesd Beta. At that point, payments will be enabled. Some problems could occur but at worse only a few minutes worth of data could be lost. Some downtime could happen because some operations could still be done manually. It will also mean that I have launched my first MVP.
During Beta, I will probably start to implement an HTTP-based API to store and retrieve files. I will also probably resume my work on the private Docker registry. And most importantly I hope to receive feedback from early customers! If you have any question, comment or suggestion, or if you want some help to use Reesd, please get in touch with me ().