--- /dev/null
+We considered a number of services that you can use to recover your data in the event that you accidentally delete it, our hardware fails, or some malicious party finds a way to corrupt it.\r
+\r
+'''For now, we don't have any backup services. All of the text below describes techniques that we tried and stopped using for various reasons.'''\r
+\r
+= On-disk backups =\r
+\r
+For the sake of the first two categories above, we maintain multiple "snapshots" of the filesystem in previous states. You can browse through these at your leisure and retrieve old versions of files. At the moment, we archive only `/home` (where most user-associated files live, thanks to our attempts to keep everything users can modify significantly on this partition) and `/etc`.\r
+\r
+This will save you if you delete your files accidentally, but the old versions were intact at the time of one of our 6 most recent hourly backups, 7 most recent daily backups, 4 most recent weeklies, or 6 most recent monthlies. These frequencies are the defaults in the Debian rsnapshot package, and we'll see how well they mesh with the realities of disk space usage, and probably adjust them accordingly.\r
+\r
+If our hardware fails, it's likely that the failure will be in at most one of our RAID disk pair at once, in which case the backup information will remain stored safely while we copy the filesystem onto a replacement disk.\r
+\r
+We use the [http://www.rsnapshot.org/ rsnapshot] program to manage these backups incrementally. This handy rsync-based approach uses hard links in a way that allows backups whose size is mostly proportional to the size of files that have ''changed'' since the last backup. You can find the current snapshots in `/backup`, where, for example, `/backup/hourly.0` is the most recent hourly snapshot, `/backup/daily.2` is the third most recent daily snapshot, etc..\r
+\r
+'''/backup is currently unavailable and will remain so until someone figures out how to mount the same directory in both read-only-for-users and read-write-for-root-only modes, when that directory isn't the root of a partition. We currently don't know the right firewall configuration to allow a self-NFS-mount, which is the recommended method on the rsnapshot site.'''\r
+\r
+= Off-site backups =\r
+\r
+If some unauthorized person ever finds a way to get root access to one of our servers, he can trash any on-disk backups. Moreover, even if we ever switch to storing that kind of backup on a remote filesystem, it's probably true that root has access to write to that filesystem arbitrarily, since we want the backup process to be automatic. As a result, we need some backup storage that even root on our own servers can't modify.\r
+\r
+Our current approach to this is ad-hoc. About once a month, an admin tars and compresses all of `/home` and `/etc` and transfers it to some other location. The resulting archive file can be burned to CDs or similar.\r
+\r
+Naturally, users can't automatically restore their files from such an image. However, this should only be a problem if someone gets root access on a server, in which case it will be a problem for everyone, and admins will be very involved, anyway.\r
+\r
+Suggestions are welcome for a more automated solution, or a solution that involves paying a company a reasonable price to do this.\r
HCoop / hcoop / zz_old / ikiwiki / blobdiff