Alternative version control for NextWiki (Changedir)

NextWiki's current changes.log system is OK, but not great. There are significant issues with it, which basically mirror the problems of the mbox format:

• If two people come in and edit pages, even unrelated pages, simultaneously, there's a potential for corruption.
• The obvious solution to this is file locking, which will instead cause one of the changes to fail.
• Another issue is that it's not exactly optimised for "random-access," or checking out old revisions. (For diff, etc.) The changes index file helps with this a bit.
• One other slight problem is that the "current revision" is stored in two places on disk, in the changes.log file and in the actual file which gets served. This is a bit wasteful.

An unrelated problem NextWiki currently has which my proposed scheme could also solve is atomic rewriting of pages, so that people don't get blank or incomplete pages when they load them in the middle of a change.

My solution is to convert the changes.log file into a Changedir, somewhat similar to djb's Maildir scheme.

Outline of a Changedir

A changedir contains a directory for every file under revision control, in which each revision of that file is stored. It also contains a directory called .tmp and a directory called .all. The rôles of these folders will be explained later.

Each directory for a file has a name which is the same as that file itself, and contains timestamp-named files. (I recommend ISO 8601 for this as it sorts correctly ASCIIbetically.) Each of these timestamped files contains the entire source of a version of a file. (I'm not sure where to store metadata. More on that later.) Writing to the version-files is done atomically, as we'll see.

To make a revision to a page and store it in revision control, the PUT handler should do this:

1. Create a uniquely-named file in .tmp. It doesn't matter what the name is as long as it's unique, but in practice it will probably be a timestamp together with the pathname of the page, and perhaps the author.
   
2. Write the source of the page to this file. fsync and check that it was written properly.
3. Hard-link the file into the revision directory, under a timestamped name, again making sure that it worked. Then overwrite the existing file in the site itself with a hard-link to the same.
4. Unlink the file from the .tmp directory.

This brings up the question of how to deal with commit metadata. The best idea I'm able to think of is to create a separate file for each revision for commit metadata. Those should be stored in .all, so a "recent changes" page can work for all pages on the site.

This system takes advantage of filesystem hard-links, so each revision is only stored on disk once and used for both public file access and for revision control.

Example

There's a file called info.html which bob creates at 2013-02-27T10:23:43Z. When he creates it:

• The directory Changedir/info.html is created.
• The source of his revision is written to Changedir/.tmp/uniquename. That file is then hard-linked to Changedir/info.html/2013-02-27T10:23:43Z, and to info.html. Then it's unlinked from .tmp.
• Metadata is processed in a similar way — perhaps actually in advance of the source being written. (Potentially dangerous if writing the metadata works but writing the source fails — need to consider this.)​ It's linked to a file called something like Changedir/.all/2013-02-27T10:23:43Z-info.html.
• Success is reported.

When john comes along later to edit info.html, the process is much the same, but this time Changedir/info.html doesn't need to be created, of course, because it already exists.

Problems

OS X still kinda doesn't like colons in file names; a relic of the System 1 days. Could be solved by breaking ISO 8601 a bit and using hyphens instead of colons.

Still doesn't solve the problem of nuking other peoples' edits any better than ^B does.

Further versioning thoughts

• Diff? Blame? Aaron's HTML Diff is OK but apparently slow for large documents, and perfectly happy to churn out code like <ins><p class=a></ins><del><p class=b></del> ... </p>. In my (admittedly limited) experience the similar HTML Diff by Richard Cyganiak and Ian Bicking is better in both regards.
  
• If we let people look at old revisions, how to let people know that it's an old version? This is a problem of storing the whole page source for each page, instead of just the content to stuff into a template stored elsewhere. The latter system is too complicated for this, imho …

Further non-versioning thoughts

• "What Links Here" is a non-trivial problem without some kind of a database …