14 Aug 2011

File Coordination in Mac OS Lion

With Lion, Apple again did what it does best — solving problems that we've gotten used to working around for so long that we forget are problems.

But before that, it's interesting to quickly trace the evolution of file access. Initially, files were held captive to the machine they were stored on. Then came the cloud, as web apps, but it was a separate world by itself, so you had to choose between the cloud on one hand, and the convenience, power, and complexity of the filesystem on the other. Dropbox and its ilk unified the two to some extent, providing a filesystem and a web interface to the same data (and mobile apps, too).

Now Lion builds this into the system, and provides an API for apps to use, as opposed to Dropbox, which has to work with apps that don't use its API. That was necessary for Dropbox's success, for sure, but the more interesting question is: can do you better if you can get apps to use your sync APIs? Enter File Coordination, Autosave and Versions, new APIs in Lion. Here's how they provide a better user experience than Dropbox:
  1. Documents are automatically saved to iCloud, and when iCloud changes are synced down, the document on screen automatically updates.
  2. Files are autosaved periodically and when another app wants to access them. For example, when you drag a document to Mail, it's autosaved just before being attached.
  3. Writes are atomic -- other apps don't see inconsistent state, as long as they use file coordination. This works for packages, too. (Packages are directories that are presented to the user as a single file, like a .app.)
  4. Operations on giant directories are atomic as well, provided again that all apps accessing the directory use the file coordination API. If an app tries to move a directory that contains a file another app is reading or writing, it will wait for that app to finish whatever it's doing. Similarly, an app that wants to read or write something deep in a directory that's being deleted is asked to wait till the deletion finishes, so that it doesn't do a partial read, or write into a file that will be subsequently deleted.
  5. When one app has a document open, other apps aren't allowed to write to it and overwrite the first app's changes. If the other app goes ahead and writes to it, because it's running on a different device, or because it doesn't use the file co-ordination APIs, the UI updates immediately. The latter is currently possible, with FSEvents, but far simpler with file coordination.
  6. Apps handle open documents being renamed or moved. This is currently possible with file reference URLs, but many apps don't use them correctly. File coordination may make it easier for apps to do the right thing.
  7. Versions snapshots files periodically so the user can easily go back in time.
  8. When you have a conflict, the system informs your app of the conflict and lets it resolve it. After all, the app is in the best position to do so, because it understands its document format best. This is not possible in the Dropbox world.
  9. When an app wants to resolve the conflict, that file access is prioritized below other writes, so the app merges the most up-to-date version.
It's interesting to see how much better you can do when you control the stack and can get apps to adopt your APIs.


  1. The app may understand the format best; but will it play nice with export/import? Or does it hold the user's data hostage? The major deficiency in my view is that apps have MY data and developers are somewhat notorious about treating import and export to other applications as an afterthought at best or at worst as something to be actively avoided. Which would be fine if the world didn't change or they didn't get bored with supporting their app five years later.

    The strength of a file system with data files in xml rather than databases is that, at least in theory, one can access one's data independently of the application if need be or even if just plain desired. With apps holding the keys instead of the user this is not possible.

  2. Agreed 100%. That's an advantage of Dropbox over iCloud or most other cloud systems, like Google. The ideal seems to be for Finder to have an iCloud view where it shows you all your files and lets you open any file with any app.

    On the other hand, security is also a concern -- you wouldn't want every app to be able to access every other app's data for privacy reasons. I don't want a random app on my machine going through my email, bank statements, chat logs, documents, etc. Maybe the solution is to give each app a private area where it can store its stuff without risk of other apps accessing them, and a different place for user documents, like Documents/AppName, where you can say Open With . Note that even in this case, apps by default should not be allowed to read these files without user action.

    In general, access control and sync are mostly orthogonal issues.