So here we are - I hope you enjoyed the trip. Time Machine looks like a great step forward for preserving data. But we will have to keep an eye on Apple whether they are serious about it or not.

And then there are future questions, most of which were touched somewhere above: Quotas trying to fairly split backup space among several users? Dealing with small changes to large files (frankly, I consider Apple’s idea of simply splitting them up - as demonstrated for bundle disk images - to be a cheap hack rather than a good solution)? The apparent problems with applications like Aperture (no first hand experience)? Getting the ability to easily revert system updates you didn’t like with Time Machine? The numerous glitches which have been reported by people trying to do full restores (i.e. your data are saved but recovering them may not be totally fool proof). The whole issue of encryption. And in a next step file systems (there will always been someone mentioning ZFS which isn’t a reality yet but sounds like it might be a good match for Time Machine although it may require a number of changes to the concept). And in yet another step, networked file systems for backups to another location (with encryption being a must-have feature there). Which leads to more questions about which file server volumes Apple support (currently OS X.5 computers and Time Capsule devices by default and give or take a hidden preference for everybody who likes doing unsupported things with their backups). This is far less than they initially promised and it’s usually said technical rather than marketing issues cause the scrapping of that feature. Of course with Apple always achieving all their goals we will never learn what the real reason was. And while them limiting the features of their software just to shift more Time Capsule hardware would be a shame and really do injustice to a topic as important as backups, it wouldn’t seem completely along the lines of Apple’s ‘personality’ to do exactly this.

On the other hand there are still plenty of practical issues at this stage. Ranging from the - not necessarily Time Machine intrinsic - fact that OS X.5’s support for external drives could be much better performance-wise and even seems kernel panic prone (from my own experience ). Then there’s the sub-par UI, both for setting up the backup (no way tell Time Machine ‘include this specific folder in future backups’) and for recovering files (which is both tacky and giving much less information than would be helpful and possible). I’d say that technically those UI points aren’t biggies. And perhaps we can hope for someone givng us a better UI for it. With Apple’s personality being a bit too far on the self-consumed and narcissistic side to admit they shipped something less than ideal or - gasp! - poorly designed, I wouldn’t hold my breath while waiting for them to improve things.

The impression I get from this - my limited experience spanning a few months and the reports I read - is the following: Time Machine is worth having because it does make an additional copy of your data and is unlikely to accidentally destroy that additional copy.

Even though this text is both wordy (for sure) and rich in information (hopefully), it still doesn’t present a full coverage of Time Machine. Mostly because I am running a single machine in a single environment which means that I am not able to know about the full range of usage cases, the opportunities, the conceptual problems, the practical problems. All I can go by is word of mouth. Where word of mouth isn’t even that but just what I picked up from the garbage dump known as the internet.

Once you start thinking about network backups - possibly storing your data on servers of some third party on the internet - which hopefully becomes an option in the future, encryption of backups would become pretty much mandatory. Call me paranoid, but at least I wouldn’t want all my data to be stored in clear text on their machines.

Unfortunately Time Machine doesn’t offer to encrypt backups. Which is a shame - it will use disk images anyway and encrypted disk images do exist, so it seems like this would be an easy thing to do. Of course that extra level of encryption would require a bit of careful design about where the password for the backup image is stored on a shared computer. And it could lead to agony if someone ran their backup for years and ended up needing it without remembering the password. It’d still be a nice option.

The other aspect is encryption of backups. Usually backup software offers it as an option. And it makes a lot of sense to me. Even if you say that someone has to break into your home to steal your data and that you don’t need local encryption because of that, it seems entirely possible that you would (or rather, should) store your backup somewhere else. Put differently: There will be another copy of all your data now which you rarely need to access. Thus the inconvenience of the encryption plays less of a role and it would be nice to have.

If you really are more concerned about your data remaining private than about it being unaccessible, you are probably using FileVault on your machine. It stores your home folder in an encrypted disk image (these started off with a reputation for easily being irreparably broken in system crashes or so - I heard the format improved since, but never really checked that). And you get plenty of inconvenience for that - such as your home folder not being available via file sharing and possibly a bit of a speed hit. Oh - and you will only get very limited Time Machine goodies as well. Because, rather than saving the files in your account, the computer will only make a backup copy of your complete encrypted disk image if you are using File Vault. To make that a bit more efficient, Apple introduced a new disk image format, the ‘bundle image’ which splits its contents up in several ‘slices’. Meaning that Time Machine only has to back up the slices which changed rather than the complete image. This isn’t exactly neat but - assuming that they got the encryption right - it will keep your data encrypted even in the backup. But your only option for restoring will be to restore the whole image in one go, rather than the individual files contained in it.

Security seems a topic which they mostly ignored when designing Time Machine. If you consider your data to be private, you can be screwed in two ways by Time Machine.

And that’s before I get into real bitching mode, my friends! Let’s say you have a hard drive and a single folder on that hard drive which you want to include in your backup. If you naïvely assumed that this should be a ‘super easy’ single drag-and-drop operation in Apple’s OS, you’d be wrong. As far as I can tell you are expected to add all other folders on the disk in question to the exclude list and of course update its state whenever you add another folder. Excellent!

In fact those sheets are quite anaemic. They are tiny and all they let you do is select the backup volume and exclude stuff from backups. The excluded items from various volumes are then displayed alphabetically in a single list with no sense of their location. So if you excluded a folder named ‘Pictures’, good luck finding out which one it actually is. You’ll have to hover over the line in the list to get a ‘tooltip’ with a Unix style path to the folder in question. Very user friendly and Mac like!

With space being so generously used for the ‘pretties’, little remains for actual preferences. You mainly get a display of the current backup status, some information about how backups work and two odd looking buttons which in open sheets containing the actual settings. Just get that: There are a mere five settings they let you make for Time Machine and you need a whole preference pane and two sheets for them.

Time Machine also comes with its own preference pane in System Preferences. It features a huge logo with the name beneath it in Myriad which doesn’t really belong to OS X. It also features a huge ‘slider’ On/Off switch that is in no way a standard UI element and in no way behaves like the real world slider it’s supposed to represent. It works like a checkbox, so why isn’t it a checkbox?

The integration with applications also makes me slightly uneasy. The beauty of standard Time Machine backups is that they are a-file-for-a-file. It’s easy to understand what goes on there. And recovering a file from the backup is essentially a copy operation. The more ‘clever’ behaviour applications show here means that the transparency of what is going on is lost. I will have to hope that the software does the right thing.

As things are, the integration with Apple’s non-Finder applications isn’t first class anyway. Not only is half of the date at the bottom of the screen covered frequently covered by a button on my ‘small’ MacBook screen (a problem so obvious that it suggests nobody at Apple ever bothered to think about this properly or to try it out on such a machine), the general behaviour also seems to be quite erratic (iPhoto windows showing up empty at first and only filling eventually after I scrolled back and forth a bit, iTunes playback skipping while the computer made that ‘effort’, the lack of displaying actual differences in the Address Book, to give a few impressions I collected when trying things out for a few minutes).

Letting the application access its old database and restore the little bits and pieces from it is a sweet idea. But I wonder how many applications will offer that. I am sure that doing so may be quite a bit of effort. And doing that in a correct way for the rare situations when a user both needs it and knows the feature is there, seems to be a lot to ask.

I haven’t really needed this so far, so I can’t say terribly much about it. The idea behind it seems to be a laudable one. As many applications store their data in an opaque format which consists of many little files and perhaps some database keeping them together, it’s not feasible to do fine-grained restores for these applications. You can either revert the full database to a previous state - undoing many other, wanted, changes in the same move or you can’t undo anything.

Another feature of Time Machine is that applications are encouraged to integrate with it. Some Apple applications like the Address Book, Mail or iPhoto offer such integration (and applications like iTunes, iCal or Safari lack it). But I fear that many people won’t even notice that as you have to invoke Time Machine while the application is active to get the Time Machine interface for that application. And there isn’t an out-of-the-box way to invoke Time Machine for most of those applications if you haven’t cluttered your Dock or menu bar with its icons. Only iPhoto seems to have an explicit menu item for accessing backups at this stage.

The Mac OS X.5 installer also offers an option to restore the system from a Time Machine backup rather than setting up or migrating accounts. That’s a very appealing option if the whole hard drive breaks. I haven’t tried it myself yet but reports I read so far seem to be mixed. While no harm is done to the backup data, doing the full restore doesn’t seem to be terribly reliable yet. It may be a good idea to keep those cloned drives handy for a while if you need to be able to recover quickly.

Both these methods use the Finder for copying the files. And the Finder will strip the extended attributes relevant to the backup from them when doing that. Which is probably a good idea. Copying the files using the cp command leaves the extended attributes in place and you’ll probably want to remove them afterwards.

Or you can use the ‘official’ starry sky Time Machine GUI described above and grab the files from there if you have the nerves for that. You will even be asked whether you want to use the backup file to replace the newer version of itself or whether you’d like to have both files next to each other. With the files I recovered so far, both methods worked fine.

The most important thing about a backup is that you can get your files back if you need them. As a Time Machine backup is nothing but a folder with files in it, you always have the ability to just browse into the Backups.backupdb folder in the Finder and get the file you want.

Time Machine also comes with two UI elements: An icon which appears in the Dock by default in X.5 (Apple’s tendency to clutter your Dock with stuff starts reminding me of the Windows ‘Start Menu’ and Desktop which are littered by every installer - Apple just have a better monopoly to get more exclusive spamming opportunities for themselves) and a menu bar item which was added in the X.5.2 update. It lets you run a backup and launch the Time Machine UI. And it rotates like hell , the wrong way round, while files are being backed up. (The icons the Finder displays in its sidebar next to the icon of the Time Machine volume rotate the right way round, btw.)

Other aspects of the UI such as the navigation arrows could also be better in that what happens when you click them would be predictable. Right now clicking the backwards arrow will slide things around but I have no idea how far they will slide before seeing it. Even if it slides just ‘the right’ distance, why not indicate that in the UI beforehand by omitting the superfluous layers or greying them out? It’s not really clear to me anyway what all those Finder windows actually represent on screen. Usually they do not represent the state of the file system at the time the window in question represents. They just seem to be a copy of the frontmost window and will only be updated when they come to the front themselves.

… and so on. When playing with just the Finder’s Time Machine interface you quickly get the idea that some Sci-Fi geek had an UI idea which they then implemented but never really tried out. If you are using a large sidebared Finder windows in column mode, it will work all right for recovering a single file. In any other situation things quickly appear a bit broken and not exactly well planned and thought-through.

While not the biggest Bauhaus fan, I am a ‘form follows function’ type. And that concept certainly didn’t play a role when ‘designing’ the GUI for Time Machine. There are so many things wrong with it that I don’t know where to start or how to make sense of it. So I’ll just give a semi-random list of points to think about.

Not last but least there is the graphical user interface for Time Machine. Like many recent GUI ‘innovations’ by Apple it leaves me with a ‘WTF What were they thinking?!’ expression:

As Forrest points out in the comments , changing access permissions to a file is also asking for trouble: As permissions are the same for all hard linked versions of a file, the system would have to add that file to the backup again to avoid changing access permissions for all old instances of the file int the backup as well.. But that doesn’t happen. Interestingly, Time Machine does seem to catch a change of file owner and add a new copy to the backup in that case.

This may seem like a tiny point and a situation that shouldn’t happen. I think it’s good to have a clear sense of what the words we use are. And in the way Time Machine uses the word ‘changed’ it means that a file that has been moved to a different location has been ‘changed’ and that a file whose content has been replaced by other content of the same size in a way which didn’t modify its other metadata is not ‘changed’. As my usual understanding of the word ‘changed’ differs from that, it’s worth noticing and keeping in mind to avoid disappointment.

And thus I can create a file (I used a 1 by 1 gif image for a simple test), back it up, edit it (with that file not changing its size and dates in the process), run the backup again and have only the first version of my file in the backup. If my hard drive breaks now, the changes to that file will be lost.

And although the last changed date of a file is supposed to change when that file is edited, that isn’t necessarily the case. (In fact, with the shambles that are metadata on computers the ability of GraphicConverter to not modify the last changed date of a file when saving it is convenient as it will not cause converted versions of an image to have a date that differs from the original file’s.)

What is potentially even more harmful is the following behaviour: Time Machine considers two files to be identical if their path and dates and size are the same. I.e. it relies exclusively on metadata to determine whether or not a file changed, which ironically means it checks everything but the only relevant information. Luckily this isn’t too much of a problem in most practical situations, but it still strikes me as ironic. Having something like a hash of the file’s content could improve this a lot.

The file with the new name exists in the backup. But its link count is just 1. Thus it is a new file. The harm done by this varies. For a tiny file like this one it’s hardly worth mentioning. For a big video project where renaming may mean a wastage of many gigabytes of storage space just because it has been renamed, that may be more painful.

The first test is renaming. Renaming a file only changes its name in the folder but not its contents. Thus - keeping in mind all the hubub about hard links above - I’d expect Time Machine to create a hard link with the new name to the same old file in the backup when running the next time after renaming the file. So let’s try just that. Create the file

All this sounds fine and dandy. But you really have to keep in mind how it works: fseventsd only notes which folders have been changed. Then backupd scans them for files which have changed. And backupd doesn’t make a big effort when doing that job. Two obvious tests for its cleverness are failed with ease.

To do its job particularly well, backupd will check for any files that may have changed during a lengthy backup run and add those files to the backup at the very end.

Once the list of files needing backup has been collected, the process will copy them over to the Time Machine volume, creating all the metadata described above. And of course making sure there is enough space to accomplish the task before starting it. It also purges old backups when more space is needed on the backup volume as well as those which aren’t needed anymore (that is, hourly backups which are older than a day as well as daily backups older than a month).

Finally, there is backupd , the workhorse of Time Machine. It uses the information collected by fseventsd to zone in on the files which have been changed since the previous backup. If that isn’t possible because there has been a problem with fseventsd, that will be recognised [see that the fsevents store gets a new UUID in the log message above and that the extended metadata for the previous backup include the UUID of the fsevents store used] and kicks off a ‘deep traversal’ of the file system. In that case it will go through all files to ensure it is aware of all changes. In the logs that’ll look like this:

For this to work reliably it is crucial that the fseventsd logs are complete as otherwise files could accidentally be omitted from a backup. And thus if you ‘improperly’ remove a drive, say, because of the system crashing or being too daft to properly unmount it, fseventsd creates a fresh catalogue of changes the next time you use that drive. In the logs that will look like this:

But luckily Apple gave the System the ability to just know when a file has been changed in Mac OS X.4 where Spotlight needed it to know which files require re-indexing after being modified. This approach has now been extended in a way that the fseventsd background process monitors all file changes on the system and. It notes the folders containing files which were changed on a volume in a file inside the .fsevents folder at the top level of that volume. A more detailed description of how that works can be found in the Ars Technica X.5 review .

One thing that hasn’t been looked at so far is how the System knows which files need backing up. The traditional way of just scanning all the folders and subfolders on the attached drives would be quite an undertaking with hundreds of thousands of files hiding on a normal Mac OS X startup drive these days. Particularly if you want to update your backups hourly.

In case you want to edit it at a later stage you need a Finder window with a toolbar and an action/cogwheel menu in there. That menu contains a command for editing the search query. Thanks to Dirk and S for pointing this out.

The Finder lets you save elaborate search queries like this one as a ‘smart folder’ for easy accessibility later on.

[This didn’t work when I initially tested it and the text used to contain a more complicated way to create a query. Thanks to tinbert for pointing out that this does work after all and encouraging me to give it another try.]

If Spotlight’s graphical user interface weren’t a complete POS, this could even be fun. But currently it’s a bit painful to do. And given Apple’s past speed of improving the Spotlight UI, I won’t expect considerable improvements there before 2050. But still, we can attempt to do it. Just be sure to take this with a lump of salt. I have seen loads of strange behaviour when testing this. ‘Strange’ behaviour included the Spotlight server process mds crashing and happily starting a complete re-index after being automatically restarted by the System. It also included recently backed-up files not being in the Spotlight index and thus not appearing in the results for these searches. All that seemed a bit random, so while in principle possible, I wouldn’t rate these steps as reliable but more as an interesting exploration at this stage.

An interesting aspect is that extended attributes beginning with com.apple.metadata can also be found in the Spotlight index. In particular, this gives you a way to look for all files added to the backup in a certain time range. So if you sometimes wonder ‘what the heck caused my last backup run to take so long and to contain that much data, Spotlight might be helpful.

Time Machine also ties in with Spotlight. Items in the backup are indexed by the Spotlight engine to help you find stuff when needing it. The index for the Time Machine backup is stored separately from the remaining index of the volume, indicating that Spotlight has become a bit more flexible in this regard. I suppose separating things in this way may also improve performance for everyday searches in the non-Time Machine content of the volume.

Update: As this report points out , Time Machine’s default and unreported exclusions are even more problematic and potentially destructive, particularly when using Mac OS X Server and its unixoid features.

A point that is rather sore as well for me is the fact that Caches folders are completely excluded from backups. By Spotlight’s poor design the Caches folder contains a Metadata folder which applications have to store their data in if they want it to be found in the index. Those metadata files will not be in a backup. As a consequence restoring your system from a backup will leave you with an incomplete Spotlight index until you run all the applications which stored data in the Caches/Metadata folder and make sure they re-create them. Ultimately this is a Spotlight issue, I think, but with Time Machine being made by the same company, they should have had an eye on it.

While there certainly is a point to many of the exclusions made by Time Machine, I do find them slightly problematic. Particularly as many people will not know they exist and it is non-obvious how to find out which files will not be backed up. Even more so, as applications can easily remove your documents from future backups by accident (or, maliciously, on purpose). It’s a shame that there isn’t an easily accessible list of files which aren’t backed up. I guess the sufficiently paranoid may want to keep an eye on the property lists and attributes mentioned above.

In the way Time Machine currently works, exclusion is a vital concept for it. The software defaults to backing up a complete volume and you can control things a little by excluding folders from the backup. Say, to ensure that huge but frequently changing files - like disk images from a virtual machine or a large database - don’t clutter up the backup too much. There are various ways files can be excluded from the backup. (Don’t read too much into the order of that list as I haven’t checked which of these items takes precendence in case of conflicts. That might be an interesting thing to do.)

Time Machine also uses extended attributes on your working volume to mark files and folders as excluded from the backup.

With all these extended attributes in place, Time Machine’s archive is more than just a copy of the files. It also contains guides to match the backup up with local data and to see where the previous and next versions of a file are stored.

But there’s more to see for those who like poking around. Apple introduced extended attributes in Mac OS X.4 and Time Machine uses them a lot, both inside the backup copy and on your usual files. Mac OS X.5 comes with the xattr command line tool for reading and writing them.

Suggests that the ‘mach_kernel’ file has been replaced 30 backup runs ago. The same heuristic doesn’t work as easily for folders as you’ll have to subtract the number of items in them before knowing the number of hard links around. Also be careful when examining folders lower down in the file system. You will frequently see link counts of 1 there. That happens when no item in that folder or in any folder contained by it has been changed. Then Time Machine only created a hard link for the enclosing folder and thus every file inside it has only been backed up and hard linked to once.

When peeking around your backup you can learn a few things about it if the numbers are interpreted correctly. For example for files at the top level of your startup volume the number tells you in how many backups the file has been included. E.g.

Once such hard links come into play, you lose the sense of which one is the ‘original’ and which one is ‘just being pointed to’ - as both of them are the original. And thus, when browsing your Time Machine folders on the backup drive, all files will look like proper files and not carry the little superimposed arrow icon which Mac OS X normally uses to indicate aliases or symbolic links. But with the Terminal commands above you can hope to get an idea of what is going on.

And observe how the numbers change: The file with the hard link to it has an increased link count of 2. Finally, we delete the original file and convince ourselves that it survived thanks to the other hard link to it:

Up to here we created a file originalfile containing the string test and looked at the folder contents. There is a link count of 1 for the file and of 3 for the folder. Next, create both a hard link and a symbolic link to that file:

Perhaps a little demonstration will be a good way to finish this off. You can play around with the relevant commands in the Terminal without much effort, creating files, creating hard links with the ln command as well as soft links with the ln -s command. And keeping a close eye on the output of the file listing command ls -l . Its output contains a mysterious number after the initial permissions (e.g. drw-r--r-- ) bit. That number is exactly the number of ‘hard’ links to the file in question. Looking at these a bit you quickly realise that most files have a number of 1 and a folder has a number of at least 2 [the parent folder pointing to it and the folder pointing to itself - the ‘.’ item in the listing of the folder. Every additional item inside the folder will increase that number by 1].

Having multiple hard links to a folder turns out to be problematic once you start thinking about circular references and how deleting files should work: currently you can only remove a folder from the file system if it is empty; if you have multiple hard links to the folder it wouldn’t seem unreasonable to remove one of them without the folder being empty. But then imagine the only remaining link to that folder being from an item that is contained in the folder itself: You wouldn’t be able to reach that folder from the file system hierarchy anymore! In fact, the System’s ln tool doesn’t let you create hard links to folders yourself.

One way in which Time Machine goes beyond what is usually done is that it makes extensive use of hard links for folders. These are usually avoided as they require very careful handling in situations when multiply hard linked folders are nested within another. But the feature is essential for Time Machine so it can just use a single hard link as a reference for a folder and everything it contains instead of having to create actual folders for everything and hard links for every single file in each snapshot.

Just to get a better feeling for this, let me answer a few questions I wished you had asked:

In fact, that reference inserted to the folder is just a hard link. The new thing is just that once you have several such hard links to the same file, the file is listed several times in the file system’s folders but the actual data only exist exactly once. And that’s exactly what Time Machine uses.

Time Machine’s trick to avoid duplicates of unchanged files in its archives is to use hard links. Hard links are a very old and very basic concept. Yet they can be a bit mind boggling. When looking at how a file system works, you end up discovering that there are two things: The hierarchical folder structure and the files themselves. A folder essentially contains nothing but a bunch of references to the files themselves rather than the actual files: You can move even a huge file from one folder to another on the same volume in a split second because all that happens is that a reference to the file is inserted in the destination folder and it is removed from the source folder. The data themselves remain in the same place throughout.

Unlike many other backup solutions Time Machine doesn’t store backups in a single file backup database which may have an unknown format and may make it hard to extract data from it without the software that created it. Instead, Time Machine uses ordinary folders and files. That may be a bit less efficient space-wise but has the advantage of being simple as well as browsable on any machine that’s able to read HFS Extended volumes without Time Machine at hand.

At the top level of that volume Time Machine creates a folder named ‘Backups.backupdb’. And inside that folder there is a subfolder with the computer’s name - meaning you can back up different computers to the same Time Machine volume. Inside that folder there is a number of folders whose names are time stamps. These folders include the data of the drives Time Machine backed up at that time.

First, let’s look at the data storage: As things are, Time Machine requires a HFS Extended file system to work. This has been the default format for Mac hard drives for a long time, so it’s not a big deal. But if you don’t have a Mac OS Extended volume (but a drive shared by another Mac OS X.5 computer via Apple’s File Sharing or a drive in a different format), things will work as well as Time Machine creates a disk image in HFS Extended format and saves your data into that if necessary. In what follows I will just assume we are dealing with a HFS Extended volume.

Ideas about what a backup is vary a bit. The most obvious - and very widespread - interpretation of the word is simply making a ‘clone’ of your hard drive which you update from time to time. Obviously that’s better than nothing but once you work with, or think about, backups, you soon realise that it’s really not enough. At least not if you want to be able to do more than just restore your system in the case of mass deletion or a hardware failure. Say, some data corruption happens to a file, but you don’t realise it at the time. You’ll run your clone ‘backup’ that evening and your only existing copy of the file’s previous version will be lost. Ouch! Suffer from this once and soon you’ll join the people who consider a backup to be an initial copy of all your data plus fresh copies of all files that have been changed since the previous backup run. Time Machine provides just that and I shall try to give a rough sketch of how that works (note for the overly optimistic: I have neither written Time Machine myself nor seen its code, so I can only go from observation of Mac OS X.5 and the documentation).

There are two aspects to how Time Machine works: The user interface and the background activity needed to do the actual magic. Both of them are probably of equal importance as only a good user interface will entice people to create backups of their data and enable them to recover data which would be lost otherwise. But without the behind-the-scenes work nothing is going to happen. To keep up the good spirit, I will dig into the more obscure topic of background activity first before bitching about the user interface. This first part will be both lengthy and slightly technical. If you can’t take that skip it .

Which I wish they do. Simply because it will help people’s data survive.

As a consequence of all this Apple will have to earn the trust needed for Time Machine as a viable backup software. And while a lack of actual experience with the technology as well as initial software problems still dominate the reports at this stage, I think they have a reasonable chance at succeeding in the long run.

Idiot! I hear you scream. Sure, he should have checked the recoverability of his data. But keep in mind that .mac customers are exactly the people more technical minded folks consider as ‘idiots’. If Apple tells them they have a backup and essential parts are missing at the end of the day, it doesn’t exactly build trust. It wasn’t possible to reconstruct how exactly that situation came to be. But even if he turned off backing up documents himself, the software should have told him about the implications. Just as it warns you before erasing a drive.

And from a good friend of mine I heard a possibly worse report about .mac Backup. He had a hardware failure of his hard drive. Which he wasn’t too concerned about because he had run the .mac Backup application all the time. He got a new drive and restored his data. Just to find that all his photos and iTMS purchases were there. But, the most essential data, his documents, were missing. Panic ensued and a large amount of money had to be spent for hard drive recovery to rescue his thesis.

While Apple’s software isn’t generally destructive, they have had their run of installers deleting users’ files and they also had the .mac Backup application gaining a reputation for not being something you’d want to entrust your data to. There were reports about people being not being able to restore data from backups (i.e. failure at its single most crucial function) a few years back. That seeds a bad reputation which is deadly for a backup tool - particularly for one by a tight lipped company which will neither admit errors nor document the changes they make in updates.

All that said, Time Machine is also a product by Apple. And - as far as I can tell - Apple never went out of their way to protect their users’ data . The quality of their applications’ backwards compatibility varies widely (the OS X Address Book will export contacts [up to the photos associated to them] to the standard VCARD format, OS X Mail has no export feature at all, Mac OS X.5 doesn’t voluntarily play System 7 sound files, …). If Apple were self-confident enough to worry about their users’ data more than about locking their users in to peddle more machines, I’d certainly start off with a more positive impression.

None of these points is trivial to achieve. And - optimistically speaking - Time Machine is a brave attempt to address all these problems: Its mere existence raises awareness about backups. It creates and maintains backups automatically. It does so in a relatively simple way. And it makes restoring data easy.

Once you start thinking about it you will discover many potential points of failure for a backup solution. Things you need are:

Unfortunately, making backups is a bit of a hassle and thus of the few people who have a ‘backup strategy’ even fewer actually stick to it for a long time. It’s tedious, it’s boring and there is no day-to-day benefit from it as long as nothing goes wrong. Without ease-of-use and automation, backups don’t happen.

Often people don’t worry too much about this, simply because they aren’t obsessive compulsive. But everybody seems to have weak points. Once the half finished dissertation or the wedding photos go missing because of a broken hard drive, careless operation or theft, panic strikes. And the count of people who become aware of backups increases.

Unfortunately many people aren’t aware of all this. They work with their data every day and spread it across machines and systems without much care. Parts of their e-mail will be at some web mail provider, other parts of their e-mail are still at their old web mail provider - each provider having a different state and subset of their address book. Phone numbers are stored in the mobile phone only. And documents reside in the folder the application creating them suggested as a default. The documents created after 2005, that is, because back then old computer with all the older documents on it was stolen…

Backups are a sore topic. On the one hand, thinking about them is a painful and dull endeavor. On the other hand, not having a backup when shit hits the proverbial fan can be a disaster. The most important thing on a computer are your data. It’s not the machine itself. It’s not the operating system. It’s your files you will be worried about. Your work, your collected resources and work environment, your e-mail of the last decade or two, your photos and possibly your music collection or the playlists you created from it. Many of these things cannot be re-created if they were lost others would demand a massive effort to re-create which you wouldn’t undertake because it exceeds the ‘worth’ of the data in question.

Time Machine is the backup technology Apple introduced in Mac OS X.5. It is possibly the biggest new feature of the operating system. And I consider it to be a great step forward. In what follows I look at backups in general, a number of details of how Time Machine works and their ramifications as I can make them out today, after a few months of usage.

Wow, this is the definitive Time Machine review / postmortem / dissection!

Hopefully google will index it prominently.

March 5, 2008, 3:33

A minor note (hopefully Markdown will work here):

com.apple.backupd.BackupMachineAddress — containing a string of the Ethernet port’s MAC address (and I guess something else but similarly unique on machines without Ethernet like the MacBook Air)

AirPort cards also have a MAC address (you can see both wired and wireless in System Profiler, I think; I’m not at a Mac right now) so I imagine they just use that instead.

Otherwise, thanks for the detailed look, in particular the mention of FileVault and sparse bundle images, of which I was otherwise unaware.

March 5, 2008, 15:21

Me and the people I know don’t have a MBA at hand so I couldn’t verify that it really does use the Airport MAC address. If anyone wants to fill in that gap, I’ll be happy to learn about it.

This topic may be worth discussing as well: is a MAC address a good choice as an identifier? What will happen to my backup when I continue using my hard drive in a different machine? What happens when Apple need to replace my main board and thus change the MAC address? What happens when I change the MAC address through software?

I.e. is the backup really chained to the hardware the system happens to run on? And shouldn’t it rather be chained to something else. My gut feeling is that my system and data are relevant but the machine they are running on isn’t. In fact, when I switched MacBooks last year I simply moved the old hard drive into the new one. I suspect Time Machine would do a fresh backup after that although it seems a silly thing to do.

[BTW, you can simply see the MAC address in the advanced sheet of the Network preferences. Tougher people might just type ifconfig into their command line.]

March 5, 2008, 15:51

is it possible to get a list of what files/folders are backed up ?

March 5, 2008, 22:03

@bg:

I didn’t find a one-click way to retrieve all files in a backup. But if you read the section on Spotlight above, you should have a technique at hand to determine them anyway.

March 5, 2008, 23:34

Regarding hard links, you can examine if two files are linked together by using the command ‘ls -i’. This works on any Unix-y system as it corresponds to showing the i-node that the file is referenced by (called the “serial number” under Mac OS):

-i For each file, print the file's file serial number (inode number).

March 6, 2008, 3:21

Excellent review, Sven.

In theory, backups to Firewire drives should be less disruptive to foreground activities as Firewire is supposed to require less CPU supervision for I/O. I haven’t done any empirical testing.

March 6, 2008, 14:19

@d.w.:

Probably there are two points here: One is whether or not USB really was what caused my problems (I saw other people report fairly similar panicing behaviour with FireWire drives), but it doesn’t seem entirely unlikely.

The other issue is the eternal (?) FireWire question. It seems to be correct to say that, in comparison to FireWire, USB is crap. It’s less flexible and it requires more effort by the machine running it. However, I do wonder whether, these differences need to be noticeable when performing activities that is far from saturating the bus and has no real-time requirements. Does making many small reads and writes to a USB drive have to affect the computer’s performance. I like to think that it shouldn’t. And what I have seen on Linux or Windows machines suggests that it doesn’t necessarily make a difference. Certainly the magical ‘drivers’ used by the operating system play a role as well. And my impression is that OS X’s aren’t particularly performant (and may even have become worse over time).

Perhaps a requirement for the Time Machine implementation should have been “Time Machine background activity must not affect performance of other applications.”

I don’t know enough about the low level stuff to find out why it doesn’t feel right. I imagine it may even be quite tricky to pin this down. But then again, I’m a ‘Just Works™’ person, so I don’t think I should need to worry about this to begin with.

March 6, 2008, 14:40

I’m still working my way through the article, but while I’m thinking of it:

Mail has the ability to “Archive…” a mailbox as a standard mbox file. Highlight the mailbox (folder) you want to export, click the “Mailbox” menu and select “Archive…”

March 17, 2008, 21:33

Mail.app doesn’t have an explicit export feature, but the “Archive” function actually saves mailboxes out in standard mbox format (albeit rolled up into a Bundle-like folder).

Excellent write-up, though :)

March 17, 2008, 21:45

Nice in-depth text. The only thing I feel is missing is a test of how good it is at actually restoring files; lucky for us, I already did that, using Nate Gray’s awesome Backup Bouncer. Here are the results: http://mublag.boinkor.net/post/177807/Measuring-Time-Machine-backup-quality

Cheers, Andreas.

PS: As a stylistic note, I think this text as a technical document would benefit from leaving out much of the bile. I understand you find some elements infuriating - I do too - but it gets kinda old after the third paragraph detailing how much stuff annoys you.

March 17, 2008, 21:57

Are you sure that folder hard links are reflected in their link count? Last time I checked, I thought they weren’t, and none of the “I’m a link” flags seem to get set for these hard links. The best method I could work out to detect them was checking whether that inode occurred multiple times on the disk.

There’s no way to manually create hard linked dirs, as far as I can tell. Certainly ln refuses to hard link dirs, even when run as root.

Incidentally, that hard-linked structure plays merry hell with lots of file and disk utilities. Scans of Time Machine disks will take ages to complete (as its like scanning your hard drive n times over, where n is the number of TM backups). Even rm’ing can take a long time.

checksums in the metadata - that would be nice. I’m actually about to start storing my own (for a small subset of files over a threshold size, that have already been matched on quick checksums of the first and last 512 bits - typically about 1% of the files on disk), to optimise duplicate file detection.

I’ll be adding support to Baseline (see my link) for Time Machine aware scanning in an upcoming version, especially if that tip about the directory link count being accurate is right - I was using inode counting, but its a bit too slow for general use.

March 17, 2008, 21:57

Excellent overview of TM. I have been using it since the day of the OS X 10.5 release (yes it’s “OS X 10.5”, not the odd “X.5” you use).

I have only encountered a few random issues, but for the most part it works well. I’ve only now had my drive fill up, and am letting it delete the oldest backups. I agree that the TM interface is bizarro..I really wish they had a “power user” UI so we could skip all the starfield nonsense and just dig through backups efficiently.

March 17, 2008, 22:03

I wish TimeMachine was more flexible by allowing backups to be stored in a mounted DMG file. This way, I could create an encrypted DMG, and the problem of encrypted backups is at least ‘worked-around.’ I have a firewire disk at work I use for file storage, but there’s no way I could leave a plaintext backup there — too paranoid.

March 17, 2008, 22:06

Quotas are totally doable and work as expected. I setup another Leopard machine that had a spare HDD and I applied quotas for some network users on the volume. Finder appropriately reports to these connected clients the quota that I set, thus Time Machine respects this limit.

March 17, 2008, 22:14

While your technical analysis is thorough, I can’t say I agree with most of your user interface criticisms (and this is from someone who thumps on his Addison-Wesley “Macintosh Human Interface Guidelines” frequently). The key to understanding the Time Machine interface - the simple pref panel with almost no settings, the bizarre space restoration UI - is to get people to use it.

Getting people to back up - even with something as simple as Time Machine - takes effort. People have to know about the feature, it has to catch their interest, and it has to be simple. That’s what the Time Machine interface is. Earlier developer seeds had more options in the pref panel; this was simplified over time. The space UI is a crazy-ass thing that only semi-works, but for the most common cases it gets the job done, and it gets people to think about their backups. I’m willing to tolerate such insanity if it makes even a handful of people backup their data.

Please note that none of this is intended to defend the UI as a good UI, just that it has different goals. Also note that since Time Machine does use such a transparent backup format, there is nothing stopping a third party developer from creating a new UI - something more familiar and with more features.

A couple other notes: - The Time Machine menubar icon is correct for its purpose. The whole point is you’re “backing up” - thus the whole time metaphor, clock hands going backwards, and the arrow circling backwards. - The use of disk images on network volumes was excellent; not using them on local volumes was a huge mistake. Not only does it make it harder to migrate backups from one volume to another (or use a single disk for multiple machines), but it limits what you can do with the backup (as you noted, disk images can be encrypted, limited in size, etc). I go into this on my blog: http://diamondsw.dyndns.org/Home/Et_Cetera/Entries/2008/3/6_Time_Machine_Design_Decisions.html - Being able to include a specific file in an excluded directory is useful (I do this with SuperDuper), but the UI starts to get complex quickly (although Apple’s old and much-maligned Backup.app did a decent job of it). Apple really intends you to backup everything, and the approach reflects that.

March 17, 2008, 22:18

Are you sure that folder hard links are reflected in their link count? Last time I checked, I thought they weren’t, and none of the “I’m a link” flags seem to get set for these hard links. The best method I could work out to detect them was checking whether that inode occurred multiple times on the disk.

There’s no way to manually create hard linked dirs, as far as I can tell. Certainly ln refuses to hard link dirs, even when run as root.

Incidentally, that hard-linked structure plays merry hell with lots of file and disk utilities. Scans of Time Machine disks will take ages to complete (as its like scanning your hard drive n times over, where n is the number of TM backups). Even rm’ing can take a long time.

checksums in the metadata - that would be nice. I’m actually about to start storing my own (for a small subset of files over a threshold size, that have already been matched on quick checksums of the first and last 512 bits - typically about 1% of the files on disk), to optimise duplicate file detection.

I’ll be adding support to Baseline (see my link) for Time Machine aware scanning in an upcoming version, especially if that tip about the directory link count being accurate is right - I was using inode counting, but its a bit too slow for general use.

March 17, 2008, 22:18

Quotas are totally doable.

I have another Leopard (client not server) machine in the office with an extra HDD in it. I setup quotas for a couple network users and finder appropriately reports the quota that I set, thus Time Machine respects this limit.

You could do the same on a local drive for local users.

March 17, 2008, 22:29

Your snowman icon is the same as used by ssx, the hasn’t-been-updated-in-years Mac soulseek client. Care to explain the geneology?

March 17, 2008, 22:48

Someone has already pointed out that Airport cards also have a MAC address. Indeed, 802.11 devices are not technically 802.3 Ethernet, but most all applications, such as Airport, are bridged to it, and so must have a MAC address. In the System Profiler, the Airport interface is listed as Ethernet device en1 (or perhaps en0 on an Air). The wired connector in many modern computers is merely one form of Ethernet known as 10BASE-T,100BASE-T or 1000BASE-T depending on speed. So, really, 802.11 is just yet another transport medium for Ethernet, like twisted pair cable, coaxial cable or one of the many others.

March 17, 2008, 23:01

Apple does indeed listen to feedback and does change interfaces for the better. Saying they don’t betrays one’s inexperience with the history of the platform.

Criticism is fine and dandy but so is giving credit when it’s due.

Have you used Mac OS X since the public beta? Or NeXTSTEP for that matter?

Things have changed and things will improve yet again.

March 17, 2008, 23:03

My two largest beefs with Time Machine are both performance related. The first is that, when Time Machine is running against anything other than a local volume attached to the system being backed up, it takes up so many CPU cycles that everything else slows down, sometimes to a crawl. Not a big problem with desktop systems — you can just plug in the disk and leave it behind the desk — but very annoying for laptops: you have to unmount the drive (potentially waking the machine first) before you can unplug it, so you can’t just pick up the laptop and go.

You would think that the obvious solution would be network-attached storage, except that this is my second Time Machine problem. At first, I had a 1TB disk attached to my server (an Xserve, as it happens) to use as backup. This made the server so unstable that I had to undo it and go without client backups for a while. When Apple started shipping Time Capsules, I decided to get one (since I didn’t want to mess with using an unsupported disk attached to my existing Airport Extreme). I followed Apple’s advice, making it the main Airport base station, and had such a network performance bottleneck when backing up my four systems that I turned it off again. (How bad? How about 10% packet loss and average round trip ping times (using ping -f -c 1000) of about 285ms, with some packets taking over 1s!) In the meantime, the backups would generally crash out at some point due to lost network connectivity (not a problem before installing the Time Capsule) and often that would render the existing backup images unusable, meaning deleting the images manually and starting over.

Frankly, Time Machine and Time Capsule have really annoyed me, as a general rule. If I had a better alternate solution than Retrospect, I’d use it. But the practical upshot right now is that my only real option for backup is to put a drive on each system, and each the $500 for the Time Capsule, using it as a base station. Bleah. Apple is usually better than this.

March 17, 2008, 23:03

And given Apple’s past speed of improving the Spotlight UI, I won’t expect considerable improvements there before 2050.

IIRC, Spotlight was introduced with Tiger. The UI was vastly improved in the next version of the OS. So, what about the “past speed of improving the Spotlight UI”? Did you expect it would happen in a bugfix release?

And that’s before I get into real bitching mode, my friends! Let’s say you have a hard drive and a single folder on that hard drive which you want to include in your backup. If you naïvely assumed that this should be a ‘super easy’ single drag-and-drop operation in Apple’s OS, you’d be wrong. As far as I can tell you are expected to add all other folders on the disk in question to the exclude list and of course update its state whenever you add another folder. Excellent!

Haven’t all this made you realize that what you want to do is actually unsupported by Time Machine?

You could reasonably complain about it not being supported, but you can’t complaint for the fact that it takes numerous steps to achieve something it clearly wasn’t designed to do. Strike the cause, not the symptom.

With Apple’s personality being a bit too far on the self-consumed and narcissistic side to admit they shipped something less than ideal or – gasp! – poorly designed, I wouldn’t hold my breath while waiting for them to improve things.

Really? IIRC, Apple just recently took back two design decisions of Leopard 10.5 on 10.5.1 and 10.5.2. Namely they added the option for a non-transparent Dock and they fixed the Stacks implementation (folder view, etc).

So, there…

March 17, 2008, 23:58

Come on, the clock goes backwards because it’s going back in time. I think it works well. Besides, the prettier a program is, the more someone will want to use it—something important when it comes to a backup program.

March 18, 2008, 0:37

I really wonder if there is a reason to make Time Machine backup FAT32 partitions (e.g. Boot Camp) — I really wonder why Apple decided to ignore these by default. Perhaps it can be done via IncludeByPath? (which oddly is non empty on my system without me changing it… odd.) I also think I saw a RootPaths (don’t remember the exact name) pref being in a plist which by default only contained /.

On another note I wrote a tool for diffing Time Machine backups. It can be used to see what actually changed between two snapshots. It’s just a few hundred lines of Ruby code. Right now you’ll need to change some of the variables directly inside the code. I usually run it through TextMate. I might add a nice command line interface with options in the future if there is interest, though. It’s available at http://flgr.0x42.net/tm_diff.rb — feedback very welcome.

March 18, 2008, 1:30

At the start of the article, it would have been a good idea to review backup from the perspective of what risks one needs to protect against. A few are electrical surge, theft, vandalism, fire, wide-area disaster (e.g., flood), user error, hardware error. Time Machine only protects against a few UNLESS the backup is moved offsite, but that defeats the always-on nature of it.

I run Time Machine, and it’s great, but one still needs other forms of backup! Basically, one has to rotate hard drives that are either clones or at least contain important user data, and keep one at least 10 miles away.

(I also echo what Jeff said about Time Capsule. Horribly unreliable. I sent mine back. Still use Time Machine, though.)

—Marc

March 18, 2008, 1:44

Verified - the MAC address of a MacBook Air is used as the value of the com.apple.backupd.BackupMachineAddress xattr.

March 18, 2008, 1:44

“Where word of mouth isn’t even that but just what I picked up from the garbage dump known as the internet.” Ha! It’s so rare to read an article that’s technically insightful yet has that zest of personality that makes it an enjoyable read. Well done good sir!

March 18, 2008, 2:49

“e-mail of the last decade or two”

A bit obsessive? Who needs their e-mail from a decade ago?

March 18, 2008, 3:19

Whether or not the MAC address is a good choice for identifying a particular machine for Time Machine purposes, I can’t say. What I can say is that the situation is not handled gracefully when one’s MAC address changes.

The Apple Genii kindly replaced the logic board in my MacBook a few weeks ago, and when I brought it home and connected my external drive I was somewhat disconcerted to find that Time Machine no longer recognized that the backups on the external drive were from the computer I had just connected it to. After a bit of googling, I found a relevant MacOSXHints thread [1] (thank goodness!) and I was able to convince Time Machine to reclaim its progeny and pick-up where it had left off, so everything worked out fine…

But it was pretty disappointing. On the customer-service end, I specifically told the Apple Genius pre-repair that my computer was backed-up via Time Machine, and they replaced my logic board and handed my computer back to me without a word about this little glitch. On the Mac OS X end, it was somewhat distressing that Time Machine did not automatically figure this out on its own and, given that it didn’t, that there was no straightforward (i.e., non-command-line) way to help it.

[1] http://www.macosxhints.com/article.php?story=20080128003716101&query=time%2Bmachine%2Blogic%2Bboard%2Bchange

March 18, 2008, 4:01

I’m going to be a bit redundant and second Ochs’ comment, but add some context. The Time Machine Menu Bar icon animates as it should—not because you are “backing-up” but instead you are arranging to go back in time. Furthermore, you can tell that Time Machine Menu Bar icon works as it should if you look at the icon on the Time Machine pref panel; you’ll note that the Time Machine icon on the pref pane bears a very strong resemblance to that on the Menu bar; the arrow head is pointed to go counter-clock wise and we can assume that, were the Time Machine pref pane icon also animated that the clock hands to would rotate counter-clock wise.

I think if your critique of the Time Machine UI was referenced to the Leopard HIG it would carry more weight. But it’s your blog for your opinions, HIG or no HIG.

Still, thanks for the post and know that I and others really appreciate the time and effort that went into explaining how Time Machine works.

Tchuß!

Jim

March 18, 2008, 4:15

My main takeaway from your description of Time Machine and your suggested technical improvements is that backup SW is hard to do. On the technical level, I hope and expect that Apple will continue to improve TM, e.g. file renaming, encryption, FileVault, quotas, full restore, etc. However, don’t expect the UI to get much more complex. A little, perhaps, but not much. Remember, Apple’s target customer is much different than you and your readers. I, for one, think the TM metaphor and UI is brilliant. Space-time has never been so easy. For most customers, it perfectly conveys the backup and restore process. And, as bad as you think the UI is, most will use TM precisely because the UI is so friendly and approachable.

On a side note, I do agree that saved searches should editable.

March 18, 2008, 4:24

Thanks for the thorough and pretty accurate write up; I have a couple points that haven’t really been touched upon yet:

Time Machine does no integrity checking of backups as it runs, nor is there any easy way to do so offline. As you said, it only uses file size and modification date when checking for changes, but due to the way it handles metadata, it makes it tough to compare things with a tool like mtree or other ad hoc integrity verification tools. I discovered the hard way that I need something like this, when I had to do a full system restore after a botched 10.5.2 upgrade, and 2/3 of the Application directory had disappeared from several snapshots. It wasn’t a critical loss, but it was a serious annoyance. Any backup system based on hard links is going to have problems with permission changes (change the permissions or ownership on one version, they all change), and Time Machine just promotes these issues to the directory level. Most damningly, Disk Utility / fsck_hfs seriously corrupted my Time Machine volume in the guise of repairing it. I had 40-something snapshots of the 80GB or so of data on my MBP on a 500GB USB drive, and after I discovered that the directory was too big for the newest version of DiskWarrior to repair (worrisome in itself), I decided to proactively check the filesystem with Disk Utility instead. After letting it run for five hours, it claimed to have fixed a bunch of files and directories with incorrect link counts and orphaned inodes. What it actually did was corrupt a majority of the snapshots, replacing directories with empty inodes, and replacing owner and group information with more or less arbitrary IDs. When I attempted to salvage the disk by deleting the corrupted snapshots, none of the disk space held by them was freed — even after I ran Disk Utility again. At that point, I decided that trying to fix the existing backups was likely to lead to serious pain down the road, so I reformatted the disk and started over. I wasn’t happy about this.

I like Time Machine (even its goofy UI appeals to me), and I think I’m unusually hard on my computer and its operating system, but my faith in it to get me out of a jam is pretty low at this point. I’m going to keep using both Time Machine and SuperDuper, but I’m also going to keep an eye on Retrospect X. Being able to reliably recover from a catastrophe is important to me.

March 18, 2008, 7:01

Great overview on some not-so-obvious gems in Time Machine. Thank you so much. Just one improvement: Apple’s raw query syntax has some special words for relative dates. You may greatly simplify your search for backup in the last 3600 seconds. Use “‘_kTimeMachineOldestSnapshot > $time.now(-3600)’”. More examples at Apple: http://developer.apple.com/documentation/Carbon/Conceptual/SpotlightQuery/Concepts/QueryFormat.html

This allows for a smart search entry :)

March 18, 2008, 9:03

(A follow up comment to Martin Redington.)

‘ln’ won’t let you link directories, but (as root) ‘link’ almost certainly will.

It’s always been possible, in all Unixes. It’s just that when you later fsck the filesystem, most will complain and clear the duplicates out because it’s not a supported filesystem arrangement.

March 18, 2008, 12:58

Can you elaborate on what data dies once prethinning starts? Does never changed data that is still on the disk remain in the backup set after the initial backup is thinned?

March 18, 2008, 13:55

Sven, nice job on the Time Machine review. Overall, this was a job well done. I just differ from your opinion in one area. Specifically, I believe Apple made the correct choice by choosing to omit caches from backups. Caches should be considered temporary and they are meant to be rebuilt. Recreating the caches is the “clean” thing to do, expecially during a restore process. Further, I’m hard pressed to even think of a practical situation whereby temporarily not indexing application caches would have any real impact on Spotlight searches anyway. Perhaps, if you had a practical example of how this would present a real problem, I would consider changing my opinion on this.

Otherwise, thanks for the Time Machine overview. Nicely done!

March 18, 2008, 15:09

Quote: “While the Finder will let you save elaborate search queries like this one as a ‘smart folder’, it currently doesn’t let you edit the query […]”. You can edit the query by selecting the search in the sidebar and choosing “Edit search criteria” from the action menu.

March 18, 2008, 18:15

As of present, external USB drives hooked up to an Airport Extreme N station can not be used for time machine backups. I’d be interested o read your perspective on why this is so. Is it a technical issue or a marketing issue?

March 18, 2008, 19:03

@Drew and Mo:

Obviously it never occurred to me that Apple could be hiding a standard format in there. That makes me feel a lot more comfortable about Mail. Thanks for the hint.

@Andreas Fuchs:

I didn’t have sufficiently many spare drives to try a full restore. So I could only restore files or folders one by one. As mentioned in the text, this worked for me on my own files, but I wouldn’t want to deduce much from these tests as they were neither systematic nor done in large numbers. Such testing is certainly a good thing to read up on.

@mattyohe:

Thanks for sharing that info. Great to hear this works.

@Joshua Ochs:

I agree about the importance of getting people to use Time Machine to begin with. And certainly one can discuss details of the implementation. But the current approach with needlessly huge icons and buttons looks more like dumbing down to me, while the few settings you can do are unnecessarily hard to reach. I guess we can agree that Apple could have done quite a bit better while aiming at people who never backed up before.

(Obviously) I disagree with your assessment of the Time Machine menu bar icon. Note that I don’t disagree with the still icon for Time Machine. Just with the fact that it rotates counter-clockwise while a backup is running. How does that make sense? Particularly as the Finder displays arrows rotating clockwise next to the backup volume’s icon in its sidebars to indicate the very same action?

@jhn:

There’s no genealogy I know of. I have been using the ☃ glyph of hiragino for ages in the icon I use for myself on computer systems and web sites. I did that for the looks and for being a Unicode aficionado. No idea why that application is using the snowman as its icon.

March 18, 2008, 21:34

Excellent Review! I especially agree on the PrefPane and the way you have to “Exclude” Files. Here, i only want to Backup my “Documents” folder and nothing else because my external harddisk is pretty small. In order to do this, Apple makes me jump through hoops by having to exclude about every other folder on my Harddisk.

Why can’t we simply also have the option to “Only Backup” these folders preferably via a drag-and-drop of the folder into the preference… This is usually how a Mac works. Lately, however things seem to change and a certain sluttishness is visible across the OS (e.g. transparent menu bar). Some days — depending on my mood, i have the feeling that Macs are now made for people soley running iTunes and iPhoto. Ah well, the good old days…

March 18, 2008, 21:47

One annoying thing I notice about the exclusion list. The list doesn’t get updated if you rename an excluded volume. I have a third drive that I use to store a bootable mirror copy of my main hard drive (using SuperDuper). I obviously want this drive excluded from Time Machine, but some time after excluding it, I renamed the volume. The next time TM ran it tried to backup my (previously excluded) duplicate of my main drive. As there wasn’t room for it, TM had to remove all my previous backup states to fit the mirror.

March 18, 2008, 22:10

Can you please learn to identify products properly? There is no such thing as X.4, X.5, or X.5.2. They are 10.4, 10.5, and 10.5.2. Thanks!

March 19, 2008, 2:31

Am I the only one who thinks that this is a very poorly authored article?

March 19, 2008, 5:03

Thanks for the review. One item I’m not clear on is the idea of deleting an unwanted file from the backup disk to free up some space. I was working on a 8 GB video conversion project when TM started and backed up everything I was working on. At the time this was desirable because I didn’t want to lose that 8 GB file (which was the result of an overnight processing effort). But after I was through with it I wanted it gone from the backups. How does one accomplish this?

March 19, 2008, 6:16

Small point: you absolutely can edit Smart Folder parameters. After saving the folder, open it and click on the Options menu (the gear). You’ll see a command called “Show Search Criteria.” This pops the interface back onto the toolbar and allows you to make and save changes to the folder. It’s not quite the “Edit” button it should rightly have, but it’s there nonetheless.

March 19, 2008, 7:35

@Jeff Medcalf:

Indeed, there may be some performance issues with Time Machine as well. On my setup with a USB backup drive I was tempted to blame that on less-than-optimal USB handling. But perhaps your experiences suggest that there may be further problems.

@bananaranha:

1. Yes, here have been some improvements to Spotlight in X.5 and its UI has changed. From my POV I’m not even sure those changes are improvements. 2. If Apple took back two design decisions, they would have simply removed the transparent menu bar. Instead, they introduced an additional preference, which is the coward Linuxy thing to do.

@Marc Rochkind:

Yeah that totally deserves a discussion as well. But this text is already a bit long as it is. If you know any good discussions of the topic, please send links.

I suppose it’s unrealistic to suppose home users will rotate media and use off-site backups, as desireable as that would be. That’s why I was hoping for better encryption and network support. Perhaps most DSL lines don’t have enough upstream bandwidth at this stage to support backups smoothly, but it certainly seems the way to go and Apple have never been shy to design their products in a way that they work fine on the fastest connections only.

@Bondi:

Totally obsessive. I kept the postcards as well.

@George:

I haven’t stress tested Time Machine’s thinning algorithms in tricky situations, but they seem to make sure that you don’t lose any of your currently existing files from the backup, meaning that unchanged files will simply stick around.

@Steve:

There are plenty of practical examples for the caches situation. For example our UnicodeChecker application can add all Unicode codepoints to your Spotlight index if you are so inclined. The files for indexing should be stored out of the user’s sight and yet be inside the index. The Caches/Metadata folders are the canonical place for them to live. And thus you will lose those search results after doing a restore from Time Machine unless you make sure these files are recreated.

The same is true for many other applications (eg Transmit, Safari Bookmarks and History, NewsFire, Bibdesk, …) which you will have to launch at least once for their index files to be restored. As stated in the text, I think that this is ultimately bad design of Spotlight, but Time Machine should have kept that in mind.

@Dirk, S:

Thanks for pointing that out. As a non-sidebar user I missed that ‘feature’.

@swissfondue:

I have no idea about the technical side of wireless base stations and perhaps Apple’s aren’t sophisticated enough to handle this. But I’m sure if Apple wanted to, they could have made this work by designing their base stations sufficiently capable to begin with.

March 19, 2008, 22:07

After thinking about it more, a file that has never changed still has hard links pointing to it (or to some enclosing folder). With one hard link, the file should be “pinned” and automatically safe from deletion.

I’ve also written up my observations on Time Machine, see the web page link.

@Duncan

any file or folders that are no longer wanted in the backup set can be purged from within the Time Machine user interface. The secret command is in the “action” menu (gear thing). Just find the thing that you want to purge and delete it via the action menu. Note that the space will be immediately recovered on a locally connected disk, but on a remote disk, nothing will appear to happen. The sparsebundle will not shrink, but it will have white space created inside it that Time Machine can reuse.

March 20, 2008, 5:04

@George

Thanks for the tip about deleting a file from a backup. Unfortunately, when I tried it (from the non-admin account that owned the original file), Time Machine presented a warning (OK) followed by an authorization panel (admin account + password). When I tried to enter an admin username and password, the panel wouldn’t allow any text input.

Everything else on my machine is pretty stock software, with minimal 3rd-party additions. Any thoughts why the panel entry fails?

March 20, 2008, 23:04

Tried from an Administrator account? Tried making the user an admin for a while?

March 28, 2008, 4:08

Subject: Duplicating a whole machine’s 160 Gb disk from a Time Machine backup

I just moved from an old iMac 20” running 10.5.2 to a new 24” two days ago.

Installing from an external backup disk worked smoothly: after choosing which backup to start from, 3.5 hours was the duration estimation, so I let it alone.

When I came back, I just connected the new machine to the twisted pair network and clicked on the Restart button. Everything has run fine as before since. Only thing I had to do was to redefine the screen saver’s image and screen corners actions, and that’s it!

April 3, 2008, 18:00

Jacques:

Thanks for sharing your experiences.

[I think that your screensaver preferences are lost ‘by design’ rather than by accident. At least they are stored in the ‘By Host’ folder of your Preferences folder which is attached to the machine’s MAC address and thus won’t be used after moving to a different machine. No idea why they think that’s a good thing to do, though.]

April 3, 2008, 22:23

@Martin Redington:

Yeah, you seem to be right that the link count for folders doesn’t take into account additional hard links to the folder while that for files does. Any idea why that may be the case? Do utilities use this number to find out the number of files in a folder?

@Forrest L. Norvell/2.:

Thanks for pointing out the issue with the permission change. It seems that current Time Machine simply ignores changes to permissions if the file’s size and modification date remain unchanged.

April 4, 2008, 1:17

Firstly, thanks very much for this detailed and helpful description of Time Machine; I learned a lot.

Time Machine is a great concept that currently has a couple of notable shortcomings from my perspective (I appreciate that there are others that you noted.)

(1) It does not run sufficiently in the background, which means that it bogs down a user trying to do work while Time Machine is running. If Apple cannot figure out a way to make TM really run in the background with throttled disk accesses, then I see a couple of workarounds:

(a) Only run TM backup when there has been no (human) user activity for, say, 5 min. (b) Provide an option to only run TM overnight; this will miss the hourly updates, but make the Mac usable.

Does anyone have any hacks to address this problem; I doubt that Apple will address it soon, if ever.

(2) As you noted, TM only allows exclusion of files and folders. I appreciate that many users only have one external drive, which is the TM destination, but some of us have two or more. I have two external drives; A and B. (B is the TM destination.) I have a few folders on A that I really want to back up with TM, and many files and folders that I do not. This makes use of TM laborious and frustrating, and has to be re-done every time a new folder or file is added that does not need backing up.

It ought to be easy for Apple to allow specification of items to be included as well as excluded; have they ever commented on this? Until they do, I have to continue using the old Backup utility, notwithstanding its flaws that you noted.

Best regards

June 5, 2008, 11:46

Kevin: try http://timesoftware.free.fr/timemachineeditor/

You can set it to back up at intervals of your choice.

You can also change the interval in Terminal. If you google it, you’ll get instructions.

June 14, 2008, 6:10

I just stumbled to this great article.

Can you comment on how Time Machine “fold” the hourly into a daily and from daily into weekly?

Without “folding”, I would assume it just pick up one of the hourly (maybe the latest of the day), rename it into daily, then purge the rest.

Let say that I was working on a document throughout the day, so that I have multiple revisions on the hourly snapshot. Let’s say for one reason or another, the file was deleted on the same day and there were a few more hourly snapshots taken after it was deleted. Will I be able to recover this a few days later?

February 18, 2009, 0:19

@Aaron:

I never got round to running these long-time tests, so I only have assumptions to offer on that but no facts. That’s certainly a good question you bring up about the deleted files which one should know the answer to (or stop deleting files…)

February 18, 2009, 0:45

thank you very much for this informative article

the number of com.apple.backupd.SnapshotCompletionDate are the microseconds since 1.1.1970. It’s quite easy to retrieve the date in Objective-C:

NSTimeInterval snapshotCompletionInterval = 1246788798664224; NSDate *snapshotCompletionDate = [NSDate dateWithTimeIntervalSince1970:snapshotCompletionInterval / 1000000]; NSLog(@”%@”, snapshotCompletionDate) // —> 2009-07-05 12:13:18 +0200;

July 12, 2009, 10:40

just a simple thanks.

August 28, 2009, 5:17

Very nice in depth article …

I was wondering, is it possible to “Talk to” Time Machine PrefPane directly from the command line or through a cocoa app ?

It would be interesting to know so.

Cheers

September 25, 2009, 4:34

Time machine is a lifesaver. I agree with many of the issues raised, however Time Machine should not be analyzed as a universal backup solution. This is an emergency backup system. Or small-file backup system, or system restore, email restore, document version restore, etc.

If you need backups of data (images, video, music) get a dedicated backup program. TM’s database system is limited by partition size (sure you can slam together a huge partition in a JBOD raid, but not wise). The database file system is not ‘user-friendly’ for direct access (I learned the hard way how easy it is to ‘bric’ a time machine). Furthermore, the TM partition should be attached and running with the system (not the most archival backup strategy).

As for ‘exclusion’ — Let TM backup your system files. If anyone has used MS restore/backup TM blows it away, and the star-trek UI is really a useful ‘no-brainer’ interface. Even Spock would see the logic behind the simplicity.

My recommendation - Use a backup program (I use deja vu from propagandaprod.com) for the big stuff, and for disk cloning. My deja vu backups are exact copies of partitions of Raw and Photoshop files. I can easily copy files from the external backup into my macbook for portable editing. Deja Vu is my ‘main’ back up — TM takes care of everything else. In two years, I’ve never needed the Deja Vu backup - but Time Machine has saved the day several times. The safety net alone is worth twice the price of the OS.

Kevin

December 16, 2009, 2:12

I do agree with many of the issues raised, however Time Machine should not be seen as a universal backup solution.

January 20, 2010, 13:40

Thanks for the great article. You really present a systematic analysis of TM. TM is so awesome and yet it is so close to being even more amazing. It’s feels like they could see the peak of the mountain and decided to stop and plant their flag just shy of the peak.

Lacking features: - Backing up to encrypted disk images - The “file vault” mess - Friendly support of rotated or even just duplicate backup media (why can’t it realize that i backup at work and home every day by itself!! (I use marco polo and bash scripts to swap plist config files) - Upgrade path for new backup disks / resized partitions

Ignore all of the complaints. I think X.5 makes plenty of sense and is less redundant than OS X 10.6. It’s like the car salesmen selling the “O”-ten model cars (like “O”7, 08, 09).

January 27, 2010, 18:59

@rox0r:

You can ‘hack’ your way towards using encrypted disk images for the backup. Works well for the drive I store at the office every day.

Couldn’t agree more on your other points, though.

January 27, 2010, 19:08

Dankeschön, that was quite instructive. I wonder if you are still interested in this topic, but I would like to find out:

What is an elegant way of deleting all the back-ups and start afresh? If I just delete a folder containing a dozen back-ups or so, it takes an hour or so to put in the trash and another hour to empty the trash. (And believe me, my trash was not big enough to hold all of that anyway)

Is there a way to tell time machine - look, I really don’t need all these backups anymore. Get rid of this one, this one and this one. (Like for example, I want to keep the first one, and then the last 3) - I would accept a script or tool to do that.

BTW - I’ve replaced my hard drives in my iMac and MacBookPro with SSD drives and both times ran a full restore from TimeMachine. (Installed OS X from DVD until it offered me the full restore). No problems whatsoever.

In the future, I want to be a bit more sophisticated and exclude the VM so I don’t have to delete all the backups all the time - defeats the purpose, I guess.

September 21, 2010, 21:44

@mafux777: I don’t think there’s a supported way to do what you have in mind. As Apple built Time Machine to automatically delete some of the older backups, I guess they don’t think you’d need such a feature.

If you have only backups on your drive and want to get rid of all of them, you can use Disk Utility to re-format the drive. That’ll be much quicker.

September 22, 2010, 20:48

This would be a great review except for the author’s sneeringly negative attitude. Stuff like (paraphrase) “the ‘Time Machine’ name appears in Myriad font which certainly doesn’t belong in OS X” and “shouldn’t science fiction backgrounds remain in Linux distributions only” and so on really detract from what would otherwise be a startlingly useful treasure trove of information. Seriously, must the author criticize every picayune thing?

January 11, 2011, 15:57

I came by your page when searching for app keys, thanks to you I could configure TimeMachine from the OpenDirectory to exclude some big VM files which resided on my mac. I am wondering how you found out that TimeMachine uses these app keys? Is it possible to get a list of all available app keys which TimeMachine uses?

January 13, 2011, 22:10

I cam across this page looking for something slightly different. I have MBA with SSD so didnt want to put massive itunes library on the SSD to waste it so when it is “docked” at home it has a 2T USB drive attached called “library”. I have a time capsule and the MBA backups over the home wifi to the time capsule. In my itunes folder I made music, movies and TV shows symbolic links to the USB drive so it has the content but the itunes xml file and mobile apps folders etc are local.

Time Machine backup appears to only save the link file and not the content out at the USB drive. If I click within time machine on the links you can see the reference from the tim machine backup folder revert back to the USB drive.

Is there any way to make time machine follow those three links out to the external USB drive and back up that content too?

Thanks

August 2, 2012, 7:34

@james:

Yes, TimeMachine does not follow symlinks (which is probably the right thing to do by design).

What should solve your problem is going to the Options sheet in Time Machine System Preferences. There, your external volume is probably set to be excluded from backups. Time Machine seems to do that by default for external volumes. Try removing it from the list.

August 3, 2012, 8:25

I realize this is an older post but you seem to know a lot about Time Machine. I’m trying to restore a file folder from Time Machine but the folder in the results is grayed out. When I look at the actual folder on my hard drive, it is also grayed out. When I look at get info, I unlock it, but it won’t let me switch “everyone” to read/write. When I look at other backup days for that same folder, it’s grayed out going back to mid January. Any idea what’s going on?

March 13, 2018, 18:25