AnyBackup Explained

I've been working on AnyBackup for a number of months now. It's gone through some drastic changes in that time, including the language it was written in. I'd like to take a little time now to try to explain exactly what it does and why I think it's useful, or -- at the very least -- why it's useful to me.

The impetus for creating the program was the fact that I have a large amount of data on a single volume (via Greyhole) which I'd like to keep backed up to multiple smaller drives. For a long time I was accomplishing this mostly manually with a little help from some scripts I wrote. There is backup software out there which spans drives, but not exactly in the manner I was looking for. Most backup programs think that disk spanning ends at CD's or DVD's, and I had no intention of feeding several hundred DVD's into my computer once a month, I'd never finish my backups before it was time for the next! There are a few programs out there that allow spanning across hard drives, but they do a few things that break their usefulness to me.

• They assume all backup drives are connected
• They back up one large, proprietary file

I wanted a program with flexibility. So what if backup drive four isn't connected? Let me know that and give me a chance to connect it! What if I want to bring my backup drive to a friend's house? I'd like that data to be readily readable! And so my search faltered, and I gave up. A program that suits these needs may be out there, but it eluded me. So it was then that I decided I'd just make my own.

AnyBackup is powerful and simple. The basic idea: backup any number of drives to any number of drives. Or, more specifically, back up all the files which match your criteria ( a list of valid file extensions and a blacklist of regular expressions ) on a given set of content drives to a given set of backup drives.

A high-level, crude chart to visualize what AnyBackup does.

In the chart above you can get an idea of what AnyBackup does visually, take the data from X volumes and back it up to Y volumes. Either side in the above chart could be content / backup, it doesn't matter. What matters is that you have enough room on either side to fit all your files.

What AnyBackup does:

• Easily lets you back up large volumes (in my case an 11tb Greyhole volume) to several smaller drives
• Backup groups of drives to groups of drives
• Blacklist filenames with regular expressions
• Provide a list of file extensions you want to backup
• Identifies drives based on volume name and serial number, drive letters can change and it will not effect AnyBackup's record keeping
• Gives you the flexibility of connecting one backup drives at a time (i.e. using a dock)

What AnyBackup does NOT do:

• Perform automated backups
• AnyBackup plays things loose and has no integration with the OS, so it has no way of knowing when things have been updated, so it requires reindexing before backing things up
• Everything is driven through the GUI, there is no service
• AnyBackup largely assumes your backup drives won't be connected most of the time
• Allow multiple backup sets
• AnyBackup only lets you backup one set of drives to another, whether it's my instance where you have one large volume and a handful of backup drives or if it's several backup drives and several content drives
• This is an issue for enhancement in the google code tracker, if I get time or someone else is inspired it should be easy enough to setup
• Backing up only certain directories of a drive
• Right now you can only backup up whole drives, you can choose what file types get backed up, but that's the extent of it
• There's a standing enhancement issue to allow addition of directories in drives instead of just drives
• Update: This is no longer true as of version 0.9!
• Perform incremental indexing
• Since AnyBackup does not have any integration with the OS, it has no idea what has happened to your data between runs, so before a backup you'll need to refresh your drives to make sure all new files have been picked up
• This isn't necessarily a bad thing, even if we had incremental indexing, what if you disconnected a drive and took it somewhere, we'd have no way to know if you put / removed files on / from the drive on another system!

Below is a screenshot of the latest release of AnyBackup (0.8 as of writing this):

AnyBackup 0.8

AnyBackup 0.8 Released

Changes:

• Issue 33 : Write to user's home dir to avoid UAC conclicts
• Issue 35 - Missing drive prompt crashes AnyBackup
• Issue 34 - Add backup drive lock feature
• Issue 32 - Use last modified time in hash check
• Issue 37 : Allow multiple selections in Drive listNewFiles
• Issue 36: Remote Indexing
• Issue 39 : Make indexing on drive addition optional
• Issue 40 : Allow addition of multiple drives at once
• Issue 41 : Revise GUI to show backup and content drives at the same time

Notable differences in this version:

• Remote index server script provided to bypass samba for indexing a linux shared directory
• This GREATLY speeds up indexing of large volumes over samba, testing showed 20-30 minutes of indexing go to ~3 minutes
• There is no drop down for selecting Backup or Content drives, all drives are now displayed in one list, content drives are listed in bold

Download at http://code.google.com/p/anybackup/downloads/list

Regular Expressions Excluding Strings

I ran into a situation recently where it would be very, very handy to be able to write a regular expression which would both look for certain content and exclude others. I admit this is probably not the most efficient way to go about things, but for small and quick use cases I don't see why it shouldn't be used. See below for some explanations!

Negative lookahead:


Let's say you have a string set of strings, 'foobar','barbar','barfoo'. Now let's further speculate that for some unknown, but perfectly valid to you, reason, you want only the strings in the above set which contain a 'bar' but only where 'bar' is not followed by 'foo'. (I'm making this distinction now, this means it's OK to have 'foo' before 'bar', just not after.)

If your regular expression engine supports it, and most do -- at least Perl and Python do, you can write something like this:

• bar(?!foo)
• Python: re.search('bar(?!foo)',string)
• Perl: string =~ /bar(?!foo)/
• 'foobar' and 'barbar' would match the above regular expression, 'barfoo' would not -- perfect!

Now, as I said, this is for looking ahead, you cannot write something like (?!foo)bar it will not do what you want, as you're attempting to lookbehind. Conveniently, see below for how to do a negative lookbehind.

Below is a Python snippet to really flesh things out:

Negative lookbehind:

We can use the same list as above to demonstrate a lookbehind, but this time let's assume we only want strings which contain 'bar', but only where 'bar' is not preceded by 'foo'.

We can write a regular expression for negative lookbehinds like this:

• (?<!foo)bar
• Python: re.search('(?<!foo)bar',string)
• Perl: string =~ /(?<!foo)bar/
• 'barfoo' and 'barbar' would match the above regular expression, 'foobar' would not, again, exactly what we set out to do!

Another Python snippet below:

Note:

The only thing which makes these lookahead and lookbehinds negative is the exclamation points, you can easily turn this requirement around by removing it, so bar(?foo) would suddenly make the string 'barfoo' the only valid string in our set, pretty intuitive!

AnyBackup 0.7.1 Released

Changes:

• Issue 23 : Display drive name next to letter in the add dialog
• Issue 28 : 0.6 broke paginated results
• Issue 30 : Display file count / name during indexing
• Issue 31 : Allow deletion of multiple items at once for valid extension / skip list

Download at: http://code.google.com/p/anybackup/downloads/list

Restoring Deleted Files in Greyhole And Terminology Explained

Greyhole has a lot of interesting terms that might not offer an immediate explanation as to what they actually represent. I also see a lot of people asking how they can restore deleted files in Greyhole. Well, let's get to it!


Update 7/20/2011: I submitted a change to my forked Greyhole github which gboudreau merged into the main Greyhole git repo. This change simplifies all the terminology, so I've updated the below guide to show the new terms along side their old world counterparts. These new terms will be live in 1.0.0! Everything that looks like (This) is referring to Greyhole 1.0.0+.

First let's get a list of terms together.

• Tombstone (Metadata File)
• Attic (Trash)
• Graveyard (Metadata Store)

None of these make much sense right away (well, they do if you understand the thought process behind them, but that can take time!) So let's go through and analyze each item. I'll put them through the layman's translator for you!

Tombstone (Metadata File)

Tombstone (Metadata File) -- "A file containing meta data about a file in your Greyhole pool."

• Tombstones (Metadata files) are automatically created and stored for every file that is written to your Greyhole pool
• Tombstones (Metadata files) are stored in a collection called a graveyard (metadata store) (we'll get to this later)
• Every drive in your pool has it's own collection of Tombstones (Metadata Files)
• Tombstones (Metadata Files) mirror the structure of your share
• You have a file in your share stored at /path/to/sharename/folder/file
• Let's say Greyhole moves this file to drive sdb1 which is mounted at /mnt/hdd1
• The Tombstone  (Metadata File) for file will be created at /mnt/hdd1/gh/.gh_graveyard/sharename/folder/file (mnt/hdd1/gh/.gh_metastore/sharename/folder/file)
• If you have a share set to save multiple copies of a file, there will be a Tombstone  (Metadata File) created on each drive that contains a copy
• If you have only one copy of files per share you will actually have two Tombstones  (Metadata Files).
• One will be one the drive that contains the file
• The other will be in in a backup graveyard (metadata store) -- this is so you know what files have gone missing if a drive dies!

Graveyard (Metadata Store)

Graveyard  (Metadata Store) -- "A storage pool drive's collection of tombstones (metadata files)"

• Every drive in your Greyhole pool has a Graveyard  (Metadata Store).
• The Graveyard's (Metadata Store's) location is /path/to/pool/drive/gh/.gh_graveyard (/path/to/pool/drive/gh/.gh_metastore)
• The directory structure inside .gh_graveyard (.gh_metastore) mirrors that of your share, the only difference being that the files it contains are not your files, but rather meta data (Tombstones)  (Metadata Files) about them, you'll notice that they are small and contain just a little bit of text (see the above definition for more about Tombstones  (Metadata Files))
• There may also be a .gh_graveyard_backup (.gh_metastore_backup) folder on pool drives which contain Tombstones (Metadata Files) for files on other shares when the files copies for a share is only one

Attic (Trash)

Attic (Trash) -- "Greyhole's recycling bin"

• Whenever Greyhole get's into a situation where it would delete a file, Greyhole moves the file into the Attic (Trash) instead.
• If you do a delete, Greyhole moves the file to the Attic (Trash).
• Note: If you have a program that creates temporary files when opening a file (like word or vim, etc) and then deletes those temporary files you'll end up with files in your Attic (Trash) that you don't necessarily recognize. (See below for how to access files in your Attic (Trash).)
• If you have >1 copies of files per share and you write to a file the out of date copies (those that weren't modified) are sent to the attic (trash).
• Each drive has it's own Attic (Trash) folder.
• The Attic (Trash)  folder is at /path/to/pool/drive/gh/.gh_attic/ (/path/to/pool/drive/gh/.gh_trash)
• The folder structure for an Attic (Trash), like a Graveyard (Metadata Store), mirrors that of your share, but, unlike a Graveyard (Metadata Store), the files inside an Attic (Trash) are real files.
• To get to the files in the Attic (Trash) you can either browse to the path above for each of your Greyhole drives or you can setup a Greyhole Recycle Bin Share
• You can create a special share name with one of the following names in Samba: 'Greyhole Attic', 'Greyhole Trash', 'Greyhole Recycle Bin'
• Create the above share like you would any other Greyhole share (that is, use the vfs object and dfree properties)
• When Greyhole sees this in your Samba config it will create symlinks to all files deleted after the share is created -- older files in the Attic (Trash)  must be accessed via the paths above -- in the Attics (Trashes) in the share path you specify.
• This won't take effect until after the Greyhole service has been restarted, so remember to do this after making changes to your Samba or Greyhole configs!
• From this share you can copy your deleted files back to the pool or delete them.
• Files deleted from the Attic / Recycle Bin share are deleted permanently.
• Having deleted files move to the Attic (Trash) is the default behavior. If you do not want this to happen you can change the delete_moves_to_attic (delete_moves_to_trash) property in greyhole.conf (either globally or per share)
• If you set this property to "no" Greyhole will permanently delete all files, they will not be moved to the Attic (Trash) ever.

Media Player Project

I've got a project I've got in beta right now that I might eventually release. Here's the lowdown.

Concept: Broadcast network emulation.

What does this mean? Basically, a random distribution of tv series you have ripped from dvd to your htpc. Before you say you can just shuffle a playlist, read on. If you shuffle a playlist, you do get a random distribution of shows, this is true, but the chronological order of those shows is not preserved. (i.e. you get Season 5 Episode 2 of show X and then a few positions down the playlist you're suddenly watching Season 1). For some shows this doesn't matter, especially if there's no overarching story lines, but for others it can lead to a very disjointed viewing experience.

The ideal solution for me here is to have a player that:

1. Chooses a random television series
2. Finds the lowest unwatched episode for that series
3. Plays the episode
4. Marks the episode as watched in a persistent store once finished

I also want this to be light weight, so I've decided to forgo writing something like this for XBMC.

Instead I've wrapped this around VLC, more specifically, VLC's http interface. (I may switch to using VLC's python bindings for an internal controller later for a more all-in-one experience -- it depends on the momentum for the project.)

The project as it stands will do the steps outlined above and a little more. It makes web calls to an open VLC player (with the http interface enabled) in the background and will constantly play new episodes while the player is active and record each episode's status as it goes.

It's very basic right now, you can add shows, remove them, and play. You can't 'unwatch' shows, etc yet. But I plan to build this out. I may even add media scrapers and turn this into a slick interface for VLC in general. Time will tell. I'm not sure if anyone else has this same desire, I could be alone. And if so, I'll happily keep this to myself. :)

AnyBackup 0.6 Released

I've released AnyBackup 0.6 today!

Changes:

• Resize-able elements
• File sizes are now displayed (in MB) alongside the file names in the browser and result tables
• Result page selector is only cleared at the appropriate times
• Sticky backup mode keeps track of pending write directories to cluster files appropriately
• Windows exe version now comes in an easy to use installer package

For the few people who've discovered it, enjoy! I've been using it regularly for my own backups and it accurately backs up my greyhole pool.

I've had discussions with a friend about how a branch would be approached for linux, it sounds doable, but right now I don't have the interest since my primary concern is Windows where I use the application. If anyone would like to take a crack at it I can explain my ideas. I certainly wouldn't mind any additional python developers helping maintain / improve AnyBackup.

AnyBackup 0.6

Download at http://code.google.com/p/anybackup/downloads/list