Sunday, April 17, 2011

AnyBackup 0.7.1 Released


  • Issue 23 : Display drive name next to letter in the add dialog
  • Issue 28 : 0.6 broke paginated results
  • Issue 30 : Display file count / name during indexing
  • Issue 31 : Allow deletion of multiple items at once for valid extension / skip list

Thursday, April 14, 2011

Restoring Deleted Files in Greyhole And Terminology Explained

Greyhole has a lot of interesting terms that might not offer an immediate explanation as to what they actually represent. I also see a lot of people asking how they can restore deleted files in Greyhole. Well, let's get to it!

Update 7/20/2011: I submitted a change to my forked Greyhole github which gboudreau merged into the main Greyhole git repo. This change simplifies all the terminology, so I've updated the below guide to show the new terms along side their old world counterparts. These new terms will be live in 1.0.0! Everything that looks like (This) is referring to Greyhole 1.0.0+.

First let's get a list of terms together.

  • Tombstone (Metadata File)
  • Attic (Trash)
  • Graveyard (Metadata Store)
None of these make much sense right away (well, they do if you understand the thought process behind them, but that can take time!) So let's go through and analyze each item. I'll put them through the layman's translator for you!

Tombstone (Metadata File)
Tombstone (Metadata File) -- "A file containing meta data about a file in your Greyhole pool."

  • Tombstones (Metadata files) are automatically created and stored for every file that is written to your Greyhole pool
  • Tombstones (Metadata files) are stored in a collection called a graveyard (metadata store) (we'll get to this later)
  • Every drive in your pool has it's own collection of Tombstones (Metadata Files)
  • Tombstones (Metadata Files) mirror the structure of your share
    • You have a file in your share stored at /path/to/sharename/folder/file
    • Let's say Greyhole moves this file to drive sdb1 which is mounted at /mnt/hdd1
    • The Tombstone  (Metadata File) for file will be created at /mnt/hdd1/gh/.gh_graveyard/sharename/folder/file (mnt/hdd1/gh/.gh_metastore/sharename/folder/file)
  • If you have a share set to save multiple copies of a file, there will be a Tombstone  (Metadata File) created on each drive that contains a copy
  • If you have only one copy of files per share you will actually have two Tombstones  (Metadata Files).
    • One will be one the drive that contains the file
    • The other will be in in a backup graveyard (metadata store) -- this is so you know what files have gone missing if a drive dies!
Graveyard (Metadata Store)
Graveyard  (Metadata Store) -- "A storage pool drive's collection of tombstones (metadata files)"

  • Every drive in your Greyhole pool has a Graveyard  (Metadata Store).
  • The Graveyard's (Metadata Store's) location is /path/to/pool/drive/gh/.gh_graveyard (/path/to/pool/drive/gh/.gh_metastore)
  • The directory structure inside .gh_graveyard (.gh_metastore) mirrors that of your share, the only difference being that the files it contains are not your files, but rather meta data (Tombstones)  (Metadata Files) about them, you'll notice that they are small and contain just a little bit of text (see the above definition for more about Tombstones  (Metadata Files))
    • There may also be a .gh_graveyard_backup (.gh_metastore_backup) folder on pool drives which contain Tombstones (Metadata Files) for files on other shares when the files copies for a share is only one
Attic (Trash)
Attic (Trash) -- "Greyhole's recycling bin"

  • Whenever Greyhole get's into a situation where it would delete a file, Greyhole moves the file into the Attic (Trash) instead.
    • If you do a delete, Greyhole moves the file to the Attic (Trash).
      • Note: If you have a program that creates temporary files when opening a file (like word or vim, etc) and then deletes those temporary files you'll end up with files in your Attic (Trash) that you don't necessarily recognize. (See below for how to access files in your Attic (Trash).)
    • If you have >1 copies of files per share and you write to a file the out of date copies (those that weren't modified) are sent to the attic (trash).
  • Each drive has it's own Attic (Trash) folder.
    • The Attic (Trash)  folder is at /path/to/pool/drive/gh/.gh_attic/ (/path/to/pool/drive/gh/.gh_trash)
      • The folder structure for an Attic (Trash), like a Graveyard (Metadata Store), mirrors that of your share, but, unlike a Graveyard (Metadata Store), the files inside an Attic (Trash) are real files.
  • To get to the files in the Attic (Trash) you can either browse to the path above for each of your Greyhole drives or you can setup a Greyhole Recycle Bin Share
    • You can create a special share name with one of the following names in Samba: 'Greyhole Attic', 'Greyhole Trash', 'Greyhole Recycle Bin'
    • Create the above share like you would any other Greyhole share (that is, use the vfs object and dfree properties)
    • When Greyhole sees this in your Samba config it will create symlinks to all files deleted after the share is created -- older files in the Attic (Trash)  must be accessed via the paths above -- in the Attics (Trashes) in the share path you specify.
      • This won't take effect until after the Greyhole service has been restarted, so remember to do this after making changes to your Samba or Greyhole configs!
    • From this share you can copy your deleted files back to the pool or delete them.
      • Files deleted from the Attic / Recycle Bin share are deleted permanently.
  • Having deleted files move to the Attic (Trash) is the default behavior. If you do not want this to happen you can change the delete_moves_to_attic (delete_moves_to_trash) property in greyhole.conf (either globally or per share)
    • If you set this property to "no" Greyhole will permanently delete all files, they will not be moved to the Attic (Trash) ever.

Sunday, April 10, 2011

Media Player Project

I've got a project I've got in beta right now that I might eventually release. Here's the lowdown.

Concept: Broadcast network emulation.

What does this mean? Basically, a random distribution of tv series you have ripped from dvd to your htpc. Before you say you can just shuffle a playlist, read on. If you shuffle a playlist, you do get a random distribution of shows, this is true, but the chronological order of those shows is not preserved. (i.e. you get Season 5 Episode 2 of show X and then a few positions down the playlist you're suddenly watching Season 1). For some shows this doesn't matter, especially if there's no overarching story lines, but for others it can lead to a very disjointed viewing experience.

The ideal solution for me here is to have a player that:

  1. Chooses a random television series
  2. Finds the lowest unwatched episode for that series
  3. Plays the episode
  4. Marks the episode as watched in a persistent store once finished
I also want this to be light weight, so I've decided to forgo writing something like this for XBMC.

Instead I've wrapped this around VLC, more specifically, VLC's http interface. (I may switch to using VLC's python bindings for an internal controller later for a more all-in-one experience -- it depends on the momentum for the project.)

The project as it stands will do the steps outlined above and a little more. It makes web calls to an open VLC player (with the http interface enabled) in the background and will constantly play new episodes while the player is active and record each episode's status as it goes.

It's very basic right now, you can add shows, remove them, and play. You can't 'unwatch' shows, etc yet. But I plan to build this out. I may even add media scrapers and turn this into a slick interface for VLC in general. Time will tell. I'm not sure if anyone else has this same desire, I could be alone. And if so, I'll happily keep this to myself. :)

Saturday, April 9, 2011

AnyBackup 0.6 Released

I've released AnyBackup 0.6 today!


  • Resize-able elements
  • File sizes are now displayed (in MB) alongside the file names in the browser and result tables
  • Result page selector is only cleared at the appropriate times
  • Sticky backup mode keeps track of pending write directories to cluster files appropriately
  • Windows exe version now comes in an easy to use installer package
For the few people who've discovered it, enjoy! I've been using it regularly for my own backups and it accurately backs up my greyhole pool.

I've had discussions with a friend about how a branch would be approached for linux, it sounds doable, but right now I don't have the interest since my primary concern is Windows where I use the application. If anyone would like to take a crack at it I can explain my ideas. I certainly wouldn't mind any additional python developers helping maintain / improve AnyBackup.

AnyBackup 0.6
Download at