AnyBackup 0.7.1 Released

Changes:

• Issue 23 : Display drive name next to letter in the add dialog
• Issue 28 : 0.6 broke paginated results
• Issue 30 : Display file count / name during indexing
• Issue 31 : Allow deletion of multiple items at once for valid extension / skip list

Download at: http://code.google.com/p/anybackup/downloads/list

Restoring Deleted Files in Greyhole And Terminology Explained

Greyhole has a lot of interesting terms that might not offer an immediate explanation as to what they actually represent. I also see a lot of people asking how they can restore deleted files in Greyhole. Well, let's get to it!


Update 7/20/2011: I submitted a change to my forked Greyhole github which gboudreau merged into the main Greyhole git repo. This change simplifies all the terminology, so I've updated the below guide to show the new terms along side their old world counterparts. These new terms will be live in 1.0.0! Everything that looks like (This) is referring to Greyhole 1.0.0+.

First let's get a list of terms together.

• Tombstone (Metadata File)
• Attic (Trash)
• Graveyard (Metadata Store)

None of these make much sense right away (well, they do if you understand the thought process behind them, but that can take time!) So let's go through and analyze each item. I'll put them through the layman's translator for you!

Tombstone (Metadata File)

Tombstone (Metadata File) -- "A file containing meta data about a file in your Greyhole pool."

• Tombstones (Metadata files) are automatically created and stored for every file that is written to your Greyhole pool
• Tombstones (Metadata files) are stored in a collection called a graveyard (metadata store) (we'll get to this later)
• Every drive in your pool has it's own collection of Tombstones (Metadata Files)
• Tombstones (Metadata Files) mirror the structure of your share
• You have a file in your share stored at /path/to/sharename/folder/file
• Let's say Greyhole moves this file to drive sdb1 which is mounted at /mnt/hdd1
• The Tombstone  (Metadata File) for file will be created at /mnt/hdd1/gh/.gh_graveyard/sharename/folder/file (mnt/hdd1/gh/.gh_metastore/sharename/folder/file)
• If you have a share set to save multiple copies of a file, there will be a Tombstone  (Metadata File) created on each drive that contains a copy
• If you have only one copy of files per share you will actually have two Tombstones  (Metadata Files).
• One will be one the drive that contains the file
• The other will be in in a backup graveyard (metadata store) -- this is so you know what files have gone missing if a drive dies!

Graveyard (Metadata Store)

Graveyard  (Metadata Store) -- "A storage pool drive's collection of tombstones (metadata files)"

• Every drive in your Greyhole pool has a Graveyard  (Metadata Store).
• The Graveyard's (Metadata Store's) location is /path/to/pool/drive/gh/.gh_graveyard (/path/to/pool/drive/gh/.gh_metastore)
• The directory structure inside .gh_graveyard (.gh_metastore) mirrors that of your share, the only difference being that the files it contains are not your files, but rather meta data (Tombstones)  (Metadata Files) about them, you'll notice that they are small and contain just a little bit of text (see the above definition for more about Tombstones  (Metadata Files))
• There may also be a .gh_graveyard_backup (.gh_metastore_backup) folder on pool drives which contain Tombstones (Metadata Files) for files on other shares when the files copies for a share is only one

Attic (Trash)

Attic (Trash) -- "Greyhole's recycling bin"

• Whenever Greyhole get's into a situation where it would delete a file, Greyhole moves the file into the Attic (Trash) instead.
• If you do a delete, Greyhole moves the file to the Attic (Trash).
• Note: If you have a program that creates temporary files when opening a file (like word or vim, etc) and then deletes those temporary files you'll end up with files in your Attic (Trash) that you don't necessarily recognize. (See below for how to access files in your Attic (Trash).)
• If you have >1 copies of files per share and you write to a file the out of date copies (those that weren't modified) are sent to the attic (trash).
• Each drive has it's own Attic (Trash) folder.
• The Attic (Trash)  folder is at /path/to/pool/drive/gh/.gh_attic/ (/path/to/pool/drive/gh/.gh_trash)
• The folder structure for an Attic (Trash), like a Graveyard (Metadata Store), mirrors that of your share, but, unlike a Graveyard (Metadata Store), the files inside an Attic (Trash) are real files.
• To get to the files in the Attic (Trash) you can either browse to the path above for each of your Greyhole drives or you can setup a Greyhole Recycle Bin Share
• You can create a special share name with one of the following names in Samba: 'Greyhole Attic', 'Greyhole Trash', 'Greyhole Recycle Bin'
• Create the above share like you would any other Greyhole share (that is, use the vfs object and dfree properties)
• When Greyhole sees this in your Samba config it will create symlinks to all files deleted after the share is created -- older files in the Attic (Trash)  must be accessed via the paths above -- in the Attics (Trashes) in the share path you specify.
• This won't take effect until after the Greyhole service has been restarted, so remember to do this after making changes to your Samba or Greyhole configs!
• From this share you can copy your deleted files back to the pool or delete them.
• Files deleted from the Attic / Recycle Bin share are deleted permanently.
• Having deleted files move to the Attic (Trash) is the default behavior. If you do not want this to happen you can change the delete_moves_to_attic (delete_moves_to_trash) property in greyhole.conf (either globally or per share)
• If you set this property to "no" Greyhole will permanently delete all files, they will not be moved to the Attic (Trash) ever.

Media Player Project

I've got a project I've got in beta right now that I might eventually release. Here's the lowdown.

Concept: Broadcast network emulation.

What does this mean? Basically, a random distribution of tv series you have ripped from dvd to your htpc. Before you say you can just shuffle a playlist, read on. If you shuffle a playlist, you do get a random distribution of shows, this is true, but the chronological order of those shows is not preserved. (i.e. you get Season 5 Episode 2 of show X and then a few positions down the playlist you're suddenly watching Season 1). For some shows this doesn't matter, especially if there's no overarching story lines, but for others it can lead to a very disjointed viewing experience.

The ideal solution for me here is to have a player that:

1. Chooses a random television series
2. Finds the lowest unwatched episode for that series
3. Plays the episode
4. Marks the episode as watched in a persistent store once finished

I also want this to be light weight, so I've decided to forgo writing something like this for XBMC.

Instead I've wrapped this around VLC, more specifically, VLC's http interface. (I may switch to using VLC's python bindings for an internal controller later for a more all-in-one experience -- it depends on the momentum for the project.)

The project as it stands will do the steps outlined above and a little more. It makes web calls to an open VLC player (with the http interface enabled) in the background and will constantly play new episodes while the player is active and record each episode's status as it goes.

It's very basic right now, you can add shows, remove them, and play. You can't 'unwatch' shows, etc yet. But I plan to build this out. I may even add media scrapers and turn this into a slick interface for VLC in general. Time will tell. I'm not sure if anyone else has this same desire, I could be alone. And if so, I'll happily keep this to myself. :)

AnyBackup 0.6 Released

I've released AnyBackup 0.6 today!

Changes:

• Resize-able elements
• File sizes are now displayed (in MB) alongside the file names in the browser and result tables
• Result page selector is only cleared at the appropriate times
• Sticky backup mode keeps track of pending write directories to cluster files appropriately
• Windows exe version now comes in an easy to use installer package

For the few people who've discovered it, enjoy! I've been using it regularly for my own backups and it accurately backs up my greyhole pool.

I've had discussions with a friend about how a branch would be approached for linux, it sounds doable, but right now I don't have the interest since my primary concern is Windows where I use the application. If anyone would like to take a crack at it I can explain my ideas. I certainly wouldn't mind any additional python developers helping maintain / improve AnyBackup.

AnyBackup 0.6

Download at http://code.google.com/p/anybackup/downloads/list

AnyBackup 0.5

I put up version 0.5 of AnyBackup on the google code page today.

Change Log:

• Issue 19: AnyBackup throws exception on trying to view empty drives.
• Issue 8: Update restore function to use pending write feature 
• Bugfix: Directories were indexed with their own name in their path variable, this has been fixed.
• Bugfix: Directories should come before files in search results. 
• Cleanup: Removed unnecessary free space checks during backups / restores since the pending write feature ensures we won't use a drive with inadequate space.
• Issue 11: Paginated Results 
• Issue 17: AnyBackup keeps track of empty folders 
• Bugfix: Folders were not included in search results, they are now.
• Issue 1: Create AnyBackup Property File 
• Issue 2: Add option to use balanced backup volume selection
• Issue 15: AnyBackup doesn't report failure upon attempting to add an unnamed drive.
• Issue 14: Backup Sort Bug -- this is now resolved by initializing the to backup array at the same time as the old files array.

Not a lot has visually changed about the program aside from the new result page selector, nonetheless here is a screenshot of the updated AnyBackup:

Please note that the above mkv files are rips from DVD box sets I own. :)

AnyBackup Beta 2 -- Now with 100% more Python!

Following up on Beta 1 I've now finished Beta 2 of AnyBackup! What's different? Well, I ported the entire project to Python instead of Perl. Not because I think that there's any great functional benefit to one over the other, but rather because it's something I needed to pick up for work anyway. The main benefit to this release is that pretty much all the standing issues have been resolved. See below. I'll be looking into setting up a google code page in the near future for AnyBackup since it seems like a much cleaner approach than just updating my blog + mediafire account all the time.

Note: I work on this program in my spare time primarily to solve my own backup needs, I release it for others to use since I figure others may have similar backup needs that AnyBackup can fulfill. That said, this is beta software (and even if it weren't) I can make no guarantees that it will always 100% back up all your data and that no data loss will occur. There can be bugs that I don't know about / haven't hit. So basically, buyer beware, use at your own risk, I can't be held responsible for any issues that arise. If you do hit issues, don't hesitate to report them!

Changes:

• Now written in Python instead of Perl (see explanation above)
• File comparisons now made against file sizes (in KB) and directory paths in addition to file names
• See issues below for more details
• File object revised to store directory root and directory path separately
• This allowed me to get rid of some ugly regular expression hacks going on in the backup and indexing methods
• When backing up files or restoring files AnyBackup now creates a pending size change (the total of file size deletes and additions) and makes sure that any additional adds will fit, this keeps things from running into an issue where a drive will run out of space and copies fail
• Icon added to show whether a drive is currently connected or not (+ and green for connected, x and red for disconnected)
• Drive background color in the drive list now changes colors based on free space (>15% green, >10% yellow, <10% red)
• Drive free/total space added to drive list (in GB)

Known Issues:

• If a backup volume is full but there is new content to backup that prefers this full volume (due to the above backup logic based on parent folders), it will still choose this full volume as the destination and then throw an error that there isn't enough space on the destination volume.
• Follow up -- my solution for now is preferring a drive based on the above logic, but it creates a pending write total and checks to see if all the files we want to add will fit, if not it will instead grab the most free drive and back the files up there, in the next release I'll add a property file which will allow you to turn this sticky files feature on or off. (For now it's on -- in the next release you can turn it off and it will always place files on the most free drive at the time of choosing)
• For some reason when running a new backup, AnyBackup will often leave a few old files (that is, files that are no longer found on your content drives -- meaning they've been deleted / renamed / written to) -- this normally has no negative repercussions, but I'll figure out what's causing this eventually. 
• Some people are having issues launching the exe packaged version of AnyBackup, I'm not sure what's causing the issue, I've been using Cava Packager to create the exe's + installers and it runs perfectly on my test machines. I'll put up an archive of the raw perl later for those that have issues -- the downside to using the raw perl is that you'll need to have a valid perl install with all the packages I use.
• Since this is now in python being packaged with py2exe, I'm assuming this will no longer be an issue.
• File comparisons are made only against file names, so if you change/write to a file or have duplicate file names in multiple directories this can lead to inconsistencies such as only backing up one file instead of multiple or not backing up the updated version of a file
• I augmented this naive comparison, it now checks the file sizes and directory names in addition to file names. The only possible complication I see at this point is that the file size comparison is done in KB and not bytes, I'm not sure if this will cause issues (I'm thinking not) Feel free to let me know if this impacts small files with minute changes, though! (This is the only situation I really see it being an issue.) If I get reports I'll look into changing this.
• Minor issue with the most free drive logic, it needs to be updated to incorporate the new pendingwrite total otherwise it may not be grabbing the drive with the most free space, but rather than drive with the most free space before any operations have taken place (i.e. if we have 10gb pending to a drive which has 8gb more free than any other volume, it should no longer be considered the most free drive, but right now, it will be) 
• The CLI interface for the app, backupWrapper.py isn't guaranteed to work yet and all features are definitely not built out (primarily restore definitely does not work, I backup will not remove old files from backup volumes). What does work I haven't tested, so buyer beware when using it in this release. In the next release I will finish building out the features for it and also package it as an exe alongside the GUI interface.

 I'm only going to add one new screen shot to show off the (very small) UI changes to the main window. Despite the complete port of the app to Python, it's still using wxWidgets which is pretty much identical across languages.

Download:
(As before, this code is released under the GPL license!)

• AnyBackup Beta 2 Python Source 
• AnyBackup Beta 2 Win32

Update: Google Code project created! You can find it here.

Dynamically Convert A Raid5 Array to Raid6

Transferred from my old blog:
If you have the newest mdadm tool, version 3.1.1 and above, it is now capable of changing the raid level of an array. If you cannot find a copy of the latest version for your distro you can compile from source via mdadm’s git repo, git://neil.brown.name/mdadm
The below assumes that you have 1 spare drive ready to add to your array (/dev/sdb) and that /dev/md0 is the raid5 array that you would like to move to raid6 and that you have 4 raid devices in /dev/md0 to start with.
Use the below commands:

mdadm --add /dev/md0 /dev/sdb
mdadm --grow --level=6 --raid-devices=5

Once this completes, you should have a fully functioning raid6 array. Enjoy your dual parity.
Further, you can also change the chunk size dynamically while you’re at it, the default chunk size of mdadm (which I believe they plan to up in future versions) is a paltry 64k, you’d be much better off with something in the 256-512k range. To change the chunk size of an array, use the following:

mdadm --grow /dev/md0 --chunk=512

I’ve seen several references now using –chunks-size, so it’s possible in future versions this may be the correct flag instead of –chunk, just something to be aware of. Also, upping your chunk size to 512 may not be possible depending what the total size of your array is. It’s possible that mdadm will spit out an error stating that the total array size is not divisible by 512, in which case you’ll have to settle for something smaller. (i.e. try 256 or 128).