AnyBackup 0.5

I put up version 0.5 of AnyBackup on the google code page today.

Change Log:

• Issue 19: AnyBackup throws exception on trying to view empty drives.
• Issue 8: Update restore function to use pending write feature 
• Bugfix: Directories were indexed with their own name in their path variable, this has been fixed.
• Bugfix: Directories should come before files in search results. 
• Cleanup: Removed unnecessary free space checks during backups / restores since the pending write feature ensures we won't use a drive with inadequate space.
• Issue 11: Paginated Results 
• Issue 17: AnyBackup keeps track of empty folders 
• Bugfix: Folders were not included in search results, they are now.
• Issue 1: Create AnyBackup Property File 
• Issue 2: Add option to use balanced backup volume selection
• Issue 15: AnyBackup doesn't report failure upon attempting to add an unnamed drive.
• Issue 14: Backup Sort Bug -- this is now resolved by initializing the to backup array at the same time as the old files array.

Not a lot has visually changed about the program aside from the new result page selector, nonetheless here is a screenshot of the updated AnyBackup:

Please note that the above mkv files are rips from DVD box sets I own. :)

AnyBackup Beta 2 -- Now with 100% more Python!

Following up on Beta 1 I've now finished Beta 2 of AnyBackup! What's different? Well, I ported the entire project to Python instead of Perl. Not because I think that there's any great functional benefit to one over the other, but rather because it's something I needed to pick up for work anyway. The main benefit to this release is that pretty much all the standing issues have been resolved. See below. I'll be looking into setting up a google code page in the near future for AnyBackup since it seems like a much cleaner approach than just updating my blog + mediafire account all the time.

Note: I work on this program in my spare time primarily to solve my own backup needs, I release it for others to use since I figure others may have similar backup needs that AnyBackup can fulfill. That said, this is beta software (and even if it weren't) I can make no guarantees that it will always 100% back up all your data and that no data loss will occur. There can be bugs that I don't know about / haven't hit. So basically, buyer beware, use at your own risk, I can't be held responsible for any issues that arise. If you do hit issues, don't hesitate to report them!

Changes:

• Now written in Python instead of Perl (see explanation above)
• File comparisons now made against file sizes (in KB) and directory paths in addition to file names
• See issues below for more details
• File object revised to store directory root and directory path separately
• This allowed me to get rid of some ugly regular expression hacks going on in the backup and indexing methods
• When backing up files or restoring files AnyBackup now creates a pending size change (the total of file size deletes and additions) and makes sure that any additional adds will fit, this keeps things from running into an issue where a drive will run out of space and copies fail
• Icon added to show whether a drive is currently connected or not (+ and green for connected, x and red for disconnected)
• Drive background color in the drive list now changes colors based on free space (>15% green, >10% yellow, <10% red)
• Drive free/total space added to drive list (in GB)

Known Issues:

• If a backup volume is full but there is new content to backup that prefers this full volume (due to the above backup logic based on parent folders), it will still choose this full volume as the destination and then throw an error that there isn't enough space on the destination volume.
• Follow up -- my solution for now is preferring a drive based on the above logic, but it creates a pending write total and checks to see if all the files we want to add will fit, if not it will instead grab the most free drive and back the files up there, in the next release I'll add a property file which will allow you to turn this sticky files feature on or off. (For now it's on -- in the next release you can turn it off and it will always place files on the most free drive at the time of choosing)
• For some reason when running a new backup, AnyBackup will often leave a few old files (that is, files that are no longer found on your content drives -- meaning they've been deleted / renamed / written to) -- this normally has no negative repercussions, but I'll figure out what's causing this eventually. 
• Some people are having issues launching the exe packaged version of AnyBackup, I'm not sure what's causing the issue, I've been using Cava Packager to create the exe's + installers and it runs perfectly on my test machines. I'll put up an archive of the raw perl later for those that have issues -- the downside to using the raw perl is that you'll need to have a valid perl install with all the packages I use.
• Since this is now in python being packaged with py2exe, I'm assuming this will no longer be an issue.
• File comparisons are made only against file names, so if you change/write to a file or have duplicate file names in multiple directories this can lead to inconsistencies such as only backing up one file instead of multiple or not backing up the updated version of a file
• I augmented this naive comparison, it now checks the file sizes and directory names in addition to file names. The only possible complication I see at this point is that the file size comparison is done in KB and not bytes, I'm not sure if this will cause issues (I'm thinking not) Feel free to let me know if this impacts small files with minute changes, though! (This is the only situation I really see it being an issue.) If I get reports I'll look into changing this.
• Minor issue with the most free drive logic, it needs to be updated to incorporate the new pendingwrite total otherwise it may not be grabbing the drive with the most free space, but rather than drive with the most free space before any operations have taken place (i.e. if we have 10gb pending to a drive which has 8gb more free than any other volume, it should no longer be considered the most free drive, but right now, it will be) 
• The CLI interface for the app, backupWrapper.py isn't guaranteed to work yet and all features are definitely not built out (primarily restore definitely does not work, I backup will not remove old files from backup volumes). What does work I haven't tested, so buyer beware when using it in this release. In the next release I will finish building out the features for it and also package it as an exe alongside the GUI interface.

 I'm only going to add one new screen shot to show off the (very small) UI changes to the main window. Despite the complete port of the app to Python, it's still using wxWidgets which is pretty much identical across languages.

Download:
(As before, this code is released under the GPL license!)

• AnyBackup Beta 2 Python Source 
• AnyBackup Beta 2 Win32

Update: Google Code project created! You can find it here.

Dynamically Convert A Raid5 Array to Raid6

Transferred from my old blog:
If you have the newest mdadm tool, version 3.1.1 and above, it is now capable of changing the raid level of an array. If you cannot find a copy of the latest version for your distro you can compile from source via mdadm’s git repo, git://neil.brown.name/mdadm
The below assumes that you have 1 spare drive ready to add to your array (/dev/sdb) and that /dev/md0 is the raid5 array that you would like to move to raid6 and that you have 4 raid devices in /dev/md0 to start with.
Use the below commands:

mdadm --add /dev/md0 /dev/sdb
mdadm --grow --level=6 --raid-devices=5

Once this completes, you should have a fully functioning raid6 array. Enjoy your dual parity.
Further, you can also change the chunk size dynamically while you’re at it, the default chunk size of mdadm (which I believe they plan to up in future versions) is a paltry 64k, you’d be much better off with something in the 256-512k range. To change the chunk size of an array, use the following:

mdadm --grow /dev/md0 --chunk=512

I’ve seen several references now using –chunks-size, so it’s possible in future versions this may be the correct flag instead of –chunk, just something to be aware of. Also, upping your chunk size to 512 may not be possible depending what the total size of your array is. It’s possible that mdadm will spit out an error stating that the total array size is not divisible by 512, in which case you’ll have to settle for something smaller. (i.e. try 256 or 128).

wxPerl ListCtrl

When writing AnyBackup I got up close and personal with the beast that is wxPerl and it's almost ridiculous reliance on its parent wxWidgets documentation. One of the most frustrating steps I encountered was trying to wrap my head around just how ListCtrl works. At first I started with the much simpler (and user friendly) ListBox, which does exactly what you think it does. It lets you display a list of strings and you can attach data objects to those strings which are accessible via click events. Very handy when you want to attach an object ref to each string. ListCtrl... well it isn't so nice.

It's not extremely apparent upon first glance and if you don't read the documentation carefully you may, as I did, stumble on blindly expecting to be able to hack in ListCtrl in ListBox's place. But this is simply not the case. To start with, you can't attach an object reference to an item, rather, you can only assign a Long id. Now this makes it possible to store data, but you've now added overhead, and additional data objects to keep track of in your code, and if you're using multiple ListCtrl objects as I did, forget about it!

Abstraction is the key here, so I created a wrapper for ListCtrl which makes it act much like ListBox, but with all the pretty options that come along with it, such as icons, colors, etc.

The best way to learn is by example, I feel, so here you'll find the wrapper class that I wrote for AnyBackup

Now this wrapper has some parts that are quite specific to my application, so don't expect to be able to just drop it in sans modification to your program, but it can give you a good idea of what you need to do to make ListCtrl be what you want!

Let's walk through the main functions and explain their... well, function.

•  GetSelectedData - this function does exactly what you think it'd do, it looks for the currently selected row and then gets the Long id for this row, and finally it grabs the previously set object reference from the storage hash and voila, transparently you've gotten an object ref for your selected row without ever dealing with the long id! 
• Note: This example wrapper specifies a ListCtrl which only accepts single selections, you could change this option, but then this function would have to be modified to return an array of object references.
• populateDrives - this function is a great example of what you'd need to modify. It takes an array of object references and grabs their names for display strings in the ListCtrl and then stashes the object refs themselves in the storage hash, mapping them with the long id.
• DeleteAllItems -- this acts like and behaves just like ListCtrl's DeleteAllItems. It delets all the items for your contained ListCtrl object and it clears out the stoarge hash and support variables in your easyListCtrl object
• InsertItem - this is where a lot of the magic happens, it takes a string (item) and an object ref (data) and maps the data to an id, and then adds the string to the ListCtrl object
• GetData - this allows you to pass an arbitrary item (string) and get it's stored data back, should that item exist, anyway.
• adjust - this function exists purely for adjusting the width of a ListCtrl object after deleting or adding items. The problem is that ListCtrl doesn't take care of this for you! If you just add items and leave it to its own devices, you'll end up with a ListCtrl which has strangely narrow columns which cut off your strings,which is most likely not what you were going for, this function will automatically resize the column to the length of the longest row, so no data will be cut off and the ListCtrl object will be wrapped, if needed, in a scrollbox.
• getListCtrl - this allows you to grab the wrapped ListCtrl object, important when it comes time to adding the object to a frame and/or sizer

Mount a CIFS share via fstab

Have a CIFS share you want to mount on every boot? Tired of manually mounting? Don't like the hack-around of a startup script doing this for you? It's pretty easy to add this to your fstab file.

//<ip.address>/<share_name>       /mount/point       cifs       credentials=/path/to/.credentials_file,_netdev,iocharset=utf8,uid=<numeric user id>,gid=<numeric group id>,file_mode=0750,dir_mode=0750,rw     0 0

So let's go over some explanations for the above fstab line, at least for the parts that aren't self explanatory.

• _netdev -- this option tells fstab that we don't want to try mounting this share until the network manager is running, probably a good idea since a cifs share is, naturally, mounted over the network!
• iocharset -- the above might differ depending on the language your files are named with, play with this if you use a different character set (i.e. Cyrillic)
• uid and gid -- setting these for the share determines who appears as the owner when you browse to the mount point, if you don't set these options root will appear to own the mount point and all the files in it, which is most likely not what you want! To find your user and group id's you can type "id <username>" in the terminal and it will print out the numeric id for your username and all the groups your id is member of
• file_mode and dir_mode -- these determine the permissions of the mount point locally and they work just like any other user permissions on a unix system, the owner permissions will be relative to the uid you set and the group permissions will be relative to the gid you set. This means if you use the above example and have 5 for the group permissions and then, as a user other than uid and who isn't a part of gid, you try to write to the share, you'll get permission denied.
• Note: even if you set permissions here to give you write access but the share source doesn't give you write permissions, you still won't be able to write to the share!
• credentials -- this is a nice way to avoid posting your username and password directly in your fstab! For example, instead of using a credentials files you could put username=SomeUser,password=SomePassword right into the options and it would work, but that would expose your password to anyone who had viewing rights to the fstab file, which may not matter in the scheme of things for a private server, but it's still poor practice.
• The contents of a credentials file is very simple, consider the below
• username=SomeUser
• password=SomePassword
• Note: CIFS can be very sensitive to extra white spaces, so make sure each line ends with the end of your username / password and not a white space, also make sure that your file ends on the password line and not a blank line, this can be a good place to check if your mount is failing and you're sure that your login information is correct. Also, do not use quotation marks, this will cause login failures.

Upgrading a Fedora Install

After making a right mess of things installing some new packages, I decided to do an offline upgrade of my Fedora 11 server to Fedora 14, and I hit a few snags along the way that I'll detail.

First off, as you may or may not know, you can only upgrade to two versions ahead. So to get from 11 to 14, I had to go first from 11 to 13 and then from 13 to 14. This isn't a big deal, but it will add some time to your upgrade process! Just make sure you have both dvd's handy.

An interesting issue I ran into is that despite upgrading, I still had many, many fc11 packages still installed, and what's more, they were preventing me from upgrading or installing new packages via yum for fc14! There are some commands that can make this easy to clean up, though. Note: If you're upgrading one release ahead, i.e. fc13 -> fc14 it can be completely normal to see fc13 packages since some packages don't get updated right away.

That said, there is a handy little command that lets you find orphaned packages installed on your system, that is, packages which are no longer in any repository your system looks at. (Note: This also includes any programs which you got from outside your repositories -- Handbrake for example, or Subsonic.) This command is:

• package-cleanup --orphans

If there are only a few items that are put out from this you can do a clean up manually, I chose to remove packages with the following command:

• rpm -e --nodeps <package name>

-e tells rpm to remove the package and --nodeps means to remove only the package, not other packages that depends on it.

I had over 100 fc11 packages leftover, so I used this little bash loop to get rid of them:

• for i in `sudo package-cleanup --orphans | grep fc11`; do sudo rpm -e --nodeps $i; done

The grep fc11 above will strip out the beginning lines (loading plugins, extra repo's, etc) which are not valid package names, as well strip out packages which are release independent (again things like Handbrake, Subsonic, etc).

Even after doing the above when installing packages via yum I was getting error messages (though my installs would still go through) about certain packages missing dependencies. The only solution I could find for this was to patiently look up the libraries (most of the errors were about .so files) on google and find out which packages they belonged to and then run a yum install for said package. It took about 5-10 packages before the errors finally cleared up.

Something else worthy of noting is that I hit some major weirdness / disruptiveness in my perl install. I had to upgrade all my perl packages after the upgrade manually via yum. The main issue, however, is that namespace, namespace::clean, and namespace::autoclean were not installed and this led to all sorts of issues attempting to run scripts which depended explicitly on namespace and also attempting to install and compile packages via CPAN.

So be prepared to deal with a little work when attempting to do a Fedora upgrade (especially from three releases back!) but on the whole I'd say it was less effort to get all this done than to do a new install from scratch as it saved me from having to worry about saving off all my custom software, scripts, configs, and then restoring these afterward.

Minesweeper Applet

From my old blog:

Almost four years ago now I had to create a version of Minesweeper using Swing for a software programming course in college. For fun later I converted it to an applet and hosted it up on my website. What you see below is the final product. The obvious drawbacks being it isn't at all configurable (either the size or the difficulty -- number of mines) but it was enough to get an A, so it was enough for me. :)