Help needed - Building a comprehensive backup system for a photographer...

Hey all,

This is my first post here and while I do understand that this is more of a security oriented board I thought I'd give asking about backup systems a go.

So as a general outline I'm building a backup system for myself. I'm not a professional photographer, but maybe enthusiastic is the word. The main goal is to try and cover all bases with the system, so from accidentally removing a photo to a visit from the fire department and everything in between.

The more refined outlines would be:

• Simple and clean solutions
• A comprehensive system
• Easily maintainable system
• Easy on the wallet - free would be great

Things to implement:

• Image-based backup for the OS
• Scheduled backups for files
• Offsite backups
• Continuous backups

Even though I'm a photographer I use a PC (which doesn't seem to be that common). I'm familiar with hardware since I've assembled and serviced computers for a living for several years. I speak Windows fluently and Linux somewhat well.

This has been my system so far:

OS backup I've used several different methods over the years, Ghost, Macrium Reflect etc, but since my OS drive consists of two SSD-drives in RAID-0, I've used Acronis True Image because of the RAID support.

This is changing because I have one bigger SSD-drive waiting for installation, so RAID support shouldn't be a problem after I re-install Windows on the new drive. Bottom line here is that I want to keep the image clean and not install anything on it before making the image. So what I'd like to have here are suggestions for a cold imaging software, preferably a free one.

I don't need an imaging software for making backups or snapshots of the system, just the initial few images in different stages of the installation so that I can easily and quickly re-install Windows when it starts to lag or something happens to my OS drive.

Backups At the moment I don't have a decent software to back up my files, so any suggestions here are appreciated. There's quite a hefty amount of data to backup, my Lightroom catalog and photos alone are taking up about 400Gb at the moment and of course that's not going to get any smaller. I cull new photos quite mercilessly to keep the amount of data as small as possible, but it does gather quite quickly still.

I've used FastCopy to just copy everything on to my Linux server, so there's no incremental backups. Ideally I'd want something like a full backup once a month and weekly incremental backups in between. My server isn't always on, so the software should be able to postpone the scheduled backup until the server is online.

I found out about Duplicati today, but I haven't had the time to look into it more yet. It seems like a software I'd like (small, clean and free), but there were mentions about it being slow. I'll have to test it out, but large backups and slow backup software isn't a good combination.

Offsite backup Since there's this much data, I don't want to push it to an online service. So I'm using the old "Adidas network" and an external drive here. It resides in a safe a few kilometers from my house so I bring it back once a month to make the backup.

Here I'd also like to have incremental backups since so far I've just copied the files on the drive. It doesn't exactly need a schedule since I'm not necessarily be able to get the drive the same day every month. So the backup software should allow me to start the backup manually.

Continuous backup I have a bought version of the Genie Timeline software, which backups my photos and a few other things to an external USB-drive. I've never had to "go back and retrieve a file" so far, so it's there as a safety net more than anything. One thing I have noticed about this software is that it's really slow, especially when making the first "full backup".

If there's a better software for Windows out there, I'd like to hear about it.

So this is my system at the moment, but it's incomplete and relies on me to make the backups (which isn't good). All help regarding getting this backup system improved is much appreciated.

If there are any questions, I'll try to answer them asap.

Janne

 

Hi Jane and welcome to the wilders forum.

What OS do you use?
As for the raid support... any WinPE based imaging application will have no problem with a raid (as long as the drivers are included in the WinPE)

For OS BackUP:
Windows 7 backup is pretty good for what you want.
Other free options are:
Keriver 1-click restore free
Macrium Reflect free
Paragon Backup and recovery
Active Image Protector free
Redo BackUp and Recovery

Regular Backups:
Ocster Backup free
cobian
FBackup
BackUp Maker
Synchredible
Syncrify (free for personal use)
BitReplica
and of course Duplicati (you already know about that)

ps. I only posted free products.

Panagiotis

 

Thanks a bunch for the suggestions Panagiotis. I'll have a look around them once I get home from work.

And yes, I forgot to specify the OS. I'm using Windows 7 ultimate 64bit and my server is running Ubuntu server 10.04LTS.

More suggestions and recommendations about software are welcome and if you have a better way of doing some part of the backup system, please chime in.

Thank you in advance.

Janne

 

Hey again,

@Panagiotis

I checked out your suggestions for backing up the OS and here's what I found:

Keriver 1-click restore seems to be mostly for hot imaging, although there is a notion about the ability to make an image in the recovery console. It seems that the software isn't actively supported anymore since the "New" release is now over an year old.

Macrium Reflect free It's been a while since I last used Macrium, probably somewhere around 4.x. The new version has gotten better feature-wise and seems to support RAID-systems as well. This could work since if I recall right, cold imaging was possible with the bootCD. Although to make the CD it has to be installed, but at this point it doesn't matter since I'm re-installing Windows in any case.

Paragon Backup & Recovery I've used this a few times, but if I recall right, it only supports hot imaging. And since I don't need scheduling and incremental image support it's in a way overkill for me.

AIP seems to support only hot imaging and also suffer from the same situation as Keriver. No active development that is.

Redo Backup & Recovery This seemed really interesting, an open source software, nothing to install and works with cold images (and also has other tools that could come in handy). I downloaded it already and have to do some testing, but for now it seems that it's either this or possibly the utility built in the Windows 7 I'll be using.

Next I'll check out the backup software suggestions, so until then.

And thanks again for the suggestions, I'm feeling that I'm closer to a whole solution already.

Janne

 

Here are my findings about the suggested backup software:

Ocster backup free Could work otherwise, but it doesn't support backing up to a network location, so it's a no-go.

Cobian backup I've used an older version of Cobian (probably v. 8.) and for some reason the software was really unstable at that point. It's a feature-rich program with an unnecessarily complicated UI. Also I'm not sure if it supports a scheduled change between full and incremental backups meaning a full backup once a month and weekly incremental backups.

FBackup Seems like a good software, but if I understood correctly, it doesn't support incremental backups.

Backup Maker & Synchredible I've used the Backup Maker, but the nag screen of the free version drove me up the damn wall That could of course be taken care of by buying the software, but there seems to be better options around.

Syncrify This could be an interesting way of setting up a personal cloud service, but for just disk to disk backups in the same LAN it seems a complicated way of doing things. I'll keep this in mind if I at some point can place a remote server and set up a backup scenario to it.

BitReplica Like the Redo Backup & Recovery, this software seems really good to me. There was no mention about backup destination choices, but I'll install it and see for myself. A definite plus is the ability to export program settings (since I re-install Windows quite often) and that the backups are not in a proprietary format.

Duplicati This was the software I mentioned in the first post and it still seems solid, but somehow BitReplica seems even better. I think the biggest problem with Duplicati is the UI. Everything has to be done via their wizard which seems like an odd choice. Also there was a notion about Duplicati being slow with the backups.

Now I have several good suggestions and have to test the candidates to see what will fit my system best. But again, if you have something to add, please chime in.

Janne

 

I'm continuing my monologue here and hopefully someone will find this information useful.

I finished pre-testing the BitReplica software and it doesn't seem bad. I made a small test scenario with folders A and B with RAW-files in them (about 5Gb) and also folder C, which was empty.

Next I created a new profile for BitReplica with following settings:

• What: The parent folder which contains folder A, B and C
• Where: A network location on my Linux server
• How: Incremental backups with full backup after two incremental sets
• How2: Keep two full backups
• When: Manually so that I could run backups every few minutes

I measured my network activity with NetWorx monitor to get a general feel about the backup speed.

So at this point I ran the first (full) backup, which was successfully created in it's own folder to my server. The speeds were quite constant between 50-60Mb/s. I can transfer files to my server at around 105Mb/s, so it's not even close to that, but it isn't that bad either.

After the first backup I copied 8 RAW-files from folder A to folder C. This added about 180Mb to the total amount.

I ran the profile again, which made an incremental backup like it was supposed to do. But this is where things got interesting. I didn't do anything else besides copy the 180Mb worth of files and still the software copied over 3Gb to the server. For the next run I copied another 8 files and again the copied size was over 3Gb.

This scenario was repeated until the third full backup was made since I wanted to see if the software would automatically remove the first full- and two incremental backups, which it did.

I'll have to dig into this a bit further since I don't think the incremental backups should work this way. It could be that there's some overhead (the 3Gb) which wouldn't be bad at all, but if it's a percentage (over 50%) of the original amount of data it will be a deal breaker. I don't want to copy 300Gb + new and changed files every week. Space itself wouldn't be an issue since I have 6x1Tb RAID-6 setup on my server, so there's 4Tb of usable space.

Next I'll test Duplicati in the same way, so I'll report back after that.

If someone has an explanation for me about the incremental backups or just something else, go ahead and post them here.

Janne

 

You are welcome.

both Keriver 1-click restore and AIP are actively supported.

As for bitreplica, the overhead that you see could be due to different timestamps between the source and the destination.

Panagiotis

 

Thanks for the info on the timestamps. I did a quick test just making the backups to another HD on the same machine and no overhead.

Do you have an idea about what causes this phenomenon between my server and workstation? I'll try to update both machines via NTP and see if that helps.

Thanks again,

Janne

 

I added both machines to my local NTP pool and checked that the times on both were same after that.

This trick didn't help with BitReplica, I ran a full backup (5.33Gb) and after adding a few files which were about 10Mb combined the incremental backup was over 2.8Gb.

Cobian backup on the other hand did the first full (5.33Gb) and next an incremental backup (10Mb) right. It did however copy an empty folder from the original data for some reason.

I'll contact Auslogics and see if they have an explanation or a solution to this issue. If someone knows the reason for this behavior though, please share it here.

Janne

 

I'm curious about this too.

Panagiotis

 

I sent a question form from their website and they should get back to me on that in the next 2-4 days (not too fast, but hey, the program is free).

If someone knows a reason or a way to avoid this issue they are welcome to share it here.

On a completely different notion, I did try a few backup solutions after the initial BitReplica testing earlier today.

Duplicati was a huge disappointment, even the 5.4Gb test scenario took ages to backup initially, so I didn't even bother with testing incrementals. This software might be ok when you have to compress, split and encrypt the backup for uploading it to a cloud. It's definitely not for me though.

Cobian backup seems to have come a long way since I last used it. The UI is much more comprehensible than it was before and making a backup job was quite simple. It also supports full backups with several incrementals in between (I thought it would, but I wasn't sure since it wasn't mentioned on the website).

This far I like BitReplica the best overall, but there is a problem with the overhead at the moment. Cobian also beats BR in transfer speeds which is also an important thing to consider when the amount of data is this big. So if things can't get any better for BitReplica and there are no more good candidates, I think it's going to be Cobian for me.

Janne

 

there is another one
FreeFileSync
http://freefilesync.sourceforge.net/usage.php#features

Panagiotis

 

Thanks for the suggestion Panagiotis.

It doesn't seem to be a true backup software, but I'll keep that in mind as an alternative to Synctoy.

Janne

 

I recommend you dump your RAID0 setup for the OS partition. Get a large enough SSD to hold your OS and common apps (I recommend the Sansumg 830 or Crucial m4). In most cases a 128GB SSD is more than enough. By keeping your OS partition native, you have a much better chance of successful full disk image restores. And, stop using Acronis unless you're willing to go with their business products.

Your data volumes (the large ones holding your photos, etc.) should be 2 x large HDD's in RAID 1. Stick to RAID 1 whenever possible. In the event the RAID1 needs to be broken, the disks can be read individually. A huge plus. Yes, you should still backup RAID 1 setups. RAID is NOT BACKUP. RAID is for fault tolerance only.

I recommend you use a common Imaging app for your OS disk. Macrium Reflect, Symantec SSR, Shadowprotect, Image for Windows would be my choices. Symantec SSR allows imaging to a secondary offsite location.

For backup, Cobain is great and free but lacks restore. Some people don't realize that until it comes time to restore a backup. Its a bit more work, but typically easy to restore data using Cobian. Cobain V11 does allow base and incremental or differential backups on schedule. Great free program. If you're willing to pay look at Syncback SE/Pro or KLS Backup Pro. My 2 fav's. Low on resources, fast, reliable, etc..

Syncback will also do continuous backup as well. You can also look at Altaro Oops Backup for continuous backup. I am no fan of Genie products. They've become bloated and inefficient.

Thats my 2 cents. Good luck.

 

Hey Claykin, thanks for the input.

I thought that I mentioned that before, but there's a lot of text, so it was hard to find even for me. But yes, I do have a Crucial M4 128Gb waiting to be installed, so the RAID-support is not going to be a big deal after the upgrade. The reason for using RAID-0 was that I have two (first or second gen.) 60Gb SSDs which are too small without putting them into RAID-0.

This is in the works, I have a couple of 1Tb 7200rpm drives (big enough for now), which are going to become my photo/Lightroom catalog drives in RAID-1. Before that I need to get a larger single drive for all my other stuff, so when I get that the photos are going to be put in a RAID-1 array.

I'll take a look at these, any suggestions as to which would be the best?

I've run a few tests using Cobian and so far I like it and it has behaved the way it's supposed to behave (even where BitReplica failed). The lack of restore feature doesn't necessarily hold me back, but I'll have a look at Syncback and KLS Backup as well.

I tend to agree here about Genie software and will definitely look into your suggestions. I do have a paid license to the older Timeline and it's done it job quite well, but I'd say there's room for improvement.

Thank you, luck is very much needed

Janne

 

I made a small graphic representation of the backup setup so far:



- So incoming and created files (+ backups from my website) either go through or are stored on my workstation. The RAID-1 setup isn't implemented yet, but it will be in the near future.

- The OS is protected by imaging software, but ideally I just use the created disk images to make re-installing Windows easier.

- There is a continuous backup system, which protects files that are not yet backed up and for example files on my desktop etc.

- I'm thinking about running the scheduled backups (which go to my Linux server) once a week with two full backups available at all times. Full backups would be made monthly, so it would be full-inc-inc-full.

- On a monthly schedule I'd run manually a backup to an external drive which is stored offsite and also push an encrypted archive of documents to my Dropbox.

Of course organization is a key thing, so culling the photos is important to keep the backups as small as possible. Also I'm organizing my documents etc. to "Live work" - folder and an "Archive" - folder depending on if I'm working on them or not.

I might revise the scheduling in the future if my working habits change, but for now this could be enough. This however is merely a plan, so if you think there's something worth changing I'd like to know about it.

Janne

 

I looked at some 40 backup solution for my windows laptop (March-July 2012). Ocster (pro version) was almost the only one that fulfilled my most important requirements.

In my tests it does backup to network locations such as Windows mapped LAN/VPN drives, MS skydrive, SugarSync, G drive. With one exception. I could not get it working through NetDrive mapping an FTP site.

Most of the backup solutions I tried failed on one or more of just 6 requirements.
1) A running or scheduled backup should not fail when closing my laptop, and taking it out of a network range. (Or the backup device, including USB, becomes unavailable)
2) Automatic resuming of an interrupted backup run when the backup device is in range again. (not restarting a backup job all over again)
3) Backup data should be secured (encrypted, and supplier should give insight in the security solution).
4) Automatic pausing of running backup jobs when using my laptop(lowering priority of backup runs when using my laptop)
5) Scheduled and on demand file backups (I was not interested in image backups)
6) Local (USB, NAS) and off-site backups
And for free I got a nice to have: Incremental backups based on changed blocks within a changed file. This makes backing up Thunderbird email files of a few hundred MB efficient (or even feasible) in case a few mail message had been added to a mail map.

Since this is also a security forum: did anybody look or have a good reference to security in backups?
For off-site backups this seems like a necessity for me. I'd like to have more info on details as:
- If I don't have to enter your password or encryption key when a backup job starts, where is that key stored? (in case my laptop is lost)
- How is the password or encryption key used in deriving the real encryption key? (Or is it just uses as the key?)
- Is the encryption done locally on my PC or on the server?
- How well is AES or the encrytion method used, implemented?
Or should I start a new thread for that kind of a discussion?

== edit:
- forgot requirement 6, which was difficult to meet by cloud backup solutions. Those solutions were strong in req 1, 2 and 4, and yes, failed on my thoughts on 3 too.
- incremental backup on block level changes

 

Hey Dick99999,

Thanks for your input about Ocster. I think it's only the freeware version that doesn't support network locations for backups.

Your list of requirements is pretty much echoing what I have in mind also. Did you by any chance try Altaro Backup FS since I'd like to add a continuous backup to the mix? The point being that I'd like to move on from the current solution (Genie Timeline) and have only one software controlling the backup jobs to different destinations.

I've also considered Crashplan since it's pretty much the only reasonable priced cloud service with unlimited plan, but there are some controversial user experiences floating around. Mainly Crashplan losing or "temporarily" changing the location of backups in their own systems. The functionality and features seem to be quite good and if the UI works more reliably than it did a few years ago, it might be worth checking out. Anyone have experiences with them?

For the security of backups, please use this thread for conversation. It's something that I'm also interested, although I don't necessarily have too much to say about it myself.

Janne

P.S. Still not a word from Auslogics, that's one long-a** response time

 

Yes I did look at Altaro and asked about Laptops leaving the network while backing up. Very responsive company and I really liked the product. The answer in this thread is from feb 2012 though.

 

Thanks for the info and the link.

For me this is pretty much a non-issue since I only have one laptop and I rarely use it. The rest are desktop machines.

I assume that this is an issue also with the BackupFS and not only Oops!Backup and I totally understand that you went with Ocster because of it. How did Altaro fare otherwise in your tests?

Janne

 

I stopped looking at FS when concluding form the description, that it was targeted at backing up servers. Like image backup, server backup was not applicable for me. If you really want to use Ocster Pro, Google for something like:
Ocster Backup Pro 7 newest download unlocked code full version
If you like somewhat more flexibility at the expense of a little easy of use, I'd also look at BackupMaker. It did not satisfy my 'can move laptop out of network coverage while backing up' requirement. (Same for allowing boots while backing up btw.)

B.t.w there was one implicit requirement, I did not mention. I was looking for a 'real' backup program and found backup programs that:

• Copy: Just bulk copy files like dragging files form a source location and drop it onto a destination.
  A renamed or deleted file/dir will still exists on the destination. No versions of older files, they are overwritten by the newest 'version'.
• Synchronize: the destination would become an exact mirror copy of the source.
  Handy for file sharing. Deleted files on the source, will be also be deleted on the mirror copy. No versions. Using Synchronize as a backup it is very dangerous: accidentally deleted files/dirs are also deleted on the destination! Some programs, however, perform a copy and call it synchronize.
• Backup: All files (Full BU) or only files that changed (Incremental/Differential/Delta BU modes) will be added to the destination.
  Already existing files will get a version mark. Old versions can be automatically deleted on the destination after a retention period set by the user. In addition to the backup mode, some do include a synchronize mode also.

==== edit
Should have listed a 4-th copy mode: an image copy, some regular backup programs include that mode also.