Questions to Add to FAQ and Vimalin Feature Requests

[vmxmr]

* How much will it cost when it is released?
* How will it be licensed? Can I use it on multiple personally owned Macs (with a similar license to VMware)? I run virtual machines on my desktop iMac, a Mac mini server (the firewall is a virtual machine appliance), and occasionally a MacBook Pro laptop when I am away from home.
* How does it work? Does it shutdown the virtual machine first, then copy the entire virtual machine package, then restart the virtual machine?
Note: I would like a backup product that can be configured to shutdown the virtual machine before the backup starts, to guarantee that it is in a restorable state if I should ever need the backup. Some of my virtual machines run as Windows Servers with open, active databases on them.
* Does it check the running state of the virtual machine (running, suspended, or shutdown?) in order to leave it in the same state when it is done?
* How does it manage backup file accumulation?
Note: I would like to keep a certain number of backup copies (say, two month's worth of weekly backups), and then have the backup program start deleting the old backups to avoid running out of disk space.
* Can it perform other actions, such as a Disk Cleanup, before restarting the virtual machine?
* Can I configure it to run one of my shell scripts when a backup completes?
* It would be nice if it could be programmed to compress the backup, save an MD5 hash of the compressed backup, and then delete the original backup, for example, which would save a huge amount of disk space.

[wila]

Hello vmxvmr,

Welcome and thanks for registering.
I answered your questions in the VMware forums, but it will make sense to put the answers here as well. It is probably even better to continue the discussion down here instead of over at the VMware forums.

Let's try to answer your questions,btw thanks for your questions as it helps me to better understand what is needed/wanted.

* How much will it cost when it is released?

It might sound weird that for a product that I have spent so much time on developing that I do not have a definitive answer on that yet.

If you want a ballpark number then the only thing I am willing to say is "less then 40 Euro", but I do not have an exact price at this moment.

* How will it be licensed? Can I use it on multiple personally owned Macs (with a similar license to VMware)? I run virtual machines on my desktop iMac, a Mac mini server (the firewall is a virtual machine appliance), and occasionally a MacBook Pro laptop when I am away from home.

This is also an open question. I will look into this.

* How does it work? Does it shutdown the virtual machine first, then copy the entire virtual machine package, then restart the virtual machine?
Note: I would like a backup product that can be configured to shutdown the virtual machine before the backup starts, to guarantee that it is in a restorable state if I should ever need the backup. Some of my virtual machines run as Windows Servers with open, active databases on them

You might not be the typical user, I assumed most users would not run servers with databases as VMs, of course the first question I get is someone who runs servers with databases .

For most people it would not be a nice thing to shut down the VM before a backup as it could interfere with their work.

The way it works in global lines is that Vimalin detects if your VM is running or not and if it runs it will run a snapshot in the background, then it copies the full set of data and commits the snapshot after the copy. So when you restore the backup you get back to the state of the snapshot. For a server there are basically two options, like you say shut down completely and start it up afterwards, or run the snapshot with quiescence so that files are forced to be written.

At the moment it isn't offering neither as I need to find a way to keep things easy to configure too. I'll add it as a feature request.

* Does it check the running state of the virtual machine (running, suspended, or shutdown?) in order to leave it in the same state when it is done?

Yes, at the moment it does not change the state, just adjusts how to backup the VM.

* How does it manage backup file accumulation?

Note: I would like to keep a certain number of backup copies (say, two month's worth of weekly backups), and then have the backup program start deleting the old backups to avoid running out of disk space.

All of that is fairly easy to setup in the scheduler. You can roll over data and assign multiple schedules to a single VM to tweak it exactly to your likings.

* Can it perform other actions, such as a Disk Cleanup, before restarting the virtual machine?

Not at this moment. Which is logical as Vimalin doesn't restart the VM. Interesting idea though had not considered it.

* Can I configure it to run one of my shell scripts when a backup completes?

Also never considered that so not at this moment, but it sounds interesting. Beware though that if you change the files (like compressing) that the md5sums no longer match and the file sizes will be off so Vimalin will complain if you select the backup job in the backup archive screen.

* It would be nice if it could be programmed to compress the backup, save an MD5 hash of the compressed backup, and then delete the original backup, for example, which would save a huge amount of disk space.

It does make an md5 hash of every file and store that in the archive database. I considered compressing the archives, but as it requires extra testing and making sure that no things happen like filling up a disk that has a VM running on it I have postponed this functionality until the basics work well. So the first version of Vimalin is more basic as I would have wanted, but it is best to first have a proven rock solid base as to have a lot of fancy features.

Having said that, it is high on my wish list and parts of the application have been designed to foresee this.



Hope this helps,

--

Wil

[vmxmr]

Goedemorgen Wil,

Thank you for your responses to my questions. Here are my actual use cases:

VIMALIN NOT REALLY NEEDED, BUT MIGHT BE HELPFUL:

1. Home: I have many virtual machines scattered on three Macs.

1a. iMac on desktop. This is where I do most of my work. I have 16 active virtual machines on the iMac, and many more on external drives which I almost never use, but they are available if needed. Backups are easy - I shut them down, compress them, and move the compressed file to a "virtual machines backups" folder. The virtual machines folder is excluded from Time Machine, but the virtual machines backup folder is not, so the backups get copied to Time Machine. I also make copies to external drives from time to time.

1b. Mac mini. It runs one virtual machine, which is a business firewall appliance. It serves as the firewall for the entire home. Before I update the firewall, I shutdown the virtual machine (which stops internet access) and make a copy of it, which takes ~15 minutes. I restart the virtual machine, compress the copy, and back it up to Time Machine and external drives.

1c. MacBook Pro. I have some Windows, Linux, and Mac virtual machines on it for when I travel or go to customer sites. They allow me to run platform-specific software, test compatibility, etc. I don't need to back them up.

NOTE: None of the above are an essential use case for your Vimalin backup program. I don't need it for them. My system works, and it does not interfere with my work or other needs.


HERE IS THE USE CASE WHERE I COULD USE HELP. VIMALIN MIGHT WORK, IF IT WERE IMPROVED TO MEET MY REQUIREMENTS:

2. Customer. I have a customer who insists on running Windows Server from a Mac mini. I set it up with VMware Fusion, which they purchased. (I once posted a procedure on VMware's forums for auto-starting this system.) This customer runs specialized software for their business. The software is integrated with an active Microsoft SQL Server database. It is complex, to say the least. Once a week (usually very early Sunday mornings), I connect to Windows Server and shut it down properly, which shuts down the virtual machine.

Weekly backup: Copy the virtual machine to a backup folder. If VMware shows that I can recover more than 500 Mbytes of space in disk cleanup, then I do it. Restart the virtual machine and notify customer personnel that it is available for use. After that: Compress the backup to a ZIP file. Compute an MD5 hash of the ZIP file and save it to a text file. Copy the ZIP file and MD5 text file to an external drive that is attached to the Mac mini. Verify the MD5 hash for the ZIP file on the external drive. Copy the ZIP file and MD5 text file over an SFTP connection across the internet to secure offsite storage. Verify the MD5 hash for the offsite copy. Done.

Monthly backup: Same as above, but mix in: Run a diff between the backup copy and the original shutdown virtual machine. Perform basic file and disk cleanup on the Mac and Windows Server. Update Mac mini with Apple updates. Update Mac applications that I use on that Mac. Update Windows Server with the monthly "Patch Tuesday" updates. Update the specialized software on the Windows Server. Run more extensive operational validation tests with the specialized software that runs on the Windows Server.

Additional Notes:
* The diff has never failed. The copies seem to be 100% reliable.
* I trust Apple's Compress to ZIP feature in Finder. I ran tests with diff, and found that the results of uncompressing the ZIP are identical to the original.
-> This implies that the Vimalin MD5 file should still be valid after a Compress and uncompress cycle.

What Vimalin could potentially do for me:
* Save me from having to wake up every Sunday morning and:
* Shutdown the server at a known time every Sunday.
* Make the full copy.
* Restart the server.
* Delete old copies to prevent disk overflow.

... and add as feature requests:
* Compress the copy to a ZIP file.
* Compute and store an MD5 hash with the ZIP file.
* Copy the ZIP file and its corresponding MD5 hash to an external drive.
* Verify the MD5 hash of the ZIP file on the external drive.
* Use SFTP to push a copy of the ZIP file and its corresponding MD5 hash to secure offsite storage.
* Verify the MD5 hash of the ZIP file on the offsite storage.

I realize that most of the above would best be done with a user-written shell script that could be launched when the Vimalin backup completes.

ONE BIG ISSUE:
I have never had confirmation that using using VMware's "Virtual Machine -> Shut Down" from the menu or "vmrun stop" is the exact equivalent of shutting down a Windows or Linux virtual machine from within the running virtual machine itself. I do not want to risk corrupting a running, active database.

Summary:
* Vimalin is a nice product, but I don't really need it for home use.
* Vimalin would be a helpful product for my customer's site if could be sure that it could create a reliably restorable backup. It must maintain database integrity and avoid data corruption, mostly likely by performing a reliable shutdown and restart of the customer's virtual machine. The extra features I listed above (such as launching a user-defined shell script after completion) would turn it from "helpful" to "nearly perfect" !!

-> May I suggest that you look at the Carbon Copy Cloner application to see an example of how Vimalin could work? Carbon Copy Cloner is a mature Mac application with many helpful features. It would serve as a good model for Vimalin.

[wila]

Hello,

Backups are easy - I shut them down, compress them, and move the compressed file to a "virtual machines backups" folder. The virtual machines folder is excluded from Time Machine, but the virtual machines backup folder is not, so the backups get copied to Time Machine.

Yes I read your approach on this before, it can work. There are a few problems that I see with this however.
While Time Machine is great for many scenarios, VMs are not a very fitting scenario. Why? Well there's a long explanation, which I will try to write out soon, but the short story on your solution is that it will work until your backup space runs out. At that moment the promise is that Time Machine will start by deleting the oldest first. Yes it does that, except when it doesn't.
VM files are big and Time Machine has a tendency to drop bigger files earlier. Here is a post of a recent victim on that in the VMware Fusion forum:
https://communities.vmware.com/thread/544749

Another more obvious issue is that it is easy to forget to make timely backups this way.
Slightly less obvious is that you have to know very well what you are doing, like forgetting to shutdown will get you a backup with corrupted disks or like making a backup on a disk with running VMs that then ends up corrupting the running VM because of running out of disk space.

The goal of Vimalin is to help customers to create more reliable backups without having to be an expert.

* I trust Apple's Compress to ZIP feature in Finder. I ran tests with diff, and found that the results of uncompressing the ZIP are identical to the original.
-> This implies that the Vimalin MD5 file should still be valid after a Compress and uncompress cycle.

Oh yes certainly. I might not have explained myself well. The problem is that if you zip the files outside of Vimalin that when you navigate over the backup archive, the files will be missing. So Vimalin will report your backup as broken as it does a quick list of files when you navigate in the backup archive screen. If you want to restore then you would have to unzip first.

What Vimalin could potentially do for me:
* Save me from having to wake up every Sunday morning and:
* Shutdown the server at a known time every Sunday.
* Make the full copy.
* Restart the server.
* Delete old copies to prevent disk overflow.

I've scripted myself a similar solution on VMware vSphere with some of my server VMs on there so understand where you are coming from. This is typically a need at Virtual Machine level. In other words, I am considering to add a checkbox in the "Select Virtual Machine" screen where you can decide to have that machine shut down gracefully for EVERY backup and then automatically restore it back to the power state it had before the backup.

I realize that most of the above would best be done with a user-written shell script that could be launched when the Vimalin backup completes.

I always figured that people who script their backup wouldn't be helped by Vimalin. How wrong could I be?
If you agree that it is OK not to have a user interface for handling this part (except perhaps a checkbox to enable/disable the functionality) then I have a few ideas on how to implement this best. For details on this I think it is best if we discuss this via email. I will contact you later today on this.

-> May I suggest that you look at the Carbon Copy Cloner application to see an example of how Vimalin could work? Carbon Copy Cloner is a mature Mac application with many helpful features. It would serve as a good model for Vimalin.

LOL, I will have a look at it, but for the moment I admit that my focus is on "getting things done" and redesigning the user interface certainly does not fit in that picture. The developer in me wants to rip it all down and rebuild in Swift, the realist in me knows that it would take a long time until I would be satisfied. Luckily the core of the system is in the worker process that handles the backup so I would certainly expect improvements in the user interface (GUI) over time, but it usually takes longer to get to that as expected.
FWIW in most programming languages building the GUI is what takes a lot of time, more as one would realistically expect, it is also the area that tends to get most bugs.

thanks,
--
Wil

[vmxmr]

Hi Wil,

I don't want to pull you in the wrong direction. Perhaps your market is with basic home users, and your focus should be there. Having worked on commercial software products before, I know that there is nothing worse than wasting a lot of time to satisfy one vocal customer who constantly whines about corner cases, instead of focusing on the broad market appeal of your product.

I urge you to take a few minutes to examine the Carbon Copy Cloner feature set and user interface. You have independently developed something very similar in both features and design. I understand "getting things done", but I also understand "NIH". Looking at CCC may help you avoid making poor decisions now that have implications for the future. Just sayin'.

I look forward to your email message.

[wila]

Hi,

You are not pulling me away in a different direction and I welcome your feedback (any feedback is welcome really)
Neither of those two things is particularly difficult to implement, most of the required code exists and it's all basically in the worker process, not GUI, which is easier to do.

I am not promising it will make the first official release, but it should not take that long either.

--
Wil