+- Antwise community forums (https://forums.antwise.com)
+-- Forum: General Info and Questions (https://forums.antwise.com/forumdisplay.php?fid=3)
+--- Forum: General Questions and Answers (https://forums.antwise.com/forumdisplay.php?fid=7)
+--- Thread: GENERAL QUESTIONS - speed (/showthread.php?tid=64)
GENERAL QUESTIONS - speed - BCA - 2021-03-08
Hello.
Let's assume that I want to make a backup as often as it is possible.
- VMDK - 120 GB- system windows10 + database. Placed on SSD - about 550 MB read/write speed.
- phisical backup disk - about 550 MB read/write
Let's assume processor is about 3,6 -4,00 GHz -new generation - i5 - 4 real cores, 4 threads.
After 10 minuts the windows 10 goes, 50 MB data changed on disc 120GB VMDK- in database mainly - no windows updates etc.
Vimalin makes copy.
QUESTION:
From the start to the end - how quick is it done? How many minutes?
1. with packing method?
2. without packing method - without checking if the backup is correct (I know Vimalin can skip checking -
I saw somewhere in forum about this possibility)
Try to guess using your experience. I expect correct answer +/-20% of the reality.
This is very important for me to think to pay for VMWare Player + second windows licence + Vimalin - if it takes to long - no sense.
Thanx in advanced.
Best regards
Adam
GENERAL QUESTIONS - speed - wila - 2021-03-08
Hello Adam,
What you would want for running backups every 10 minutes is block change tracking and none of the VMware desktop products currently offer block change tracking.
This is a feature for the data center products from VMware, not for the desktop products.
So that would mean that I would have to implement that myself and while that certainly would be an interesting challenge, it is currently out of scope.
What Vimalin currently does is to make full VM copies for every backup.
So each and every backup is 120GB in size (probably more if the VM is running as each backup will also contain RAM and VM state)
Running constant backups plus having constant backups snapshots (create and commit) is not going to make you happy with the constraints you have in mind.
For these kind of backup demands, you're probably have to go looking for an in-guest backup method.
Sorry that I do not have better news for you.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-08
Thankl you for such a qucik reply. I understand how Vimalin works. In general:
1. start backup
2. VMWare Workstation makes delta files - let's say 50 MB has changed
3. base VMDK is beeing copied [VImalin]
4. base RAM is also copied [Vimalin]
5. other config (small in size) are copied [Vimalin]
6. VMWAre joins delta files (those 50MB) with base VMDK
7. The same with RAM (maybe I am not sure)
8, This is the END of whole process.
How do you think, how long will it takes in your opinion, these steps from 1 - 8 - hardware like given above? 15, 20 ,30 min?
GENERAL QUESTIONS - speed - wila - 2021-03-09
The time it takes is limited by the hardware.
Some notes on that.
Copying is done by a single thread by the vimalinWorker process. The copying itself is done via standard Windows API calls, nothing special there.
If Windows can copy fast, then so can Vimalin.
That thread does not have any UI interactions.
UI updates while copying is something that tends to slow down copying significantly and this is my guess why some users have reported Vimalin being faster than copying via the OS.
Windows being helpful and counting all the files before starting the copy is not good there either as it will copy a lot of small files you don't even need in your backups.
You can see progress from the GUI, but that runs in another process and does not impact the copy process performance itself.
As a small test I just ran a 12GB backup on a running VM and it took me 190 seconds, but that's with the md5 hashes enabled.
I then followed the steps from this post:
https://forums.vimalin.com/viewtopic.php?pid=89#p89
and killed the vimalinWorker process from task manager to force it to read the new settings.
.. and it then took 154 seconds. Aka, making the md5 hashes is not that slow from my POV.
Note that my test in this case is using nested virtualisation (running a Windows 7 VM in Workstation in a Windows 10 VM within Fusion and running Vimalin in that Windows 10 VM as well), so not as fast as it can be on a physical host, but it can help to give you an idea.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-09
OK - thanx for such detailed explanation. But you missed that most important. What kind of components does your machine have?
Discs!
Todays, in generall, we have 3 kind of disc:
1 old HDD (about 100-120 MB/s speed)
2 the most popular classical SSD (about 550 MB/s speed)
3 ultra "speedy" NVMe M.2 (about 3200 MB/s speed, Samsung Pro M.2 MVMe).
Read and write are almost the same
So, please tell me:
- your source disc speed (vm)
- your target disc speed (backup of vm) during test for those 154/190s for 12 GB.
If you have a raid, try to figure out its write speed about (+/-20 %)
Other people if they read this thread they will have a view of what they can expect from Vimalin + Workstation + Windows 10.
GENERAL QUESTIONS - speed - wila - 2021-03-09
Adam,
I left it out because my test is flawed from a pure performance measurement point of view.
The test is in my development test area which in this particular case was a backup from a local VM disk to another local VM disk.
The Windows 10 VM in which I ran that test is running on a recent macbook pro and as such it is has M2 backing.
However... a VM disk never sees that M2 kind of performance.
Instead it is much closer to your physical SATA speed for reading and writing as to actual M2 speed because I'm backing up the inner nested VM to the outer VM virtual disk.
To give you an alternative measurement I also test a lot against network based backups. One of those tests is against a Synology DS218 with SoC Realtec RTD1296 and 2 disks (Hitachi 3TB SATA) and then the same backup takes 279 seconds. So as you can see, the M2 backing really isn't running at full speed, the Synology is limited by a 1Gb/s network, I'd be happy if it hits 90MB/s there, but file copying from within a VM is slower due to the nature of going through a virtual network card.
Back of the envelope calculation on the VM backup I made is: 12*1024/279=44MB/s so there you go, not even close to 90MB/s.
Btw, this is still with the hash calculation turned off.
If your hardware is really giving that 550 MB/s (which I've only rarely seen a SSD disk to actually do in real life situations, unless the disk is empty and very new) then I expect that Vimalin is getting close to that physical limit.
But there are so many factors to take into account when testing that I'm not surprised if it is less performant.
--
Wil
GENERAL QUESTIONS - speed - wila - 2021-03-09
More data points..
The taking of the backup snapshot itself was already about 45 seconds.. so on a small VM like this that isn't helping with the actual performance calculation, and it would have been more accurate to do 12*1024/(279-45) = 52MB/s and I'm not even counting committing the snapshot.
My outer VM only has 2 cores as well, so not very over specced there either.
It really is best to do your own tests as it will differ a lot depending on the hardware you run it on.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-11
Hello again.
I have a question Wil - may I give you an idea to make your product perfect?
In my opinion very good idea in a context of speed.
Are you interrested in developing this product or maybe you are going not to improve this any more?
I mean some module that you can also sell for 15$ like vimarun? I think everybody would buy it together with Vimalin base.
Even more - it would be quite easy to implement for you.
GENERAL QUESTIONS - speed - wila - 2021-03-11
Adam,
There's a new version of Vimalin on the way, it's been delayed a bit due to some personal stuff, but should come.
Vimalin is constantly improving and I intend to keep that going.
I have ideas on making a rolling backup using snapshots, if that's what you mean.
Not as easy as it seems, but might be do-able and is on my list of things to investigate.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-11
I am not a big specialist in virtualization world so tell me: is there something like incremental (differencial) vmware backup?
The base copy - the biggest one and next files of vmram and vmdiscs as patches for the base? Recovering = base vmware backup + all incremental/differencial backups? Is there such solution in VMWare Workstation? Is that what you meant when you were wrting: "I have ideas on making a rolling backup using snapshots" ?
GENERAL QUESTIONS - speed - wila - 2021-03-11
Hi,
I'm not aware of anything you can use for that right now.
The best alternative for now would be to use a full backup on a regular time (say daily) and do a data backup from inside the VM on a higher frequency.
You talked about a database and in that case I would suggest a database backup to increase the backup frequency.
re. "Is that what you mean".
Basically yes, but it is a bit more tricky as you don't want to get a VM with a lot of snapshots.
So you would keep on committing the data in the delta and then backup the delta (+RAM+STATE) on each backup.
The delta would then grow over time.
As this type of backup is more fragile over time, there would be a new base backup on regular times, like daily, too.
The problem of this type of backup is that it is harder to test.
It is not just writing the code, it is just as important (more important even) to make sure it is correct as having failing restore is really disappointing.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-11
deleted
GENERAL QUESTIONS - speed - wila - 2021-03-12
Hi,
[removed answer to long post]
Back to your problem.
When I said, "look at a database server backup solution" I was serious.
They are very good at doing differential backups as there's features for doing so built in almost all of the database server back ends.
You never mentioned what DB software you are using.
What I can suggest is to take a look at:
https://sqlbackupandftp.com/
I have used them for several of my customers running MSSQL servers and I can recommend them for that.
--
Wil
GENERAL QUESTIONS - speed - BCA - 2021-03-12
Yes, in fact, to big message. So I deleted it. Thanx for reply. Bye!
GENERAL QUESTIONS - speed - wila - 2021-03-12
Hi,
Don't worry. I do appreciate your ideas and the time it must have taken to make that suggestion.
Sadly I cannot always implement every suggestion.
I have also removed my part of the reply that addressed the long story as it makes little sense without the original text it was replying to.
--
Wil
GENERAL QUESTIONS - speed - wila - 2021-03-12
FWIW, I've reread your long post today and it makes more sense to me now.
The "RAID0" statement was used as "similar to how RAID0 does" not as "use RAID0 for this".
Your basic idea works, but it is not a path I want to take Vimalin to as it is too complex and has increased risks of failure and complicates support.
If it is for internal use by a technical person then it could work.
This would typically fall in a "script myself a solution" area, that works perfectly fine for a one off scenario, but is harder for a commercial product to do.
--
Wil