Home > Virtual Machine > How to backup XenServer virtual machines without service interruption

How to backup XenServer virtual machines without service interruption

August 7th, 2009
Goto comments Leave a comment

It’s quite a long time now I’m not posting on my blog: my last post was about the availability of VMWare ESXi for free.

Next to that VMWare marketing operation, Citrix decided that the time was right to do something to try to reach the competitor, VMWare. So Citrix decided to distribute XenServer 5 for free and with some great values added like central management (XenCenter) and live migration.

XenServer latest version (XenServer 5.5) introduces VM snapshot support and has the capability to create a virtual machine from an existing snapshot: the resulting VM will be a perfect clone of the original snapshotted VM.

Snapshotting requires only few seconds and is a non locking operation.

Jeremy Tirrel’s idea for a “0 downtime snapshot based XENServer 5.5 backup script” is based on these new features.

This Perl script executes the following workflow:

  1. Virtual Machine Snapshot
  2. Snapshot to new VM conversion
  3. new VM export operation to XVA (Xen Virtual Appliance) format
  4. new VM deletion
  5. snapshot deletion

Jeremy provides a backup rotation PHP script, too.

I used (and use, of course) his script with some minor modifications and I avoid using the PHP script for backup rotation leaving this operation to my preferred backup software, CrashPlan Pro by Code42.

My configuration follows:

  • 3 x HP ProLiant DL360G5, 10GB RAM (pooled)
    • 16 virtual machines
  • 1 iSCSI shared storage (http://www.starwindsoftware.com/)
  • 1 old server used for “staging”
  • CrashPlan Pro backup software

Jeremy’s backup script is executed on XenServer pool master every afternoon at 17.30 and exports single VMs from iSCSI shared storage to the staging server. Snapshotting/backup operations need about 8 hours to complete.

With my modification the script works on files with .xvatmp extension; as soon as the export operation completes, the file is renamed with a .xva extension, overwriting the previuos version.

CrashPlan Pro (http://www.crashplanpro.com) is a very good software for remote backup; it’s written using Java and uses the well known rsync-like approach to optimize bandwidth usage; being designed for Internet operations, it works like a charm over LAN!

CrashPlanPro client installed on the staging server is set up to skip files with .xvatmp extension (it’s useless to backup such files considering that they are a work in progress). As soon as the file is renamed with the .xva extension the file is transferred to the backup server. In my situation it needs about three hours to complete:

Completed backup to BackupSGR S in 3 hours: 16 files (269.09 GB) backed up, 21.64 GB encrypted and sent @ 1666.41 KB/s (Effective rate: 32699.14 KB/s)

CrashPlan manages backup rotation hence I do not need to use Jeremy’s PHP script.

Obviously you can subscribe to any online backup service plan and use backup software from different vendors; the only requirement is that the backup solution you choose supports your “staging” server OS (OpenFiler for me). If, like me, you want to use CrashPlan I can suggest ProntoBackup (well, it’s me of course!!)

Below you’ll fine the Perl code with the small modifications I made:

backup.pl

#!/usr/bin/perl
#
#XEN Server machine back, Jeremy Tirrell for Commercial Solutions INC 2009
#minor changes by Roberto Cespa

$backupdir = "/mnt/offsite/XenBackup/"; #Directory to backup to, this should be nfs share or something large.
#mail notification setup
$mailNotification = true; #false for no notifcaiton
$mailScript = "/usr/local/scripts/sendEmail"; #Brandon Zehm - Change "server" to point to smtp server
$MailTo = "address\@example.com"; #mail to
$MailFrom = "SomeOtherAddress\@example.com"; #mail from
$host = "Xen1 XenServer"; #Descriptive name
#Setup Strings
$Message = " \"XEN Backup Notification\n"; #header of message
$MailString = $mailScript . " -f " . $MailFrom . " -t " . $MailTo; #mail string to control string passed to sendEmail
$TopicHeader = "[".$host ." XEN Server Backup Script] "; #Subject.
$Topic = "";

#add chunks of uuid to skip backing up of a specifc vm
#control domain (dom0) should be added to this list!!
@skip = ('3a720810-a2fc-4025-97f1-860a1c9b6357','77822415-4e6b-46f6-bd82-821b2c1401c3','dce8ea94-c7d8-41e6-9232-630a27ab3b16');

$vmlist = `xe vm-list`; #Get the formatted list of guests
@lineList = split(/\n/,$vmlist); #Split the list of guests into and array of lines
@uuid = (); #Array to store uuid's in

foreach $line(@lineList){ #for each line in the array
if (substr($line,0,4) eq "uuid"){ #look for the word uuid at the beginning of the line
push(@uuid, substr($line,23,36)); #if its there add the uuid to the array
}
}

#@uuid=('116409d7-61b0-67a0-dbff-709e93053e00');
$tcurrent = `date`; #get the current date
print("Beginning backup of virtual machines at ". $tcurrent);
$Message .= "Beginning backup of virtual machines at ". $tcurrent."\n";

foreach $guest(@uuid){ #for each guest listed in the uuid list
if (grep $_ eq $guest, @skip){ #is guest on the skip list?
print("Skipping backup of: ".$guest ."\n"); #yes so just let it be known it is being skipped
$Message .= "Skipping backup of: ".$guest ."\n";
} else { #otherwise
$tcurrent = `date`; #get the current date
$VMName = `xe vm-list uuid=$guest | grep name-label | cut -b24-`;
$VMName=~ s/\n//g;
$VMName=~ s/\r//g;
$VMName =~ s/ /_/g;

print("Beginning backup of ".$guest ." @ ". $tcurrent);
$Message .= "Beginning backup of ".$guest ." @ ". $tcurrent."\n";
print("Taking a snapshot of: ".$guest ."\n"); #Snapshot the VM
$Message .= "Taking a snapshot of: ". $guest ."\n";

$snapshotUUID= `xe vm-snapshot vm=$guest new-name-label=$VMName-backup_vm`;
$snapshotUUID=~ s/\n//g; #remove the carrier return/linefeed
$snapshotUUID=~ s/\r//g;
print("Snapshot: ".$snapshotUUID." created\n");

$Message .= "Snapshot: ".$snapshotUUID." created\n";

print("turning ".$snapshotUUID ." snapshot into a vm\n"); #Turn snapshot into a VM
$Message .= "turning ".$snapshotUUID ." snapshot into a vm\n";
$status= `xe template-param-set is-a-template=false ha-always-run=false uuid=$snapshotUUID`;

print($status."\n");
$Message .= $status."\n";

$fdate = `date +%y%m%d-%H%M`;
$fdate=~ s/\n//g; #get the current date in a format we can write
$fdate=~ s/\r//g; #get the current date in a format we can write

print("Exporting: ".$snapshotUUID."\n"); #export the snapshot
$Message .= "Exporting: ".$snapshotUUID."\n";
$exportstring = $backupdir.$VMName."-".$fdate.".xvatmp";
$finalname = $backupdir.$VMName.".xva";
print("Name: $exportstring - $finalname\n");

$status= `xe vm-export vm=$snapshotUUID filename=$exportstring`;
print($status."\n");
$Message .= $status."\n";

print("Renaming backup file so that CrashPlan can backup it\n");
$status= `mv -vf $exportstring $finalname`;
print($status."\n");
$Message .= $status."\n";

print("Done, Removing snapshot: ".$snapshotUUID."\n"); #Done, remove the snapshot
$Message .= "Done, Removing snapshot: ".$snapshotUUID."\n";
$status= `xe vm-uninstall uuid=$snapshotUUID force=true`;
print($status."\n");
$Message .= $status."\n";

$tcurrent = `date`; #get the current date
print("Completed backup of ".$guest ." @ ". $tcurrent);

$Message .= "Completed backup of ".$guest ." @ ". $tcurrent."\n";

}

}

$tfinished = `date`;
print("Backup completed at ".$tfinished);
$Message .= "Backup completed at ".$tfinished."\n\"";

if ($mailNotification){
system($MailString . " -u " . $TopicHeader . $Topic ." -m " . $Message);
}

Roberto Cespa Virtual Machine , , , , ,

  1. P.Nacci
    August 7th, 2009 at 13:28 | #1
    Reply | Quote

    Interessante, avevo visto e “giocato” anch’io con quello script ma ho incontrato dei problemi sia di funzionamento che di “logica”:
    Lo snapshot di una VM accesa non è consistente,
    la macchina virtuale potrebbe avere più dischi virtuali (tipicamente sistema operativo/applicativi e area dati) e l’area dati potrebbe essere gigantesca e da gestire con un backup a livello applicativo (es. Exchange o SQL server),
    quindi è piuttosto pesante (oltre che lento) effettuare un backup dell’intera macchina.
    L’alternativa, che sto mettendo a punto proprio in questi giorni, è quella di effettuare uno snapshot delle singole VDI (singoli dischi collegati alle macchine virtuali) consentendo un backup “selettivo”. Resta sempre il fatto che la macchina accesa potrebbe generare un backup inconsistente (nel caso di snapshot di LUN su storage NETAPP o DELL giurano di no…) ma la soluzione di mettere in pausa o di spegnere la VM potrebbe risultare improponibile, e lo snapshot “quiesced” (xe vm-snapshot-with-quiesce) mi ha dato GROSSI problemi (intero volume marcato R/O e Hidden dal VSS).

  2. Roberto Cespa
    August 7th, 2009 at 13:47 | #2
    Reply | Quote

    Riguardo alla consistenza degli snapshot di una VM accesa ho anche io qualche dubbio, ma forse il mio caso è un po’ più semplice perché le 16 VM di cui parlo nell’articolo sono tutti desktop virtuali Windows XP; in linea di massima, quindi, lo snapshot inconsistente dovrebbe avere gli stessi effetti di una perdita di elettricità.
    L’unico vero server (SQL Server) è in conversione da VMWare in questi minuti ma per quello è prevista la procedura di backup e manutenzione di SQL Server; i file risultanti vengono poi spostati su un file server fisico.

    Personalmente ho fatto delle prove di recovery delle VM e nessuna di queste ha dato alcun problema né ha richiesto un chkdsk.
    Sarebbe interessante sapere cosa fa Xen quando converte lo snapshot in una VM: dubito che possa chiudere i vari handle delle applicazioni, ma in fondo mi chiedo ancora come funzioni l’idea di Citrix dei vari layers per i desktop virtuali in XenDesktop.

    Nella documentazione Citrix sostiene che uno snapshot sia “crash consistent” (http://docs.vmd.citrix.com/XenServer/5.5.0/1.0/en_gb/reference.html#id2584931) ma sarei curioso di capire bene cosa intendano per crash consistent.

  3. P.Nacci
    August 7th, 2009 at 17:20 | #3
    Reply | Quote

    Un avviso importante: se è abilitata l’HA e la macchina che deve essere esportata è protetta dall’HA non appena lo script imposta “is-a-template” a false XenServer avvia tale macchina, e questo è un problema, perché si tratta di una macchina “clone” senza più poterla gestire (spegnere, modificare ecc. ecc.).
    Per risolvere bisogna modificare la riga:
    $status= `xe template-param-set is-a-template=false uuid=$snapshotUUID`;
    in
    $status= `xe template-param-set is-a-template=false ha-always-run=false uuid=$snapshotUUID`;

  4. Roberto Cespa
    August 7th, 2009 at 17:53 | #4
    Reply | Quote

    Grazie per l’avviso.
    Mi sono permesso di tradurlo e di riportarlo anche sulla pagina del forum di Citrix: http://community.citrix.com/display/xs/0+downtime+snapshot+based+XENServer+5.5+backup+script.?focusedCommentId=79462767&#comment-79462767

    E’ un tipo di problematica che non ho assolutamente considerato, dato che uso la versione free di XenServer.

  5. Roberto Cespa
    August 7th, 2009 at 17:58 | #5
    Reply | Quote

    @P. Nacci
    Da quanto ho potuto capire per Crash Consistent si intende quello che immaginavo: è come se al computer togliessimo la corrente.
    Quindi rimane il problema dell’integrità dei dati tipica dei database o di Exchange.

    Citrix, comunque, insiste sul VSS anche in questo articolo: http://support.citrix.com/article/CTX122191

    Sembrebbe la soluzione più ovvia, ma tu hai detto di aver avuto parecchi problemi…

  6. P.Nacci
    August 7th, 2009 at 18:11 | #6
    Reply | Quote

    @Roberto Cespa
    I problemi li ho avuti con Xen 5.0 (su cui gira anche Marathon everRun VM), non so se siano risolti, comunque il problema era che lo snapshot partiva e dopo pochi secondi terminava con un errore che in taluni casi (VM Windows 2003 32 Bit) portava a una inaccessibilità dei volumi (=niente più boot!).
    Siccome sto parlando di un sistema “in produzione” e le coronarie quella volta hanno sobbalzato parecchio ho deciso di non fare più esperimenti con VSS, passando a backup applicativo per i dati (SQL, Exchange) e allo snapshot dei singoli dischi (snapshot VDI + LUN Clone) che è rapidissimo, ma resta sullo storage.

  7. Roberto Cespa
    August 7th, 2009 at 18:43 | #7
    Reply | Quote

    Purtroppo il mio storage non è né NetApp né niente di fascia alta; ho utilizzato un server con Win2K3 e il target iSCSI della StarWind.
    Sto cercando di fare il tutto spendendo il meno possibile (al limite anche zero).

    Nella sua incarnazione free, il target di StarWind non permette lo snapshotting.

    Tutto questo, inoltre, ha il grosso difetto legato ad un bug (secondo gli utenti) / feature (secondo Citrix) relativo agli snapshot su shared SR e LVM: http://forums.citrix.com/thread.jspa?threadID=249121&tstart=75 .

    Praticamente se faccio uno snapshot di un disco da 20GB e lo converto in una nuova VM per il backup (occupando 20GB+ x MB +20GB), poi non c’è più modo di riavere indietro quello spazio, nemmeno cancellando lo snapshot e la macchina da esso derivata.
    L’unica soluzione è di cancellare la macchina originaria (quella di cui è stato fatto lo snapshot) e reimportarla dal backup :(

    Quindi con uno storage da 800GB, puoi più o meno usare 380GB per le VM: altrettanti saranno necessari per i cloni e un po’ me li tengo di riserva per qualsiasi evenienza.
    Questa cosa, abbastanza fastidiosa quando hai dei vincoli economici, mi sta portando a valutare l’idea di farmi un file server NFS con 4 o 6 dischi SATA e metterci sopra Linux: NFS sotto Windows non ne vuole sapere di andare con performance decenti, almeno non con i file delle dimensioni di quelli usati per i VHD.

  8. Steve
    August 21st, 2009 at 01:56 | #8
    Reply | Quote

    I’ve not worked with Xenserver at this point so bear with me. Nothing in this discussion has talked about how you go about restoring the snapshot should the need arrise. Is it as simple as copying the .xva back up to the Xenserver data store?

    Thanks
    Steve

  9. Roberto Cespa
    August 21st, 2009 at 10:44 | #9
    Reply | Quote

    Hi, Steve
    Recovering a VM from an .xva file is a simple operation that you can start from XenCenter console; the whole operation is wizard driven and it’s point’n'click.
    I have not access to a XenCenter in this very moment (I’m out of office) but, as far as I can remember, is something like “File -> Import VM” from the menu to start the wizard.
    Then you have to browse your computer (or a network share) to find the .xva file, select the home server (in case of server pools), select the Storage Repository and click the finish button.
    I tested recovery several times and it needed something like 30 to 45 minute each time to bring back the VM to life.
    If you don’t want to set up Windows networking again you should manually set the network adater MAC address to the same value of the initial machine.

    If you do the recovery by CLI, you should have an option to preserve the MAC (as “Anonymous” says
    here: )

    Add “preserve=true” when importing to keep the MAC, from the administrator’s guide:

    Import a VM from a previously-exported file. If preserve is set to true, the MAC address of the original VM will be preserved.

    Ciao,
    Roberto

  10. Roberto Capua
    September 24th, 2009 at 01:47 | #10
    Reply | Quote

    Su questo blog c’è qualche linea guida per fare gli script di backup da windows e la cosa carina è che funziona anche quando cambia il master del pool con winscp…
    http://virtualgreen.wordpress.com/2009/09/22/uno-xen-virtual-consolidated-backup-server-artigianale/

  11. Gianluca C.
    March 18th, 2010 at 12:32 | #11
    Reply | Quote

    Ho provato lo script e non mi parte, per farlo funzionare ho dovuto usare perl5.8.8 ( sia sostituendolo sulla prima riga dello script, che per farlo runnare perl5.8.8 backup.pl), usando il comadno perl backup.pl non mi restituiva nessun output..non so perche’ essendo abbastanza ignorante in ambito linux :)

  12. Jan Schulenburg
    June 3rd, 2010 at 09:46 | #12
    Reply | Quote

    Hi Roberto,

    I ran your script (slightly adjusted) on XenServer 5.5.0 Update 2 and it ran fine! Many thankz! I updated to XenServer 5.6.0 (grr, lot’s of issues) and now i try to get the backup script running again. It seems i’ve got a few issues:
    - @ snapshot, error: ^CFatal error: exception Unix.Unix_error(13, “tcsetattr”, “”)
    - i added coalesce-leaf, this isn’t supported anymore.

    I could sent you my backup script if needed.

  13. admin
    June 3rd, 2010 at 10:05 | #13
    Reply | Quote

    Hi Jan,
    I’m happy you found the script useful.
    Unfortunaley, I moved to from Xen to KVM mainly for two reasons:

    • KVM has KSM (Kernel Same Page) feature that permits a very high overlapping: if you have several cloned virtual machines probably the RAM contents mostly the same data and KSM will do data deduplication and you’ll be able to run more VMs with same amount of RAM
    • Citrix Xen Server 5.5.0 had the ugly bug of snapshot coalescing on my storage and I’ve found my NFS repository filled with snapshot

    Now I’m focusing my attention on KVM (mainly using Proxmox VE distro: you can really be up and running in 10 minutes) and I hope to find some spare time to post something about VM migration from Xen to KVM.
    I’m also starting to use MongoDB, a document database, and I hope to write something about MongoDB + PHP ( + Lithium PHP framework).

    I’ll be glad to publish your script in this post once debugged!
    Ciao,
    Roberto

  14. felice
    June 29th, 2010 at 08:48 | #14
    Reply | Quote

    Io ho un problema, spero si possa risolvere.
    Non posso fare snapshot perche il disco locale non ha spazio.
    Non posso usare iscsi e simili.
    E’ possibile eseguire le snaoshot su un disco usb collegato a xen o a una macchina virtuale su cui c’e’ xencenter?
    Utilizzo xen 5.5.0.

    Grazie.

  15. Robert Wunsch
    July 4th, 2010 at 19:25 | #15
    Reply | Quote

    Hello Roberto, Hello Jan,

    I am using Xen 5.6.0 and just starting off virtualising.
    i found your script, Roberto, and want to try using it in my desaster recovery.
    Jan, could you let me know if you ot that script running on 5.6?
    It would be very kind of you to let me know your insights.
    Maybe you could send it to Roberto, so he would be able to publish it?

    Thanks in advance!
    Robert

  1. September 2nd, 2009 at 17:31 | #1
    Francesco Dipietromaria WebLog