Expat-IT Tech Bits

Home

Contact

Links

Search this site:

Categories:

/ (287)
  Admin/ (122)
    Apache/ (10)
      HTTPS-SSL/ (4)
      PHP/ (3)
      performance/ (2)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (6)
    Monitoring/ (2)
      munin/ (2)
    SSH/ (6)
    SSL/ (1)
    Samba/ (1)
    VPN-options/ (6)
      OpenVPN/ (1)
      SSH-Proxy/ (3)
      Tinc/ (1)
      sshuttle/ (1)
    backups/ (17)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (2)
    commandLine/ (24)
      files/ (8)
      misc/ (10)
      network/ (6)
    crontab/ (1)
    databases/ (15)
      MSSQL/ (2)
      MySQL/ (8)
      Oracle/ (3)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (11)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (7)
      puppet/ (1)
    iptables/ (3)
    tripwire/ (1)
    virtualization/ (9)
      VMware/ (1)
      virtualBox/ (8)
  Coding/ (14)
    bash/ (1)
    gdb/ (1)
    git/ (3)
    php/ (5)
    python/ (4)
      Django/ (2)
  Education/ (1)
  Hosting/ (27)
    Amazon/ (18)
      EBS/ (3)
      EC2/ (10)
      S3/ (1)
      commandline/ (4)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (30)
    Android/ (1)
    Awesome/ (3)
    CPUfreq/ (1)
    China/ (2)
    Debian/ (8)
      APT/ (3)
      WPA/ (1)
    audio/ (1)
    encryption/ (3)
    fonts/ (1)
    misc/ (6)
    remoteDesktop/ (1)
    router-bridge/ (3)
  SW/ (45)
    Micro$soft/ (1)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (28)
      Drupal/ (9)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (7)
      WebERP/ (2)
      WordPress/ (1)
      eGroupware/ (1)
    chat/ (1)
    email/ (1)
    fileSharing/ (2)
      btsync/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (15)
    IMchat/ (2)
    circumvention/ (2)
    cryptoCurrency/ (1)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (2)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (14)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)

Archives:

  • 2016/07
  • 2016/05
  • 2016/02
  • 2016/01
  • 2015/12
  • 2015/11
  • 2015/06
  • 2015/01
  • 2014/12
  • 2014/11
  • 2014/10
  • 2014/09
  • 2014/07
  • 2014/04
  • 2014/02
  • 2014/01
  • 2013/12
  • 2013/10
  • 2013/08
  • 2013/07
  • 2013/06
  • 2013/05
  • 2013/04
  • 2013/02
  • 2013/01
  • 2012/12
  • 2012/10
  • 2012/09
  • 2012/08
  • 2012/07
  • 2012/06
  • 2012/05
  • 2012/04
  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09
  • Subscribe XML RSS Feed

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
    PyBlosxom

    This site has no ads. To help with hosting, crypto donations are accepted:
    Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
    Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs

    Fri, 23 Mar 2012


    /Admin/backups/dirvish: Simple, Elegant, Small-Enterprise Backups with Dirvish

    I do not like Bacula, Backuppc has some issues, rdiff-backup is too crude, where to turn for rock-solid backups? Rsync-based Dirvish[1] is making me very happy at the moment:

    As a layer on top of rdiff-backup, Dirvish's basic operation is the same: the first backup rsyncs all non-excluded files and directories. (Obviously, this may be quite a chunk of time and bandwidth if we are talking about a remote server with a large file system.) Thereafter, each succeeding backup is an rsync of just the changes since the last backup. Adjacent backups appear as directories with the entire contents of the backup within, BUT files that are the same are hard linked, ie. only the delta is taking up actual storage space. This means that pruning old backups simply consists of deleting the directories they are contained in, with hard-linking automatically taking care of the house-keeping for remaining adjacent backups. And it means that maintaining a complete backup of a remote server is reasonable, since after the first backup only incremental changes go over the network on subsequent backups.

    What Dirvish adds on top of rdiff-backup is:

    On Debian-based systems the framework is already setup with a cronjob that will, by default, fire at 2200 and prune/refresh all backups. Configuration involves creating an /etc/dirvish/master.conf that orchestrates the whole process, and creating a set of directories to house your Dirvish "banks" (groupings of backups) and "vaults" (each "vault" is a directory tree on a specific machine). And then inside each "vault" adding a dirvish/default.conf file that gives specific direction for that particular backup.

    For instance, /etc/dirvish/master.conf:

    bank:   
            /home/backups/dirvish-local
            /home/backups/dirvish-officeServer
            /home/backups/dirvish-ibmProductionServer
    exclude:
            lost+found/
            core
            *~
            .nfs*
            tmp
            /proc
            /sys
            /dev
            /var/cache/apt/archives/*.deb
    Runall: 
            local-root      22:00
            ibmFull         22:00
            officeEtc       22:00
            officeWWW       22:00
            officeMySQL     22:00
    
    expire-default: +15 days
    expire-rule:
    #       MIN HR    DOM MON       DOW  STRFTIME_FMT
            *   *     *   *         1    +3 months
    #       *   *     1-7 *         1    +1 year
            *   *     1-7 1,4,7,10  1    +1 year
            *   10-20 *   *         *    +4 days
    #       *   *     *   *         2-7  +30 days
    

    In the above, for instance, officeEtc / officeWWW / officeMySQL are all "vaults" for which I have created subdirectories under /home/backups/dirvish-officeServer. This file defines what is backed up, and a set of defaults for all "vaults". For instance, the above expire rules will keep 15 days of daily backups, three months of weekly backups, and one year of quarterly backups for all vaults, unless there are local vault rules that over-ride this behavior.

    An example vault config file, /home/backups/dirvish-ibmProductionServer/ibmFull/dirvish/default.conf:

    client: prodServer
    tree: /
    xdev: 0
    index: gzip
    log: gzip
    image-default: %Y%m%d
    speed-limit: 100
    exclude:
    

    "prodServer" must appear in /etc/hosts or resolve via DNS. "tree: /" tells us we are backing up the whole server. "speed-limit: 100" uses the same syntax as rsync, and limits bandwidth to 100 kB/s.

    Normally after setting up all the configuration files, one tests the configuration by setting the "tree" parameter to a relatively small subdirectory in each vault in turn, and running:

    dirvish --vault <vault-name> --init

    for each vault. This causes that tree to be backed up immediately, and you can see and correct any errors. After all is working, reset "tree" to the desired value, and check a couple days later to see if everything is working as expected.

    [1] http://www.dirvish.org/
    [2] http://www.dirvish.org/docs.html
    [3] http://wiki.edseek.com/howto:dirvish

    posted at: 03:31 | path: /Admin/backups/dirvish | permanent link to this entry