Expat-IT Tech Bits

Home

Contact

Links

Search this site:

Categories:

/ (287)
  Admin/ (122)
    Apache/ (10)
      HTTPS-SSL/ (4)
      PHP/ (3)
      performance/ (2)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (6)
    Monitoring/ (2)
      munin/ (2)
    SSH/ (6)
    SSL/ (1)
    Samba/ (1)
    VPN-options/ (6)
      OpenVPN/ (1)
      SSH-Proxy/ (3)
      Tinc/ (1)
      sshuttle/ (1)
    backups/ (17)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (2)
    commandLine/ (24)
      files/ (8)
      misc/ (10)
      network/ (6)
    crontab/ (1)
    databases/ (15)
      MSSQL/ (2)
      MySQL/ (8)
      Oracle/ (3)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (11)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (7)
      puppet/ (1)
    iptables/ (3)
    tripwire/ (1)
    virtualization/ (9)
      VMware/ (1)
      virtualBox/ (8)
  Coding/ (14)
    bash/ (1)
    gdb/ (1)
    git/ (3)
    php/ (5)
    python/ (4)
      Django/ (2)
  Education/ (1)
  Hosting/ (27)
    Amazon/ (18)
      EBS/ (3)
      EC2/ (10)
      S3/ (1)
      commandline/ (4)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (30)
    Android/ (1)
    Awesome/ (3)
    CPUfreq/ (1)
    China/ (2)
    Debian/ (8)
      APT/ (3)
      WPA/ (1)
    audio/ (1)
    encryption/ (3)
    fonts/ (1)
    misc/ (6)
    remoteDesktop/ (1)
    router-bridge/ (3)
  SW/ (45)
    Micro$soft/ (1)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (28)
      Drupal/ (9)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (7)
      WebERP/ (2)
      WordPress/ (1)
      eGroupware/ (1)
    chat/ (1)
    email/ (1)
    fileSharing/ (2)
      btsync/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (15)
    IMchat/ (2)
    circumvention/ (2)
    cryptoCurrency/ (1)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (2)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (14)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)

Archives:

  • 2016/07
  • 2016/05
  • 2016/02
  • 2016/01
  • 2015/12
  • 2015/11
  • 2015/06
  • 2015/01
  • 2014/12
  • 2014/11
  • 2014/10
  • 2014/09
  • 2014/07
  • 2014/04
  • 2014/02
  • 2014/01
  • 2013/12
  • 2013/10
  • 2013/08
  • 2013/07
  • 2013/06
  • 2013/05
  • 2013/04
  • 2013/02
  • 2013/01
  • 2012/12
  • 2012/10
  • 2012/09
  • 2012/08
  • 2012/07
  • 2012/06
  • 2012/05
  • 2012/04
  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09
  • Subscribe XML RSS Feed

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
    PyBlosxom

    This site has no ads. To help with hosting, crypto donations are accepted:
    Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
    Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs

    Fri, 20 Aug 2010


    /Hosting/Amazon/EBS: Use python-boto to Snapshot Amazon EBS Volume

    The Amazon EC2 toolkit[1] works great, but it is a Java app which makes it an incredibly bloated way to gain access to a couple of command-line tools on a server. python-boto[2] takes up all of about 1M, and its minimal Python dependencies were already installed on my Amazon instance. The necessary python script is very simple:

    #!/usr/bin/env python thisVolume = 'vol-xxxxxxxx' waitSnapshot = 10 # wait increment (in seconds) while waiting for snapshot to complete print 'Logging into Amazon AWS....' from boto.ec2.connection import EC2Connection conn = EC2Connection('<aws_access_key_id>', '<aws_secret_access_key>') print '' print 'Stopping MySQL....' import os os.system("/etc/init.d/mysql stop") print '' print 'Beginning backup of ' + thisVolume snapshot = conn.create_snapshot(thisVolume) newSnapshot = snapshot.id print 'Created new volume snapshot:', newSnapshot import time waitSnapshotTotal = waitSnapshot snapshot = conn.get_all_snapshots(str(newSnapshot)) print '' while snapshot[0].status != 'completed': print 'Snapshot status is ' + snapshot[0].status + ', ' \ 'wait ', waitSnapshotTotal, ' secs for the snapshot to complete before re-starting MySQL.' time.sleep(waitSnapshot) waitSnapshotTotal = waitSnapshotTotal + waitSnapshot snapshot = conn.get_all_snapshots(str(newSnapshot)) print snapshot print '' print 'Restarting MySQL....' os.system("/etc/init.d/mysql start")

    The only gotcha was that the stale version of python-boto in Debian stable did not seem to have a "create_snapshot" function, so I just grabbed a newer version from Debian testing.

    And of course, just call this script from cron daily to make backups automatic.

    [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88
    [2] http://code.google.com/p/boto/

    posted at: 09:27 | path: /Hosting/Amazon/EBS | permanent link to this entry

    Mon, 30 Mar 2009


    /Hosting/Amazon/EBS: Amazon Backups: Snapshotting EBS to S3

    Amazon "Compute Cloud" services are setup such that storage for a running server is kept on an EBS ("Elastic Block Storage") volume mounted on the server. Long-term, offsite (geographic redundancy across multiple data centers) storage is kept in S3. There is a function to snapshot an EBS volume into S3.

    List the volumes associated with your account:

    ec2-describe-volumes

    This will also show what volumes are mounted on which servers. Stop the services running on the server that might be writing to the EBS (databases in particular, if you wish to have a usable backup):

    /etc/init.d/apache2 stop
    /etc/init.d/mysql stop
    Create a snapshot:
    ec2-create-snapshot vol-3159bd58
    Check status of snapshot:
    ec2-describe-snapshots
    SNAPSHOT snap-e309e38a vol-3159bd58 completed
    Restart services:
    /etc/init.d/mysql start
    /etc/init.d/apache2 start

    To restore file(s) from one of the backup snapshots, simply create a new volume from the chosen snapshot. Attach it to instance of your running server. Mount the volume temporarily on the server and grab your files. Then unmount the volume, detach it, and delete it. A process that takes only a couple minutes, especially if you use the extremely convenient Firefox ElasticFox plugin[1][2].

    [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609
    [2] http://s3.amazonaws.com/ec2-downloads/elasticfox.xpi

    posted at: 02:33 | path: /Hosting/Amazon/EBS | permanent link to this entry

    Wed, 18 Feb 2009


    /Hosting/Amazon/EBS: Keep Your Amazon Server's Data on EBS

    Amazon's EC2 servers do not provide persistent storage. If the server is turned off or even perhaps halted, then all data since the server was initialized is lost. (I am pretty sure one can reboot an Amazon server without losing data, but have not tried....)

    Therefore, at the least, all volatile data should be stored on an Amazon "Elastic Block Storage" (EBS) volume. As a further incentive, Amazon says[1] that EBS is faster then the storage associated directly with an Amazon server, and that it is replicated across multiple servers in the data center. The downside, of course, is that you pay for EBS, both by gigabyte per month and by I/O per month.

    Here[2] is an excellent tutorial on how to use EBS with a MySQL database, and here[3] is a nice general introduction.

    I am going to move my Amazon Server's /var/www/ onto EBS. First use ec2-describe-instances to find out which Amazon zone the server is located in[4]: us-east-1c. Now create a 5G EBS volume:

    ec2-create-volume -z us-east-1c -s 5

    From what I can see so far, changing the volume size probably involves snapshotting the volume, and then restoring the snapshot to a bigger volume. This small headache must be traded-off against the EBS cost of 10 cents per gigabyte per month.

    ec2-describe-volumes will tell you when the new volume is available. Now attach the volume to the server instance at /dev/sdz:

    ec2-attach-volume -d /dev/sdz -i i-c1a320a8 vol-3159bd58

    Now login to the server and format the volume:

    mkfs.ext3 /dev/sdz

    (Note that xfs has apparently been a popular format choice in the past because of its explicit freezability for snapshots, but I have seen in more then one place now that Amazon EC2 has problems with xfs.)

    Setup /etc/fstab by adding the following entry:

    /dev/sdz /vol ext3 noatime 0 0
    And mount the volume:
    cd /
    mkdir vol
    mount vol
    Now move the contents of /var/www to the volume:
    /etc/init.d/apache2 stop
    cd /var
    mv www /vol/
    ln -s /vol/www/ www
    /etc/init.d/apache2 start

    [1] http://aws.amazon.com/ebs/
    [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663
    [3] http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/
    [4] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1347

    posted at: 02:35 | path: /Hosting/Amazon/EBS | permanent link to this entry