Expat-IT Tech Bits

Home

Contact

Links

Search this site:

Categories:

/ (287)
  Admin/ (122)
    Apache/ (10)
      HTTPS-SSL/ (4)
      PHP/ (3)
      performance/ (2)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (6)
    Monitoring/ (2)
      munin/ (2)
    SSH/ (6)
    SSL/ (1)
    Samba/ (1)
    VPN-options/ (6)
      OpenVPN/ (1)
      SSH-Proxy/ (3)
      Tinc/ (1)
      sshuttle/ (1)
    backups/ (17)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (2)
    commandLine/ (24)
      files/ (8)
      misc/ (10)
      network/ (6)
    crontab/ (1)
    databases/ (15)
      MSSQL/ (2)
      MySQL/ (8)
      Oracle/ (3)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (11)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (7)
      puppet/ (1)
    iptables/ (3)
    tripwire/ (1)
    virtualization/ (9)
      VMware/ (1)
      virtualBox/ (8)
  Coding/ (14)
    bash/ (1)
    gdb/ (1)
    git/ (3)
    php/ (5)
    python/ (4)
      Django/ (2)
  Education/ (1)
  Hosting/ (27)
    Amazon/ (18)
      EBS/ (3)
      EC2/ (10)
      S3/ (1)
      commandline/ (4)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (30)
    Android/ (1)
    Awesome/ (3)
    CPUfreq/ (1)
    China/ (2)
    Debian/ (8)
      APT/ (3)
      WPA/ (1)
    audio/ (1)
    encryption/ (3)
    fonts/ (1)
    misc/ (6)
    remoteDesktop/ (1)
    router-bridge/ (3)
  SW/ (45)
    Micro$soft/ (1)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (28)
      Drupal/ (9)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (7)
      WebERP/ (2)
      WordPress/ (1)
      eGroupware/ (1)
    chat/ (1)
    email/ (1)
    fileSharing/ (2)
      btsync/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (15)
    IMchat/ (2)
    circumvention/ (2)
    cryptoCurrency/ (1)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (2)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (14)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)

Archives:

  • 2016/07
  • 2016/05
  • 2016/02
  • 2016/01
  • 2015/12
  • 2015/11
  • 2015/06
  • 2015/01
  • 2014/12
  • 2014/11
  • 2014/10
  • 2014/09
  • 2014/07
  • 2014/04
  • 2014/02
  • 2014/01
  • 2013/12
  • 2013/10
  • 2013/08
  • 2013/07
  • 2013/06
  • 2013/05
  • 2013/04
  • 2013/02
  • 2013/01
  • 2012/12
  • 2012/10
  • 2012/09
  • 2012/08
  • 2012/07
  • 2012/06
  • 2012/05
  • 2012/04
  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09
  • Subscribe XML RSS Feed

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
    PyBlosxom

    This site has no ads. To help with hosting, crypto donations are accepted:
    Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
    Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs

    Fri, 25 Jun 2010


    /Coding/bash: Add a Timestamp to Your Logging

    Suppose you have various bits of code executing in cron, sending their output to user-defined log files, ie. some cron entries as follows,

    */2 * * * * /path/to/a/script.sh >> /some/logging/directory/cron.output 2>&1
    and you want a generalized way of tagging each log entry with a time stamp. The solution is fairly simple, but the bits are sufficiently hard to track down and not all available in one place, so I thought I would document it.... Create this script, lets call it timestamp.sh:
    #!/bin/bash
    # this script prepends a time stamp to the first line of piped input.
    
    stamp=$(date "+%F %R:%S")
    echo -n $stamp
    echo -n " "
    
    while read line; do
       # print the first line of piped input
       echo ${line[0]}
    done
    
    and then use a cron entry as follows:
    */2 * * * * /path/to/a/script.sh 2>&1 | /path/to/timestamp.sh >> /logging/directory/cron.output 2>&1
    Ie. The output of script.sh is piped to timestamp.sh, where the first line is prepended with the timestamp, and then the time-stamped line is added to cron.output. Thanks to [1].

    [1] http://www.linuxquestions.org/questions/linux-software-2/bash-scripting-pipe-input-to-script-vs-$1-570945/

    posted at: 12:19 | path: /Coding/bash | permanent link to this entry


    /Hosting/Amazon/EC2: How to Backup an Amazon EBS-Boot Server

    This is a script I wrote to backup several Amazon AWS EBS-boot servers, using the Python boto library[1]. Basically it takes a snapshot of the server's root EBS volume (defined in volumeIDs list) and then builds a new bootable Amazon Machine Image (AMI) for the server based upon that snapshot.

    #!/usr/bin/env python
    
    ##########################################################
    # Installation:
    # This script requires boto ('pip install boto') to communicate with Amazon AWS API,
    # and at least version 1.9a to handle an EBS boot instance
    # Script inspired by http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html
    #
    # As of early June, we need svn version of boto:
    #   pip install -e svn+http://boto.googlecode.com/svn/trunk/@1428#egg=boto
    # until next release, which should be soon per this forum post:
    #   http://groups.google.com/group/boto-users/browse_thread/thread/21dc3482ed7e49da
    ##########################################################
    
    ##########################################################
    # This script is for the backup of Amazon AWS EBS Boot servers, and will perform one backup
    # of one server, per script invocation. Note that there is nothing machine environment-specific
    # in this script at the moment, so it can be run on any machine with the correct environment,
    # to backup any other machine.
    # Usage:
    # * adjust the "Constants" section to reflect you current account & server environment.
    # * Choose which server you would like to backup in the "which server" section.
    # * Install the ElasticFox plugin in Firefox to observer the results of the backup process.
    # * Run the script to perform a backup of the chosen server.
    ##########################################################
    
    ##########################################################
    # Constants
    ##########################################################
    instances = {"master":"i-xxxxxxxx",   "staging":"i-xxxxxxxx",   "production":"i-xxxxxxxx"}
    volumeIDs = {"master":"vol-xxxxxxxx", "staging":"vol-xxxxxxxx", "production":"vol-xxxxxxxx"}
    ownerID = 'xxxxxxxxxxxx' # I got this from ElasticFox, should be AWS account specific
    waitSnapshot = 10 # wait increment (in seconds) while waiting for snapshot to complete
    
    ##########################################################
    # Which server is this script backing up?
    ##########################################################
    # thisServer = "master"
    thisServer = "staging"
    # thisServer = "production"
    
    thisInstance = instances[thisServer]
    thisVolume = volumeIDs[thisServer]
    
    ##########################################################
    print ''
    print '##########################################################'
    print 'Backup of ' + thisServer + ' server:'
    print 'Logging into Amazon AWS....'
    from boto.ec2.connection import EC2Connection
    conn = EC2Connection('xxxxxxxxxxxxxxxxxxxx', 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
    
    ##########################################################
    from datetime import datetime
    timeStamp = str(datetime.today())
    
    print ''
    print timeStamp
    print 'Snapshotting the ' + thisServer + ' servers boot volume....'
    comment = thisServer + ' ' + timeStamp
    name = comment.replace(":",".") # AWS does not like ":" in AMI names
    name = name.replace(" ","_") # make life easier when deleting stale images
    snapshot = conn.create_snapshot(thisVolume, comment)
    newSnapshot = snapshot.id
    print 'Created new volume snapshot:', newSnapshot
    
    ##########################################################
    import time
    waitSnapshotTotal = waitSnapshot
    snapshot = conn.get_all_snapshots(str(newSnapshot))
    print ''
    while snapshot[0].status != 'completed':
        print 'Snapshot status is ' + snapshot[0].status + ', ' \
              'wait ', waitSnapshotTotal, ' secs for the snapshot to complete before building the AMI.'
        time.sleep(waitSnapshot)
        waitSnapshotTotal = waitSnapshotTotal + waitSnapshot
        snapshot = conn.get_all_snapshots(str(newSnapshot))
    
    ##########################################################
    print ''
    print 'Building a bootable AMI based up this snapshot....'
    
    # setup for building an EBS boot snapshot"
    from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
    ebs = EBSBlockDeviceType()
    ebs.snapshot_id = newSnapshot
    block_map = BlockDeviceMapping()
    block_map['/dev/sda1'] = ebs
    
    # use the same kernel & ramdisk from running server in the new AMI:
    attribute = conn.get_instance_attribute(thisInstance, 'kernel')
    kernelID = attribute['kernel']
    print 'kernel ID = ', kernelID
    attribute = conn.get_instance_attribute(thisInstance, 'ramdisk')
    ramdiskID = attribute['ramdisk']
    print 'ramdisk ID = ', ramdiskID
    
    # create the new AMI:
    result = conn.register_image(name=name,
        description=timeStamp,
        architecture='i386',
        kernel_id=kernelID,
        ramdisk_id=ramdiskID,
        root_device_name='/dev/sda1',
        block_device_map=block_map)
    print 'The new AMI ID = ', result
    

    [1] http://code.google.com/p/boto/

    posted at: 04:12 | path: /Hosting/Amazon/EC2 | permanent link to this entry