PyBlosxom |
/Coding/bash:
Add a Timestamp to Your Logging
Suppose you have various bits of code executing in cron, sending their output to user-defined log files, ie. some cron entries as follows,
*/2 * * * * /path/to/a/script.sh >> /some/logging/directory/cron.output 2>&1and you want a generalized way of tagging each log entry with a time stamp. The solution is fairly simple, but the bits are sufficiently hard to track down and not all available in one place, so I thought I would document it.... Create this script, lets call it timestamp.sh:
and then use a cron entry as follows:#!/bin/bash # this script prepends a time stamp to the first line of piped input. stamp=$(date "+%F %R:%S") echo -n $stamp echo -n " " while read line; do # print the first line of piped input echo ${line[0]} done
*/2 * * * * /path/to/a/script.sh 2>&1 | /path/to/timestamp.sh >> /logging/directory/cron.output 2>&1Ie. The output of script.sh is piped to timestamp.sh, where the first line is prepended with the timestamp, and then the time-stamped line is added to cron.output. Thanks to [1].
[1] http://www.linuxquestions.org/questions/linux-software-2/bash-scripting-pipe-input-to-script-vs-$1-570945/
posted at: 12:19 | path: /Coding/bash | permanent link to this entry
/Hosting/Amazon/EC2:
How to Backup an Amazon EBS-Boot Server
This is a script I wrote to backup several Amazon AWS EBS-boot servers, using the Python boto library[1]. Basically it takes a snapshot of the server's root EBS volume (defined in volumeIDs list) and then builds a new bootable Amazon Machine Image (AMI) for the server based upon that snapshot.
#!/usr/bin/env python ########################################################## # Installation: # This script requires boto ('pip install boto') to communicate with Amazon AWS API, # and at least version 1.9a to handle an EBS boot instance # Script inspired by http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html # # As of early June, we need svn version of boto: # pip install -e svn+http://boto.googlecode.com/svn/trunk/@1428#egg=boto # until next release, which should be soon per this forum post: # http://groups.google.com/group/boto-users/browse_thread/thread/21dc3482ed7e49da ########################################################## ########################################################## # This script is for the backup of Amazon AWS EBS Boot servers, and will perform one backup # of one server, per script invocation. Note that there is nothing machine environment-specific # in this script at the moment, so it can be run on any machine with the correct environment, # to backup any other machine. # Usage: # * adjust the "Constants" section to reflect you current account & server environment. # * Choose which server you would like to backup in the "which server" section. # * Install the ElasticFox plugin in Firefox to observer the results of the backup process. # * Run the script to perform a backup of the chosen server. ########################################################## ########################################################## # Constants ########################################################## instances = {"master":"i-xxxxxxxx", "staging":"i-xxxxxxxx", "production":"i-xxxxxxxx"} volumeIDs = {"master":"vol-xxxxxxxx", "staging":"vol-xxxxxxxx", "production":"vol-xxxxxxxx"} ownerID = 'xxxxxxxxxxxx' # I got this from ElasticFox, should be AWS account specific waitSnapshot = 10 # wait increment (in seconds) while waiting for snapshot to complete ########################################################## # Which server is this script backing up? ########################################################## # thisServer = "master" thisServer = "staging" # thisServer = "production" thisInstance = instances[thisServer] thisVolume = volumeIDs[thisServer] ########################################################## print '' print '##########################################################' print 'Backup of ' + thisServer + ' server:' print 'Logging into Amazon AWS....' from boto.ec2.connection import EC2Connection conn = EC2Connection('xxxxxxxxxxxxxxxxxxxx', 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx') ########################################################## from datetime import datetime timeStamp = str(datetime.today()) print '' print timeStamp print 'Snapshotting the ' + thisServer + ' servers boot volume....' comment = thisServer + ' ' + timeStamp name = comment.replace(":",".") # AWS does not like ":" in AMI names name = name.replace(" ","_") # make life easier when deleting stale images snapshot = conn.create_snapshot(thisVolume, comment) newSnapshot = snapshot.id print 'Created new volume snapshot:', newSnapshot ########################################################## import time waitSnapshotTotal = waitSnapshot snapshot = conn.get_all_snapshots(str(newSnapshot)) print '' while snapshot[0].status != 'completed': print 'Snapshot status is ' + snapshot[0].status + ', ' \ 'wait ', waitSnapshotTotal, ' secs for the snapshot to complete before building the AMI.' time.sleep(waitSnapshot) waitSnapshotTotal = waitSnapshotTotal + waitSnapshot snapshot = conn.get_all_snapshots(str(newSnapshot)) ########################################################## print '' print 'Building a bootable AMI based up this snapshot....' # setup for building an EBS boot snapshot" from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping ebs = EBSBlockDeviceType() ebs.snapshot_id = newSnapshot block_map = BlockDeviceMapping() block_map['/dev/sda1'] = ebs # use the same kernel & ramdisk from running server in the new AMI: attribute = conn.get_instance_attribute(thisInstance, 'kernel') kernelID = attribute['kernel'] print 'kernel ID = ', kernelID attribute = conn.get_instance_attribute(thisInstance, 'ramdisk') ramdiskID = attribute['ramdisk'] print 'ramdisk ID = ', ramdiskID # create the new AMI: result = conn.register_image(name=name, description=timeStamp, architecture='i386', kernel_id=kernelID, ramdisk_id=ramdiskID, root_device_name='/dev/sda1', block_device_map=block_map) print 'The new AMI ID = ', result
[1] http://code.google.com/p/boto/
posted at: 04:12 | path: /Hosting/Amazon/EC2 | permanent link to this entry