This site has no ads. To help with hosting, crypto donations are accepted:
Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs
Fri, 25 Jun 2010
/Coding/bash:
Add a Timestamp to Your Logging
Suppose you have various bits of code executing in cron, sending their output to user-defined log files, ie. some cron entries
as follows,
*/2 * * * * /path/to/a/script.sh >> /some/logging/directory/cron.output 2>&1
and you want a generalized way of tagging each log entry with a time stamp. The solution is fairly simple, but the bits are
sufficiently hard to track down and not all available in one place, so I thought I would document it.... Create this script, lets
call it timestamp.sh:
#!/bin/bash
# this script prepends a time stamp to the first line of piped input.
stamp=$(date "+%F %R:%S")
echo -n $stamp
echo -n " "
while read line; do
# print the first line of piped input
echo ${line[0]}
done
and then use a cron entry as follows:
*/2 * * * * /path/to/a/script.sh 2>&1 | /path/to/timestamp.sh >> /logging/directory/cron.output 2>&1
Ie. The output of script.sh is piped to timestamp.sh, where the first line is prepended with the timestamp, and then the
time-stamped line is added to cron.output. Thanks to [1].
[1] http://www.linuxquestions.org/questions/linux-software-2/bash-scripting-pipe-input-to-script-vs-$1-570945/
posted at: 12:19 | path: /Coding/bash | permanent link to this entry
/Hosting/Amazon/EC2:
How to Backup an Amazon EBS-Boot Server
This is a script I wrote to backup several Amazon AWS EBS-boot servers, using the Python boto library[1]. Basically it takes a
snapshot of the server's root EBS volume (defined in volumeIDs list) and then builds a new bootable Amazon Machine Image (AMI) for
the server based upon that snapshot.
#!/usr/bin/env python
##########################################################
# Installation:
# This script requires boto ('pip install boto') to communicate with Amazon AWS API,
# and at least version 1.9a to handle an EBS boot instance
# Script inspired by http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html
#
# As of early June, we need svn version of boto:
# pip install -e svn+http://boto.googlecode.com/svn/trunk/@1428#egg=boto
# until next release, which should be soon per this forum post:
# http://groups.google.com/group/boto-users/browse_thread/thread/21dc3482ed7e49da
##########################################################
##########################################################
# This script is for the backup of Amazon AWS EBS Boot servers, and will perform one backup
# of one server, per script invocation. Note that there is nothing machine environment-specific
# in this script at the moment, so it can be run on any machine with the correct environment,
# to backup any other machine.
# Usage:
# * adjust the "Constants" section to reflect you current account & server environment.
# * Choose which server you would like to backup in the "which server" section.
# * Install the ElasticFox plugin in Firefox to observer the results of the backup process.
# * Run the script to perform a backup of the chosen server.
##########################################################
##########################################################
# Constants
##########################################################
instances = {"master":"i-xxxxxxxx", "staging":"i-xxxxxxxx", "production":"i-xxxxxxxx"}
volumeIDs = {"master":"vol-xxxxxxxx", "staging":"vol-xxxxxxxx", "production":"vol-xxxxxxxx"}
ownerID = 'xxxxxxxxxxxx' # I got this from ElasticFox, should be AWS account specific
waitSnapshot = 10 # wait increment (in seconds) while waiting for snapshot to complete
##########################################################
# Which server is this script backing up?
##########################################################
# thisServer = "master"
thisServer = "staging"
# thisServer = "production"
thisInstance = instances[thisServer]
thisVolume = volumeIDs[thisServer]
##########################################################
print ''
print '##########################################################'
print 'Backup of ' + thisServer + ' server:'
print 'Logging into Amazon AWS....'
from boto.ec2.connection import EC2Connection
conn = EC2Connection('xxxxxxxxxxxxxxxxxxxx', 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
##########################################################
from datetime import datetime
timeStamp = str(datetime.today())
print ''
print timeStamp
print 'Snapshotting the ' + thisServer + ' servers boot volume....'
comment = thisServer + ' ' + timeStamp
name = comment.replace(":",".") # AWS does not like ":" in AMI names
name = name.replace(" ","_") # make life easier when deleting stale images
snapshot = conn.create_snapshot(thisVolume, comment)
newSnapshot = snapshot.id
print 'Created new volume snapshot:', newSnapshot
##########################################################
import time
waitSnapshotTotal = waitSnapshot
snapshot = conn.get_all_snapshots(str(newSnapshot))
print ''
while snapshot[0].status != 'completed':
print 'Snapshot status is ' + snapshot[0].status + ', ' \
'wait ', waitSnapshotTotal, ' secs for the snapshot to complete before building the AMI.'
time.sleep(waitSnapshot)
waitSnapshotTotal = waitSnapshotTotal + waitSnapshot
snapshot = conn.get_all_snapshots(str(newSnapshot))
##########################################################
print ''
print 'Building a bootable AMI based up this snapshot....'
# setup for building an EBS boot snapshot"
from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
ebs = EBSBlockDeviceType()
ebs.snapshot_id = newSnapshot
block_map = BlockDeviceMapping()
block_map['/dev/sda1'] = ebs
# use the same kernel & ramdisk from running server in the new AMI:
attribute = conn.get_instance_attribute(thisInstance, 'kernel')
kernelID = attribute['kernel']
print 'kernel ID = ', kernelID
attribute = conn.get_instance_attribute(thisInstance, 'ramdisk')
ramdiskID = attribute['ramdisk']
print 'ramdisk ID = ', ramdiskID
# create the new AMI:
result = conn.register_image(name=name,
description=timeStamp,
architecture='i386',
kernel_id=kernelID,
ramdisk_id=ramdiskID,
root_device_name='/dev/sda1',
block_device_map=block_map)
print 'The new AMI ID = ', result
[1] http://code.google.com/p/boto/
posted at: 04:12 | path: /Hosting/Amazon/EC2 | permanent link to this entry
