Expat-IT Tech Bits




Search this site:


/ (289)
  Admin/ (123)
    Apache/ (10)
      HTTPS-SSL/ (4)
      PHP/ (3)
      performance/ (2)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (6)
    Monitoring/ (2)
      munin/ (2)
    SSH/ (6)
    SSL/ (1)
    Samba/ (1)
    VPN-options/ (7)
      OpenVPN/ (1)
      SSH-Proxy/ (3)
      Tinc/ (1)
      sshuttle/ (1)
    backups/ (17)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (2)
    commandLine/ (24)
      files/ (8)
      misc/ (10)
      network/ (6)
    crontab/ (1)
    databases/ (15)
      MSSQL/ (2)
      MySQL/ (8)
      Oracle/ (3)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (11)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (7)
      puppet/ (1)
    iptables/ (3)
    tripwire/ (1)
    virtualization/ (9)
      VMware/ (1)
      virtualBox/ (8)
  Coding/ (14)
    bash/ (1)
    gdb/ (1)
    git/ (3)
    php/ (5)
    python/ (4)
      Django/ (2)
  Education/ (1)
  Hosting/ (27)
    Amazon/ (18)
      EBS/ (3)
      EC2/ (10)
      S3/ (1)
      commandline/ (4)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (31)
    Android/ (1)
    Awesome/ (3)
    CPUfreq/ (1)
    China/ (2)
    Debian/ (8)
      APT/ (3)
      WPA/ (1)
    audio/ (1)
    encryption/ (3)
    fonts/ (1)
    misc/ (6)
    remoteDesktop/ (1)
    router-bridge/ (3)
  SW/ (45)
    Micro$soft/ (1)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (28)
      Drupal/ (9)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (7)
      WebERP/ (2)
      WordPress/ (1)
      eGroupware/ (1)
    chat/ (1)
    email/ (1)
    fileSharing/ (2)
      btsync/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (15)
    IMchat/ (2)
    circumvention/ (2)
    cryptoCurrency/ (1)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (2)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (14)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)


  • 2019/06
  • 2016/07
  • 2016/05
  • 2016/02
  • 2016/01
  • 2015/12
  • 2015/11
  • 2015/06
  • 2015/01
  • 2014/12
  • 2014/11
  • 2014/10
  • 2014/09
  • 2014/07
  • 2014/04
  • 2014/02
  • 2014/01
  • 2013/12
  • 2013/10
  • 2013/08
  • 2013/07
  • 2013/06
  • 2013/05
  • 2013/04
  • 2013/02
  • 2013/01
  • 2012/12
  • 2012/10
  • 2012/09
  • 2012/08
  • 2012/07
  • 2012/06
  • 2012/05
  • 2012/04
  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09
  • Subscribe XML RSS Feed

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

    This site has no ads. To help with hosting, crypto donations are accepted:
    Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
    Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs

    Tue, 25 Oct 2011

    /SW/business/Drupal: Drupal Migrate Module Basics

    The Drupal migrate module[1] is an impressive tool for migrating databases from other web applications (or anywhere, really) into the Drupal database. It even has fairly impressive documentation[2], but not unlike a lot of other big chunks of documentation, it can be hard to extract from the documentation where to start in creating a simple functional example of the migrate module at work. Hopefully this example will help to fill that gap a little.

    First install and enable the migrate family of modules in a sandbox Drupal site:

    drush dl migrate migrate_example migrate_extras migrate_ui
    drush en migrate migrate_example migrate_extras migrate_ui

    If you then point your browser at:


    you will see the migrate module's UI, which shows a couple of example migrations ("beer" and "wine") already coded and setup for you by the migrate_example sub-module. You can play with importing and rolling back data with the UI, or with drush:

    drush help --filter="migrate"

    To create your own migration, you need to create a custom module containing the migrate code. Using a MySQL database from another application, and based upon the beer.inc example from migrate_example, I was able to get a working migrate setup with three fairly small files in a new sub-module called "migrate_test" in sites/all/modules/migrate/migrate_test/ :


    <?php /* * You must implement hook_migrate_api(), setting the API level to 2, for * your migration classes to be recognized by the Migrate module. */ function migrate_test_migrate_api() { $api = array( 'api' => 2, ); return $api; }


    name = "Migrate Test" description = "My migration data." package = "Development" core = 6.x php = 5.2 dependencies[] = taxonomy dependencies[] = imagefield dependencies[] = comment dependencies[] = migrate dependencies[] = content dependencies[] = date dependencies[] = migrate_extras files[] = migrate_test.module files[] = langex.inc ; Information added by drupal.org packaging script on 2011-09-18 version = "6.x-2.2" core = "6.x" project = "migrate" datestamp = "1316388105"


    <?php // This is the connection information for the database containing the source data. Database::addConnectionInfo('for_migration', 'default', array( 'driver' => 'mysql', 'database' => 'langex', 'username' => 'langex', 'password' => 'anypass', 'host' => 'localhost', 'prefix' => '', )); /** * To define a migration process from a set of source data to a particular * kind of Drupal object (for example, a specific node type), you define * a class derived from Migration. You must define a constructor to initialize * your migration object. By default, your class name will be the "machine name" * of the migration, by which you refer to it. Note that the machine name is * case-sensitive. * * In any serious migration project, you will find there are some options * which are common to the individual migrations you're implementing. You can * define an abstract intermediate class derived from Migration, then derive your * individual migrations from that, to share settings, utility functions, etc. */ abstract class LangexMigration extends Migration { public function __construct() { // Always call the parent constructor first for basic setup parent::__construct(); // With migrate_ui enabled, migration pages will indicate people involved in // the particular migration, with their role and contact info. We default the // list in the shared class; it can be overridden for specific migrations. $this->team = array( new MigrateTeamMember('John Doe', 'john.doe@gmail.com', t('contractor')), new MigrateTeamMember('Larry Brewer', 'lbrewer@example.com', t('Implementor')), ); } } /** * There are four essential components to set up in your constructor: * $this->source - An instance of a class derived from MigrateSource, this * will feed data to the migration. * $this->destination - An instance of a class derived from MigrateDestination, * this will receive data that originated from the source and has been mapped * by the Migration class, and create Drupal objects. * $this->map - An instance of a class derived from MigrateMap, this will keep * track of which source items have been imported and what destination objects * they map to. * Mappings - Use $this->addFieldMapping to tell the Migration class what source * fields correspond to what destination fields, and additional information * associated with the mappings. */ class LangexUserMigration extends LangexMigration { public function __construct() { // The basic setup is similar to BeerTermMigraiton parent::__construct(); $this->description = t('Language Exchange Network users'); $this->map = new MigrateSQLMap($this->machineName, array('userid' => array( 'type' => 'int', 'not null' => TRUE, 'description' => 'Account ID.' ) ), MigrateDestinationUser::getKeySchema() ); $query = Database::getConnection('default', 'for_migration') ->select('users', 'u') ->fields('u', array('userid', 'name', 'username', 'password', 'email_address', 'sex', 'signup_date')); // $this->source = new MigrateSourceSQL($query); $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE)); $this->destination = new MigrateDestinationUser(); // One good way to organize your mappings is in three groups - mapped fields, // unmapped source fields, and unmapped destination fields // Mapped fields // The migrate module automatically converts date/time strings to UNIX timestamps. $this->addFieldMapping('created', 'signup_date'); $this->addFieldMapping('pass', 'password'); $this->addFieldMapping('mail', 'email_address'); $this->addFieldMapping('name', 'username'); // Unmapped source fields $this->addFieldMapping(NULL, 'name') ->issueGroup(t('DNM')); $this->addFieldMapping(NULL, 'sex') ->issueGroup(t('DNM')); // Unmapped destination fields // This is a shortcut you can use to mark several destination fields as DNM // at once $this->addUnmigratedDestinations(array('theme', 'signature', 'access', 'login', 'timezone', 'language', 'picture')); } }

    Note that this is a "cross-database migration"[3]. The source data and the destination data are located on the same MySQL server, but in different databases.

    [1] https://drupal.org/project/migrate
    [2] https://drupal.org/node/415260
    [3] https://drupal.org/node/1014558

    posted at: 05:13 | path: /SW/business/Drupal | permanent link to this entry

    Fri, 21 Oct 2011

    /Coding/git: How to Clone a git Repository

    Backup Scenario (Server to Desktop):

    The simplest scenario is if you are developing on a server, for instance, and merely want to grab a backup copy on your laptop. Assuming you have already setup and are using a git repository on your server, on your laptop navigate to the directory where you wish to place your backup copy. Then:

    git clone ssh://user@server.com/path/to/git/repository/name/

    This will create a "name" directory on your laptop, containing all the code and the git repository. If, in future, you wish to copy over changes from server to laptop, cd into directory "name" and initialize "git pull":

    cd /path/to/git/repository/name/
    git pull ssh://server/path/to/git/repository/name/ HEAD

    In the future, to bring the laptop up to date, all that is required is to cd to the "name" directory, and do:

    git pull

    (ie. git remembers the server and the path on the server.)

    Development / Production Scenario (Desktop to Server):

    Now suppose you are doing development work on your laptop, and the public, production copy of the code is located on a server[1]. First, ssh into the server and prepare a "bare" repository:

    mkdir /path/to/git/repository/name/
    cd /path/to/git/repository/name/
    git --bare init

    Then edit the config file to enable reposharing by adding the line "sharedrepository = 1" under the [core] section. Now back to the laptop.

    cd /path/to/git/repository/name/
    git remote add origin ssh://yourname@server/path/to/git/repository/name/
    git push origin master

    The final step should push your latest and greatest software from laptop to server.

    [1] http://www.tuxmaniac.com/blog/2009/12/14/setting-up-a-remote-git-server/

    posted at: 02:13 | path: /Coding/git | permanent link to this entry

    Sun, 02 Oct 2011

    /Hosting/Rackspace: Use Your Own Kernel on Rackspace

    Rackspace recently announced[1] that they have made it possible for Rackspace cloud server users to manage their own kernel, rather than being stuck with the Rackspace-provided kernel. Ie. it is now possible to upgrade your Rackspace cloud server, just like on Amazon AWS!!

    The plan is apparently to phase in this new capability for new servers, but existing servers can be retrofitted as well. However this process did not turn out to be quite as straight-forward as I had hoped, or these documents[2][3] lead me to believe. I will try to recall the necessary steps to get from Lucid to Maverick.

    I had an Ubuntu Lucid machine. Before turning it into a Maverick machine, the first step is to MAKE A BACKUP OF YOUR SERVER. This process is highly error-prone, and the result of any small error is a server that will not boot. In fact, it is advisable to make a new server backup after each of the major steps below.

    Next step is to switch over to pv-grub and get that working:

    mkdir /boot/grub
    cp /etc/init/tty1.conf /etc/init/hvc0.conf
    sed -i 's/tty1/hvc0/' /etc/init/hvc0.conf
    sed -i 's/sda/xvda/' /etc/fstab
    apt-get install linux-virtual

    And create a /boot/grub/menu.lst that, for Lucid, looks something like this:

    title=DISTRO-NAME "Ubuntu 2.6.32-33-server"
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-33-server ro console=hvc0 root=/dev/xvda1
        initrd /boot/initrd.img-2.6.32-33-server

    Now open a ticket with Rackspace and ask them to switch your server over to pv-grub. Actually, it would probably be better to do this via chat, because they will reboot the server when they make the switch, and if something is wrong with your configuration the server will not come back. If and when it does, your still-Lucid server will be running with the Ubuntu repository-provided kernel, rather than the kernel provided by the Rackspace virtualization environment.

    The next step is the more-or-less standard Ubuntu upgrade from Lucid to Maverick. At the end, DO NOT REMOVE THE LUCID KERNEL, because after you reboot at the end of the upgrade you will find that your newly upgraded Maverick server is still running the Lucid kernel. Here is where it hit the fan for me. Up to this point, despite the less then perfect documentation, Rackspace support was very helpful. I got stuck here and they basically cut me loose, telling me they do not debug OS issues.

    ********* UPDATED SECTION **********************

    Do not linger at Maverick. Upgrade directly from Lucid, to Maverick, and then to Natty, WITHOUT REBOOTING. The Lucid kernel will work with the Natty userspace. If you do as I describe in the section marked "DON'T DO THE BELOW", yes you will get a functioning Maverick server, but your Rackspace snapshot backups will not work (will not boot) as the partition label disappears from the partition in the backup. Going directly to Natty is much cleaner, easier, and safer, as only the Maverick kernel is problematic and needs special handling. Once you have upgraded the userland to Natty, BACKUP YOUR SERVER. Then, again, modify /boot/grub/menu.lst:

    title=DISTRO-NAME "Ubuntu 2.6.38-11"
        root (hd0)
        kernel /boot/vmlinuz-2.6.38-11-virtual ro console=hvc0 root=/dev/xvda1
        initrd /boot/initrd.img-2.6.38-11-virtual

    to point to the Natty kernel, and reboot.

    (Note that I tried the Ubuntu grub-legacy-ec2 package to try to achieve automatic updates of /boot/grub/menu.lst, but the result was a non-booting server.)

    ********* DON'T DO THE BELOW *******************

    Next stop for me was to call on Canonical[4] for some help, and they were helpful as well. The obvious thing to do here is just edit /boot/grub/menu.lst to point to the Maverick kernel and initrd instead of Lucid's. This did not work, due to a quirk in the Maverick kernel. To get around it, label your root partition:

    e2label /dev/xvda1 MYROOT

    Now make the following change to /boot/grub/menu.lst:

    title=DISTRO-NAME "Ubuntu 2.6.35-30-virtual"
        root (hd0)
        kernel /boot/vmlinuz-2.6.35-30-virtual ro console=hvc0 root=LABEL=MYROOT
        initrd /boot/initrd.img-2.6.35-30-virtual

    Note the "root=LABEL=MYROOT" part. And to /etc/fstab:

    proc            /proc       proc    defaults    0 0
    LABEL=MYROOT  /     ext3  defaults,errors=remount-ro,noatime    0 1
    UUID=b9f62618-9dff-4bb4-9f58-3c2c5a95625d  none  swap  sw          0 0

    (you can get the UUID value for swap from /dev/disk/by-uuid/)

    And now when you reboot, it should all work: a Maverick server running the Maverick kernel.

    [1] http://www.rackspace.com/cloud/blog/2011/07/13/new-feature-manage-your-own-kernel-in-linux-cloud-servers/
    [2] http://www.rackspace.com/knowledge_center/linux_kernel_management
    [3] http://www.rackspace.com/knowledge_center/index.php/Using_a_Custom_Kernel_with_pv-grub
    [4] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/821466

    posted at: 09:38 | path: /Hosting/Rackspace | permanent link to this entry

    Sat, 01 Oct 2011

    /Admin/databases/MySQL: Cheat Sheet for the MySQL Command Line

    phpmyadmin is a very handy GUI front-end for MySQL administration, but if there is no web server installed on your MySQL server, it is non-trivial to get phpmyadmin talking to it from a remote host. Especially if your remote host is behind a dynamic IP. And of course, the command line is just plain good for you. ;-) Here are some other cheat sheets for reference[1][2][3]. And here is mine:

    Login to your mysql database on hostname as root (-h and -u are optional, -p forces a password prompt):

    mysql -h hostname -u root -p
    Create a database:
    mysql> create database [databasename];
    Delete a database:
    mysql> drop database [databasename];
    List all databases:
    mysql> show databases;
    Switch to a database:
    mysql> use [db name];
    List the tables in the db:
    mysql> show tables;
    See a table's field formats:
    mysql> describe [table name];
    mysql> show columns from [table name];
    Delete a db:
    mysql> drop database [database name];
    Delete a table:
    mysql> drop table [table name];
    Show data in a table:
    mysql> SELECT * FROM [table name];
    mysql> SELECT * FROM [table name] limit 10;
    mysql> SELECT * FROM [table name] limit 100,10;
    mysql> SELECT * FROM [table name] order by [col name] limit 10;
    mysql> SELECT * FROM [table name] order by [col name] desc limit 10;
    Counting the number of rows in a table:
    mysql> SELECT COUNT(*) FROM pet;
    mysql> SELECT owner, COUNT(*) FROM pet GROUP BY owner;
    Show rows where "field name" has the value "whatever":
    mysql> SELECT * FROM [table name] WHERE [field name] = "whatever";
    mysql> SELECT * FROM [table name] WHERE [field1] = "what1" AND [field2] = "what2";
    Pattern Matching (% is wildcard for an arbitrary # of chars, _ is any single character):
    mysql> SELECT * FROM table WHERE rec LIKE "blah%";
    mysql> SELECT * FROM table WHERE rec like "_____";
    To insert records into a table:
    INSERT into backup_run_log (start_time) VALUE (NOW());
    To get the index of the most recently inserted row:
    SELECT MAX(backup_run_log_id) FROM backup_run_log;
    To change the contents of a table:
    mysql> UPDATE table_name SET field1 = 'x',field2 = 'y',field3 = 'z' where fieldname = 'user';

    mysql> UPDATE tablename SET tablefield = replace(tablefield,"findstring","replacestring");

    mysql> UPDATE `mos2_content` SET introtext = replace(introtext,"{p}","") WHERE `title` REGEXP '-0';
    To delete a row, for eg. to delete a user record from MySQL:
    mysql> DELETE FROM mysql.user WHERE user='username' and host='localhost';
    Column operations:
    mysql> alter table [table name] drop column [column name]; mysql> alter table [table name] add column [new column name] varchar (20); mysql> alter table [table name] change [old column name] [new column name] varchar (50); Make a unique column so you get no dupes. mysql> alter table [table name] add unique ([column name]); Make a column bigger. mysql> alter table [table name] modify [column name] VARCHAR(3); Delete unique from table. mysql> alter table [table name] drop index [colmn name];
    Dumping databases / tables to a (backup) file:
    mysqldump -u root -ppassword --opt >/tmp/alldatabases.sql

    mysqldump -u username -ppassword --databases databasename >/tmp/databasename.sql

    mysqldump -c -u username -ppassword databasename tablename > /tmp/databasename.tablename.sql
    Restoring a database from a dump:
    mysql> create database databasename;
    mysql> quit;
    mysql -ppassword databasename < dump_databasename.sql

    [1] http://www.tech-geeks.org/contrib/mdrone/mysql-stuff/mysql-cheatsheet.html
    [2] http://www.pantz.org/software/mysql/mysqlcommands.html
    [3] http://www.nparikh.org/unix/mysql.php

    posted at: 05:50 | path: /Admin/databases/MySQL | permanent link to this entry