Expat-IT Tech Bits




Search this site:


/ (289)
  Admin/ (123)
    Apache/ (10)
      HTTPS-SSL/ (4)
      PHP/ (3)
      performance/ (2)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (6)
    Monitoring/ (2)
      munin/ (2)
    SSH/ (6)
    SSL/ (1)
    Samba/ (1)
    VPN-options/ (7)
      OpenVPN/ (1)
      SSH-Proxy/ (3)
      Tinc/ (1)
      sshuttle/ (1)
    backups/ (17)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (2)
    commandLine/ (24)
      files/ (8)
      misc/ (10)
      network/ (6)
    crontab/ (1)
    databases/ (15)
      MSSQL/ (2)
      MySQL/ (8)
      Oracle/ (3)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (11)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (7)
      puppet/ (1)
    iptables/ (3)
    tripwire/ (1)
    virtualization/ (9)
      VMware/ (1)
      virtualBox/ (8)
  Coding/ (14)
    bash/ (1)
    gdb/ (1)
    git/ (3)
    php/ (5)
    python/ (4)
      Django/ (2)
  Education/ (1)
  Hosting/ (27)
    Amazon/ (18)
      EBS/ (3)
      EC2/ (10)
      S3/ (1)
      commandline/ (4)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (31)
    Android/ (1)
    Awesome/ (3)
    CPUfreq/ (1)
    China/ (2)
    Debian/ (8)
      APT/ (3)
      WPA/ (1)
    audio/ (1)
    encryption/ (3)
    fonts/ (1)
    misc/ (6)
    remoteDesktop/ (1)
    router-bridge/ (3)
  SW/ (45)
    Micro$soft/ (1)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (28)
      Drupal/ (9)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (7)
      WebERP/ (2)
      WordPress/ (1)
      eGroupware/ (1)
    chat/ (1)
    email/ (1)
    fileSharing/ (2)
      btsync/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (15)
    IMchat/ (2)
    circumvention/ (2)
    cryptoCurrency/ (1)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (2)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (14)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)


  • 2019/06
  • 2016/07
  • 2016/05
  • 2016/02
  • 2016/01
  • 2015/12
  • 2015/11
  • 2015/06
  • 2015/01
  • 2014/12
  • 2014/11
  • 2014/10
  • 2014/09
  • 2014/07
  • 2014/04
  • 2014/02
  • 2014/01
  • 2013/12
  • 2013/10
  • 2013/08
  • 2013/07
  • 2013/06
  • 2013/05
  • 2013/04
  • 2013/02
  • 2013/01
  • 2012/12
  • 2012/10
  • 2012/09
  • 2012/08
  • 2012/07
  • 2012/06
  • 2012/05
  • 2012/04
  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09
  • Subscribe XML RSS Feed

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

    This site has no ads. To help with hosting, crypto donations are accepted:
    Bitcoin: 1JErV8ga9UY7wE8Bbf1KYsA5bkdh8n1Bxc
    Zcash: zcLYqtXYFEWHFtEfM6wg5eCV8frxWtZYkT8WyxvevzNC6SBgmqPS3tkg6nBarmzRzWYAurgs4ThkpkD5QgiSwxqoB7xrCxs

    Fri, 02 Jan 2015

    /Coding/git: Miscellaneous Git Administrivia

    Something I often run into is something modifying file permissions on my desktop end, where permissions are irrelevant on the deployment end. Here is how to tell git to ignore permissions changes in the local repo:

    git config core.fileMode false

    posted at: 03:29 | path: /Coding/git | permanent link to this entry

    Mon, 21 Apr 2014

    /Coding/python/Django: Configure Django Mezzanine for Production Using WSGI

    This turned out to be a lot harder than I expected, let's see if I can remember all the steps.....

    The first issue is that Mezzanine does an un-Django thing when creating the app with

    mezzanine-project project_name[1]

    by creating just one subdirectory. Because of the "python" way in which WSGI is going to be configured to find the "project_name" app, we need two subdirecties on your server, ie.


    Fix that. All the files are in the lowest sub-directory. Also install the Apache WSGI module, on Debian that would be libapache2-mod-wsgi. And, because Apache is now going to be serving static files outside of Django, also run this

    python manage.py collectstatic

    inside your app, to collect all the js/jpg/css/etc scattered around the file system in your Django modules into /var/www/project_name/project_name/static/. It was my experience that /var/www/project_name/project_name/settings.py is already nicely setup so that various settings like STATIC_ROOT are auto-discovered. No tweaking should be necessary.

    Finally, most of the magic is inside the Apache VirtualHost definition:

    <VirtualHost *:80> DocumentRoot /var/www/project_name/project_name ServerName yourdomain.com # https://docs.djangoproject.com/en/1.6/howto/deployment/wsgi/modwsgi/ # http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide#Delegation_To_Daemon_Process WSGIScriptAlias / /var/www/project_name/project_name/wsgi.py WSGIDaemonProcess yourdomain.com processes=2 threads=15 display-name=%{GROUP} python-path=/var/www/project_name/:/var/www/project_name/project_name/ WSGIProcessGroup yourdomain.com <Directory /var/www/project_name/project_name> <Files wsgi.py> Order deny,allow # if Apache 2.4+ # Require all granted # if Apache older then 2.4 Allow from all </Files> </Directory> # Static files Alias /static/ /var/www/project_name/project_name/static/ Alias /robots.txt /var/www/project_name/project_name/static/robots.txt <Directory /var/www/project_name/project_name/static/> Order deny,allow Allow from all # For Apache 2.4 --> "Require all granted" </Directory> </VirtualHost>

    Note that I am running this site in WSGIDaemonProcess mode, which enables the server to serve multiple such Django sites.

    Once that (both site and admin backend) is working and correctly served by Apache, set DEBUG=False in local_settings.py. I believe I finally changed NOTHING in settings.py nor wsgi.py, and the only things I changed in local_settings.py were DEBUG and the database settings. The only thing difficult / lacking in getting Mezzanine going for the first time, were clear WSGI instructions.


    posted at: 21:08 | path: /Coding/python/Django | permanent link to this entry

    Tue, 14 May 2013

    /Coding/php: Geolocation with PHP

    You will probably want:

    apt-get install php5-geoip geoip-database

    Then a command-line script is this easy:

    <?php // if (geoip_db_avail(GEOIP_COUNTRY_EDITION)) // print geoip_database_info(GEOIP_COUNTRY_EDITION); $country = geoip_country_code_by_name('any-ip-address'); switch ($country) { case 'US': echo "100"; break; case 'CN': echo "200"; break; case 'HK': echo "300"; break; } ?>

    And you might want to do something like this for a bit of web code:

    <?php $clientIP = getenv("REMOTE_ADDR"); echo 'Client IP = ' . $clientIP . '<br>'; $country = geoip_country_code_by_name($clientIP); echo $country . ' price = '; switch ($country) { case 'DE': echo "500"; break; case 'IL': echo "400"; break; case 'US': echo "100"; break; case 'CN': echo "200"; break; case 'HK': echo "300"; break; } ?>

    If the geoip database provided by the geoip-database package is too old (three years already in Ubuntu Lucid, for example!) then remove it and download the database manually:

    wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

    Unzip it, and then add this line to /etc/php5/conf.d/geoip.ini:

    geoip.custom_directory = /path/to/directory/containing/GeoIP.dat/

    posted at: 09:55 | path: /Coding/php | permanent link to this entry

    Fri, 21 Oct 2011

    /Coding/git: How to Clone a git Repository

    Backup Scenario (Server to Desktop):

    The simplest scenario is if you are developing on a server, for instance, and merely want to grab a backup copy on your laptop. Assuming you have already setup and are using a git repository on your server, on your laptop navigate to the directory where you wish to place your backup copy. Then:

    git clone ssh://user@server.com/path/to/git/repository/name/

    This will create a "name" directory on your laptop, containing all the code and the git repository. If, in future, you wish to copy over changes from server to laptop, cd into directory "name" and initialize "git pull":

    cd /path/to/git/repository/name/
    git pull ssh://server/path/to/git/repository/name/ HEAD

    In the future, to bring the laptop up to date, all that is required is to cd to the "name" directory, and do:

    git pull

    (ie. git remembers the server and the path on the server.)

    Development / Production Scenario (Desktop to Server):

    Now suppose you are doing development work on your laptop, and the public, production copy of the code is located on a server[1]. First, ssh into the server and prepare a "bare" repository:

    mkdir /path/to/git/repository/name/
    cd /path/to/git/repository/name/
    git --bare init

    Then edit the config file to enable reposharing by adding the line "sharedrepository = 1" under the [core] section. Now back to the laptop.

    cd /path/to/git/repository/name/
    git remote add origin ssh://yourname@server/path/to/git/repository/name/
    git push origin master

    The final step should push your latest and greatest software from laptop to server.

    [1] http://www.tuxmaniac.com/blog/2009/12/14/setting-up-a-remote-git-server/

    posted at: 02:13 | path: /Coding/git | permanent link to this entry

    Sun, 19 Sep 2010

    /Coding/python/Django: Django "Hello World" Example

    (Lifted from [1].)

    On my Debian system, first install Django:

    apt-get install python-django
    cd to where you want to put your new Django web application, and create a Django project:
    django-admin.py startproject djangoHello
    which will cause a subdirectory djangoHello to be created, containing manage.py, settings.py, and urls.py, among other things. Now create an application within that project:
    cd djangoHello
    python manage.py startapp helloworld
    which will create a helloworld subdirectory, containing models.py and views.py. Now edit djangoHello/helloworld/views.py to add the following lines:
    from django.http import HttpResponse def helloworld(request): return HttpResponse('Hello, World!')
    Now edit djangoHello/urls.py, and add the line below after the line above (that will already be in the file):
    #(r'^admin/(.*)', admin.site.root),
    (r'^$', 'djangoHello.helloworld.views.helloworld'),
    Make sure you are in the project root directory (djangoHello) and start the built-in Django server:
    python manage.py runserver
    and point your browser at port 8000 (localhost:8000 if browser and Django are on the same machine) and observer your "Hello, World!".

    [1] https://support.eapps.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=178

    posted at: 01:01 | path: /Coding/python/Django | permanent link to this entry

    Fri, 25 Jun 2010

    /Coding/bash: Add a Timestamp to Your Logging

    Suppose you have various bits of code executing in cron, sending their output to user-defined log files, ie. some cron entries as follows,

    */2 * * * * /path/to/a/script.sh >> /some/logging/directory/cron.output 2>&1
    and you want a generalized way of tagging each log entry with a time stamp. The solution is fairly simple, but the bits are sufficiently hard to track down and not all available in one place, so I thought I would document it.... Create this script, lets call it timestamp.sh:
    # this script prepends a time stamp to the first line of piped input.
    stamp=$(date "+%F %R:%S")
    echo -n $stamp
    echo -n " "
    while read line; do
       # print the first line of piped input
       echo ${line[0]}
    and then use a cron entry as follows:
    */2 * * * * /path/to/a/script.sh 2>&1 | /path/to/timestamp.sh >> /logging/directory/cron.output 2>&1
    Ie. The output of script.sh is piped to timestamp.sh, where the first line is prepended with the timestamp, and then the time-stamped line is added to cron.output. Thanks to [1].

    [1] http://www.linuxquestions.org/questions/linux-software-2/bash-scripting-pipe-input-to-script-vs-$1-570945/

    posted at: 12:19 | path: /Coding/bash | permanent link to this entry

    Tue, 04 Aug 2009

    /Coding/php: A Simple SugarCRM API Class

    This should give anyone needing to use the SugarCRM API a quickstart:

    <?php /* * class_sugarconnect.php * * Copyright 2009 Clayton Koenig <ckoeni@gmail.com> * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, * MA 02110-1301, USA. */ ?> <?php class sugarapi{ // connection variables var $client; // soap client var $session_id; // identifier for this login session var $user_id; // of logged in user // connection settings for SOAP var $options = array( "location" => 'https://url.of.your.sugarcrm.installation.com/sugarcrm/soap.php', "uri" => 'http://www.sugarcrm.com/sugarcrm', "trace" => 1 ); //user authentication array (version is SOAP version) private $user_auth = array(); function __construct() { // must use constructor b/c of MD5 function call $this->user_auth = array( "user_name" => 'sugaruserid', "password" => MD5('yourpasswd'), "version" => '.01' ); } // arrays for input into sugarCRM modules var $contact = array(); var $account = array(); function login() { $this->client = new SoapClient(NULL, $this->options); $response = $this->client->login($this->user_auth); // var_dump($response); // just in case you need it $this->session_id = $response->id; printf("<p></p>session id = ". $this->session_id); $this->user_id = $this->client->get_user_id($this->session_id); printf("<p>" . $this->user_auth['user_name'] . ' has a GUID of ' . $this->user_id . "<p>"); } function getModules() { $response = $this->client->get_available_modules($this->session_id); foreach ($response->modules as $i => $value) { printf($response->modules[$i] . "<br>"); } } function getModuleFields($module) { $response = $this->client->get_module_fields($this->session_id, $module); printf("<p>Module " . $response->module_name .' has the following fields:' . "<p>"); printf("<table><tr>" . "<td><u>Name</td><td><u>Type</td><td><u>Label</td>" . "<td><u>Required</td><td><u>Options</td>" . "</tr>"); foreach ($response->module_fields as $i => $value) { printf('<tr><td>' . $response->module_fields[$i]->name . '</td>' . '<td>' . $response->module_fields[$i]->type . '</td>' . '<td>' . $response->module_fields[$i]->label . '</td>' . '<td>' . $response->module_fields[$i]->required . '</td>' . '<td>' . $response->module_fields[$i]->options . '</td></tr>'); } printf('</table>'); } function setContactField($fieldname, $fieldvalue) { $this->contact[] = array("name" => $fieldname,"value" => $fieldvalue); } function storeContact() { // into sugar's Contacts database $response = $this->client->set_entry($this->session_id, 'Contacts', $this->contact); var_dump($response); } } ?>

    And the following is a simple example of how to use the above class to create a simple contact record with a name and an e-mail address, and store it in Sugar's Contacts database.

    <?php include("class_sugarapi.php"); $sugar = new sugarapi(); $sugar->login(); // populate the contact array $sugar->setContactField('last_name', 'Bloke2'); $sugar->setContactField('email1', 'xyz@wxy.net'); // var_dump($sugar->contact); $sugar->storeContact(); // to Contacts database ?>

    To find out all the possible fields in the Contacts database, use this script:

    <?php include("class_sugarapi.php"); $sugar = new sugarapi(); $sugar->login(); // look what modules sugar exposes $sugar->getModules(); // look in more detail at the fields in a module $sugar->getModuleFields('Accounts'); ?>

    Many thanks to this post[1] for getting me on my way.

    [1] http://systemsconsciousness.com/2009/04/10/sugarcrm-soap-examples/

    posted at: 04:14 | path: /Coding/php | permanent link to this entry

    Thu, 07 May 2009

    /Coding/python: Python: How to Put Some of Your Code in a Subdirectory

    Splitting one's code up into logical sub-directories is a fairly normal practice. This seems to be a little more complicated then usual with Python. One uses the "import" command to access functions in other files / directories. Of course Python automatically searches the system path and your execution directory to satisfy an import command. But to pull in a subdirectory one must place an (empty? Mine is....) __init__.py file in any sub-directory containing Python code that one wants to access. Then the directory name is added to the Python name space and is treated the same as files and functions defined within files.

    A concrete example is most definitely in order. Suppose I want to place all my database connect code into a subdirectory called "db":

    db$ ll
    total 16
    -rw------- 1 1489 2009-02-23 15:18 armor-mail-cert.pem
    -rw------- 1 3243 2009-02-23 15:18 armor-mail-key.pem
    -rw------- 1 1009 2009-02-23 14:48 armorMailSSL-cacert.pem
    -rwxr-xr-x 1  436 2009-05-07 15:55 connectArmorMail.py
    -rw-r--r-- 1    0 2009-05-07 16:33 __init__.py

    My main code is executing one directory up. Any code there that wants to access the connect function must contain:

    import db.connectArmorMail
    db = db.connectArmorMail.connect_armor_mail()

    Note that we are importing db.connectArmorMail, *not* db/connectArmorMail, and then the reference in the next line also uses a ".", not a "/". "db" is a directory, "connectArmorMail" is a file name, and "connect_armor_mail" is a function within a file, but all are treated equally in Python name space, separated by periods. db.connectArmorMail contains:

    def connect_armor_mail():
       import MySQLdb
       ssl_settings = {'ca': 'db/armorMailSSL-cacert.pem',
           'cert': 'db/armor-mail-cert.pem',
           'key': 'db/armor-mail-key.pem'
          print "

    ERROR: could not connect to database." return db

    Note that the files referenced in ssl_settings are in the same directory as db.connectArmorMail, but they are referenced relative to the execution directory (db/*).

    posted at: 05:01 | path: /Coding/python | permanent link to this entry

    Wed, 29 Apr 2009

    /Coding/php: Accessing the KnowledgeTree API

    KnowledgeTree[1] is a very popular server-based Open Source document management system. Something that some users (like me, or rather my clients) need to do is allow certain people to add or manipulate documents in KnowledgeTree without having to have a login ID and knowledge of the KnowledgeTree user interface. Enter the API, and a little custom PHP scripting....

    Oddly enough, I found at least three different documents on the wiki[2] that seemed to talk about three different approaches to using the API. Oddly (should I say suspiciously?) because the level of detail was just enough to be interesting, but just short of being useful. Ie. for two of them, I just could not figure it out. I even saw a post on the KnowledgeTree forum asking for more detail / a concrete example (me too! me too!) and the only reply was a curt link to one of the near useless wiki pages that I have already mentioned. And needless to say, my own post was ignored. Whats up? (Some conspiratorial possibilities come to mind....)

    The only API approach that I have been able to get working is the "REST web service framework"[3], which, for better or worse, only works as of the currently bleeding edge KnowledgeTree version 3.6.0 (will NOT work with current stable 3.5.4a). [3] is also sorely lacking in detail, but in combination with a little code surfing in


    I was able to divine what was needed to get it working. Here I will hopefully provide some missing detail for Google to find....

    One can of course play with the KnowledgeTree REST web service through a browser, as the means of communication with the server is via POST parameters attached to the server URL. This is also a good way to see the exact format of the XML response the server gives back.

    To achieve the same result with PHP one must use libcurl through the PHP curl extension[4]. Since [4] is also a little skimpy on detail, [5] is a very useful supplement. To cut to the chase, I created a function as follows:

    <?php function curlPost($site, $fields){ $ch = curl_init(); // initialize curl handle curl_setopt($ch, CURLOPT_URL,$site); // set url to post to curl_setopt($ch, CURLOPT_FAILONERROR, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);// allow redirects curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); // return into a variable curl_setopt($ch, CURLOPT_TIMEOUT, 9); // times out after 10s curl_setopt($ch, CURLOPT_POST, 1); // set POST method curl_setopt($ch, CURLOPT_POSTFIELDS, $fields); $answer = curl_exec($ch); // run the whole process // print_r(curl_getinfo($ch)); // echo "\n\ncURL error number:" .curl_errno($ch); // echo "\n\ncURL error:" . curl_error($ch); curl_close($ch); return $answer; } ?>

    $site is the REST URL of the KnowledgeTree server, and $fields are the POST parameters that are to go along with it. This function simply POSTs these parameters to the URL (exactly the same as entering $site?$fields into your web browser).

    Here is a concrete and currently working example of how to get the contents of the KnowledgeTree root directory:

    <?php $url = 'https://www.server.com/kt-dms-oss-3.6.0/ktwebservice/KTWebService.php'; require_once('funCurlPost.php'); // ************************** // Login // ************************** $postfields = "method=login&password=123456&username=admin"; $response = curlPost($url, $postfields); $xml = new SimpleXMLElement($response); if( $xml->status_code != 0 ){ echo 'Error - authentication failed: ' . $xml->message; } else { $session_id = $xml->results; echo "Login successful, session ID = " . $session_id; } // *********************************** // List contents of root folder (id=1) // *********************************** $postfields = "method=get_folder_contents&session_id=$session_id&folder_id=1"; $response = curlPost($url, $postfields); $xml = new SimpleXMLElement($response); echo "<p>Get root folder contents:<br>"; if( $xml->status_code != 0 ){ echo 'Error - get_folder_contents failed: ' . $xml->message; } else { // print_r($xml); // to see data structure echo "<p>folder ID = " . $xml->results->folder_id . "<br>"; echo "folder name = " . $xml->results->folder_name . "<br>"; echo "folder path = " . $xml->results->full_path . "<p>"; foreach ($xml->results->items->item as $value) { echo "item type = " . $value->item_type . " "; echo "item ID = " . $value->id . " "; echo "item name = " . $value->filename . "<br>"; } } // *********************************** // Logout // *********************************** $postfields = "method=logout&session_id=$session_id"; $response = curlPost($url, $postfields); $xml = new SimpleXMLElement($response); echo "<p>Logging out....<br>"; if( $xml->status_code != 0 ){ echo 'Error - get_folder_contents failed: ' . $xml->message; } else { echo 'successful!'; } ?>

    The key point is that there were three operations in the above script, with three corresponding POST strings:

    OperationPOST string
    List Directory get_folder_contents&session_id=$session_id&folder_id=1

    And something else that is already working -- to add a document to KnowledgeTree, use this POST string:

    $document = "bodybg.jpg"; // located in /var/uploads

    $postfields = "method=add_document&session_id=$session_id&folder_id=1&title=$document&filename=$document&documenttype=Default&tempfilename=/vol/www/vsc/apps/kt-dms-oss-3.6.0/var/uploads/$document";

    [1] http://www.knowledgetree.com/
    [2] http://wiki.knowledgetree.com/
    [3] http://wiki.knowledgetree.com/REST_Web_Service
    [4] http://php.net/manual/en/book.curl.php
    [5] http://devzone.zend.com/article/1081-Using-cURL-and-libcurl-with-PHP

    posted at: 01:08 | path: /Coding/php | permanent link to this entry

    Tue, 21 Apr 2009

    /Coding/php: XML to Object Conversion

    In working with the KnowledgeTree API[2] I found that the response to my http posts to the API came back in the form of XML. I needed to get that XML into PHP-processable form, and a quite a bit of googling mostly turned up home-grown solutions, a lot of them referring to themselves as "xmltoarray" functions . Until I found the PHP-native solution, SimpleXMLElement[1]. Hopefully this post will help to push SimpleXMLElement a little higher in Google's search listings....

    Suppose I have a big junk of XML in string form, such as this response to a KnowledgeTree API directory listing:

    <response> <status_code>0</status_code> <message/> − <results> <folder_id>1</folder_id> <folder_name>Root Folder</folder_name> <full_path>/</full_path> − <items> − <item> <id>2</id> <item_type>F</item_type> <custom_document_no>n/a</custom_document_no> <oem_document_no>n/a</oem_document_no> <title>DroppedDocuments</title> <document_type>n/a</document_type> <filename>DroppedDocuments</filename> <filesize>n/a</filesize> <created_by>Administrator</created_by> <created_date>n/a</created_date> <checked_out_by>n/a</checked_out_by> <checked_out_date>n/a</checked_out_date> <modified_by>n/a</modified_by> <modified_date>n/a</modified_date> <owned_by>n/a</owned_by> <version>n/a</version> <is_immutable>n/a</is_immutable> <permissions>RWA</permissions> <workflow>n/a</workflow> <workflow_state>n/a</workflow_state> <mime_type>folder</mime_type> <mime_icon_path>folder</mime_icon_path> <mime_display>Folder</mime_display> <storage_path>n/a</storage_path> <items/> </item> − <item> <id>11</id> <item_type>F</item_type> <custom_document_no>n/a</custom_document_no> <oem_document_no>n/a</oem_document_no> <title>Public</title> <document_type>n/a</document_type> <filename>Public</filename> <filesize>n/a</filesize> <created_by>Administrator</created_by> <created_date>n/a</created_date> <checked_out_by>n/a</checked_out_by> <checked_out_date>n/a</checked_out_date> <modified_by>n/a</modified_by> <modified_date>n/a</modified_date> <owned_by>n/a</owned_by> <version>n/a</version> <is_immutable>n/a</is_immutable> <permissions>RWA</permissions> <workflow>n/a</workflow> <workflow_state>n/a</workflow_state> <mime_type>folder</mime_type> <mime_icon_path>folder</mime_icon_path> <mime_display>Folder</mime_display> <storage_path>n/a</storage_path> <items/> </item> − </items> </results> </response>

    in a variable called $response. Converting to a structured object is simply:

    $xml = new SimpleXMLElement($response);

    And then the object might be processed as follows:

    if( $xml->status_code != 0 ){ echo 'Error - operation failed: ' . $xml->message; } else { // print_r($xml); // to see data structure echo "<p>folder ID = " . $xml->results->folder_id . "<br>"; echo "folder name = " . $xml->results->folder_name . "<br>"; echo "folder path = " . $xml->results->full_path . "<p>"; foreach ($xml->results->items->item as $value) { echo "item type = " . $value->item_type . " "; echo "item ID = " . $value->id . " "; echo "item name = " . $value->filename . "<br>"; } }

    [1] http://php.net/manual/en/book.simplexml.php
    [2] http://wiki.knowledgetree.com/REST_Web_Service

    posted at: 06:27 | path: /Coding/php | permanent link to this entry

    Fri, 27 Mar 2009

    /Coding/git: Basic Tutorial: Use Git to Manage Your Source Code

    Git[1] is a program used for distributed Source Code Management (SCM) whose popularity is increasing rapidly. Git seems to be becoming the de facto standard among Debian[2] people, for instance. The first link is to kernel.org, the home of the Linux kernel, because Git was originally created[3] by Linus Torvalds for the Linux kernel development.

    Install git command line tools (Debian and derivatives):

    apt-get install git-core

    (Note that there is a legacy "git" package which has nothing to do with "git" the SCM software....)

    You only need to know a few simple command lines[4][5] to get an astonishing amount of function out of git, and therefore git can be usefully applied to even simple scenarios like bug fixing someone else's software and submitting patches upstream. This last is the example I will work through and demonstrate here, while trying to be more verbose then [4] & [5].

    First, download and unpack your source code. Then cd into the base directory of the source code, and initialize git by running the following:

    git init

    This will create a .git subdirectory which will contain git's repository (starting point plus history of changes you make to the code). Now if you do

    git status

    you will see a long list of "Untracked files", ie. the repository is currently empty. Lets add everything you just unpacked to the repository:

    git add .
    git commit

    Note the "." after "git add". Its the "." that specifies "everything" under the current directory. "git commit" will prompt you for a commit message, just type "Initial commit" and exit the editor. If you check the status again with "git status" you should see zero untracked changes. Now type

    git log

    and you will see the record of your initial commit. Now build your unmodified source code (Debian[6]?) just to make sure there is nothing wrong with it.

    After the build successfully completes, "git status" will show a long list of changes even though you made no change to the source. Time to introduce .gitignore. Create a (hidden) file called ".gitignore" in your source directory root, and place the following on the first line: "*.o". If you run "git status" again you will discover that all *.o (object files) have vanished from the listing. The object of using .gitignore is to get rid / not track all the build noise, and *.o files definitely qualify. I am going to submit patches upstream, so I am pretty sure the contents of the debian directory are not something I want to track, and have also added a "debian*" line to .gitignore.

    Now "git add ." and "git commit" again to get a clean git slate before beginning to make source changes.

    Make your first source changes ("git status" should list the changed files). Build, install, and test the software. Let's say this was a partial change that solves part of the problem, but not the whole problem. This probably qualifies for another commit. "git add" the changed files, then "git commit". You can make several change / test / commit iterations, so that "git log" provides a detailed account of what you did. If they do not conflict with other commits, any individual commit can also be backed out.

    Should you ever mess up and want to restore your last commit:

    git-reset --hard HEAD

    should do the trick, and wipe out all changes since the last commit.

    When it comes time to submit your patch upstream, git can also generate the patch for you:

    git diff HEAD^ HEAD

    will generate a diff between your very last commit, and the one that preceded it.

    [1] http://www.kernel.org/pub/software/scm/git/docs/
    [2] http://www.debian.org/
    [3] http://en.wikipedia.org/wiki/Git_(software)
    [4] http://www.kernel.org/pub/software/scm/git/docs/everyday.html
    [5] http://git.or.cz/gitwiki/QuickStart
    [6] http://blog.langex.net/index.cgi/Linux/Debian/How-to-modify-source.html

    posted at: 02:12 | path: /Coding/git | permanent link to this entry

    Sat, 07 Mar 2009

    /Coding/php: Coding Conventions

    This[1] is, I think, an excellent and concise summary of PHP coding conventions, both for readability and for reducing load on the server. This is the first time I have heard of the advisability of avoiding double quotes, for instance.

    [1] http://www.weberp.org/ContributingtowebERP

    posted at: 00:54 | path: /Coding/php | permanent link to this entry

    Thu, 26 Feb 2009

    /Coding/python: Using Python to Connect to MySQL Over SSL

    Assuming you have already setup your MySQL server to accept external SSL connections[1], with the MySQLdb module and the help of a little documentation[2][3] getting Python to talk to the MySQL server is fairly straightforward:

    #!/usr/bin/python import MySQLdb ssl_settings = {'ca': 'db/cacert.pem', 'cert': 'db/cert.pem', 'key': 'db/key.pem' } try: db=MySQLdb.connect(host="www.yoursite.com", user="SSLremote", passwd="yourpassword", db="databasename", ssl=ssl_settings ) except: print "<p>ERROR: could not connect to database." c=db.cursor() c.execute("""SELECT * FROM thistablename""") while True: row = c.fetchone() if row == None: break else: print "<p>" print row

    Point your web browser at this piece of code (you probably have to name it "something.cgi" and put it a folder where your web server is enabled for cgi). This code will connect to the MySQL server and dump the contents of table thistablename into your browser window. If you get "ERROR: could not connect to database." then execute the same code in a terminal on the Python interpreter command line to see the actual MySQL error returned.

    Note that the path specification and permissions of the SSL files specified in "ssl_settings" are quite sensitive, as the web server must have access to them. I have stored the files with my code to allow a relative path. For security, these files should be "chmod 600" with the owner set to www-data (or whatever user name your webserver runs under....)

    [1] http://blog.langex.net/index.cgi/Admin/MySQL/
    [2] http://riskable.com/2009/02/12/how-to-use-ssl-with-the-python-mysqldb-module
    [3] http://mysql-python.sourceforge.net/MySQLdb.html

    posted at: 00:32 | path: /Coding/python | permanent link to this entry

    Wed, 07 Jan 2009

    /Coding/gdb: Analyzing a Daemon Segfault with GDB

    For the past little while cron on my laptop has been completely broken[1], with cron dumping an endless succession of segfaults (Segmentation Faults -- program attempts to access a forbidden area of memory) to the syslog. Since there was no apparent movement in the bug report, I set about trying to do something about it.

    First I ran strace and ltrace on cron to see if it was failing for some kind of obvious reason: no luck, nothing interesting in the output.

    Next thing to try: isolate exactly what line of code was failing using GDB[2].

    First build, install, and restart a debug version of cron and libpam-mount[3] (otherwise gdb output will just be a mess of numbers). Then start "gdb" in a root terminal (cron runs as root) and at the gdb prompt, attach to the running cron:

    (gdb) attach 3489
    Attaching to process 3489
    Reading symbols from /usr/sbin/cron...done.
    Reading symbols from /lib/libpam.so.0...done.
    Loaded symbols for /lib/libpam.so.0
    Reading symbols from /lib/libselinux.so.1...done.
    Loaded symbols for /lib/libselinux.so.1
    Reading symbols from /lib/i686/cmov/libc.so.6...done.
    Loaded symbols for /lib/i686/cmov/libc.so.6
    Reading symbols from /lib/i686/cmov/libdl.so.2...done.
    Loaded symbols for /lib/i686/cmov/libdl.so.2
    Reading symbols from /lib/ld-linux.so.2...done.
    Loaded symbols for /lib/ld-linux.so.2
    Reading symbols from /lib/i686/cmov/libnss_compat.so.2...done.
    Loaded symbols for /lib/i686/cmov/libnss_compat.so.2
    Reading symbols from /lib/i686/cmov/libnsl.so.1...done.
    Loaded symbols for /lib/i686/cmov/libnsl.so.1
    Reading symbols from /lib/i686/cmov/libnss_nis.so.2...done.
    Loaded symbols for /lib/i686/cmov/libnss_nis.so.2
    Reading symbols from /lib/i686/cmov/libnss_files.so.2...done.
    Loaded symbols for /lib/i686/cmov/libnss_files.so.2
    0xb7f5e424 in __kernel_vsyscall ()

    where the number after "attach" is the pid (process id) of cron (obtainable by running "ps -ef | grep cron"). Then:

    (gdb) set follow-fork-mode child
    (gdb) cont

    The follow-fork-mode parameter has to be set to "child" because cron is in the habit of spawning child processes, and it is in the child process where the error occurs. Then let cron (frozen at the moment of the "attach") continue running with "cont". And wait for a segfault:

    Program received signal SIGSEGV, Segmentation fault.
    [Switching to process 19892]
    0x00000000 in ?? ()

    Now dump out some interesting information about the error:

    (gdb) info frame 0
    Stack frame at 0xbfba1aa0:
     eip = 0x0; saved eip 0xb7b4cda1
     called by frame at 0xbfba1af0
     Arglist at 0xbfba1a98, args: 
     Locals at 0xbfba1a98, Previous frame's sp is 0xbfba1aa0
     Saved registers:
      eip at 0xbfba1a9c
    (gdb) up
    #1  0xb7b4cda1 in read_password (pamh=0x8841b00, 
        prompt=0x8846278 "reenter password for pam_mount:", pass=0xbfba1b38)
        at pam_mount.c:160
    160           retval = conv->conv(nargs, message, resp, conv->appdata_ptr);
    (gdb) up
    #2  0xb7b4ddf3 in pam_sm_open_session (pamh=0x8841b00, flags=32768, argc=1, 
        argv=0x8843ce0) at pam_mount.c:511
    511           ret = read_password(pamh, Config.msg_sessionpw, &system_authtok);
    (gdb) up
    #3  0xb7f693c1 in _pam_dispatch (pamh=0x8841b00, flags=32768, choice=4)
        at pam_dispatch.c:108
    108           retval = h->func(pamh, flags, h->argc, h->argv);
    (gdb) up
    #4  0xb7f6cfeb in pam_open_session (pamh=0x8841be8, flags=32768)
        at pam_session.c:23
    23          retval = _pam_dispatch(pamh, flags, PAM_OPEN_SESSION);
    (gdb) up
    #5  0x0804e848 in child_process (e=0x88418f8, u=0x88418d8) at 
    228             retcode = pam_open_session(pamh, PAM_SILENT);
    (gdb) up
    #6  0x0804e36d in do_command (e=0x88418f8, u=0x88418d8) at 
    102                     child_process(e, u);
    (gdb) up
    #7  0x0804e1e3 in job_runqueue () at ../job.c:68
    68                      do_command(j->e, j->u);
    (gdb) up
    #8  0x0804a777 in main (argc=142875624, argv=0x0) at 
    270                     job_runqueue();
    (gdb) up
    Initial frame selected; you cannot go up.

    Above I used the "up" and "down" commands to step through the bugtrace and see the stack of procedure calls (and which line in each procedure) that was happening at the moment of the segfault. From this we can divine that the problem seems to happen in libpam-mount, specifically in pam_mount.c:160 of frame #2. (Note that frame 0 is a mess of numbers without symbolic information, because the piece of software coinciding with frame 0 has not been compiled with debug.)

    (gdb) frame 0
    #0  0x00000000 in ?? ()
    (gdb) up
    #1  0xb7b4cda1 in read_password (pamh=0x8841b00, 
        prompt=0x8846278 "reenter password for pam_mount:", pass=0xbfba1b38)
        at pam_mount.c:160
    160           retval = conv->conv(nargs, message, resp, conv->appdata_ptr);
    (gdb) print *resp
    Cannot access memory at address 0x0
    (gdb) print resp
    $3 = (struct pam_response *) 0x0

    Above I used the "print" command to show the values the pointer *resp, which turns out to still be set to NULL, and is being passed on to another procedure (frame 0) which barfs. This is the likely problem.

    I issued a bug report against libpam-mount[4] suggesting a simple patch (which turned out to have bad side effects....) which prompted another developer to jump in and document an existing patch already applied to upstream. The fix is on the way....

    [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=484122
    [2] http://www.gnu.org/software/gdb/documentation/
    [3] http://blog.langex.net/index.cgi/Linux/Debian/How-to-modify-source.html
    [4] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=510990

    posted at: 02:18 | path: /Coding/gdb | permanent link to this entry