Intermediate and Advanced Software Carpentry in Python

By K4Paul 0 Development , , , ,

I am taking a slight break from C++, going back to Python, one of my favorite languages ever! I found the following material on structuring Python code and it is great! Posting it here so I can always come back to it. I recommend it to all the Python enthusiast looking at ways to improve their coding structure.

Intermediate and Advanced Software Carpentry in Python

Welcome! You have stumbled upon the class handouts for a course I taught at Lawrence Livermore National Lab, June 12-June 14, 2007. These notes are intended to accompany my lecture, which was a demonstration of a variety of “intermediate” Python features and packages. Because the demonstration was interactive, these notes are not complete notes of what went on in the course. (Sorry about that; they have been updated from my actual handouts to be more complete…) However, all 70 pages are free to view and print, so enjoy.
Source: ivory.idyll.org/articles/advanced-swc/

Python URL Term Scrapper (Kind of Like a Very Dumb/Simple Watson)

By K4Paul 0 Development, Python, Web , , , , ,

I wrote this a while back to scrape a given URL page for all links it contains and then search those links for term relations. The Python scrapper first finds all links in a given url. It then searches all the links found for a list of search terms provided. It will return stats on the number of times specific provided terms show up.

This may come in handy while trying to find more information on a given topic. I’ve used it on Google searches (be careful, you can only scrape google once ever 8 or so seconds before you are locked out) and wikipedia pages to gather correlation statistics between topics.

It is old code… so there might be some errors. Keep me posted!

#!/usr/bin/env python
#ENSURE permissions are 755 in order to have script run as executable

import os, sys, re, datetime
from optparse import OptionParser
import logging, urllib2

def parsePage(link, list):
    searchList = {}
    try:
        f = urllib2.urlopen(link)
        data = f.read()
        for item in list:
            if (item.title() in data) or (item.upper() in data) or (item.lower() in data):
                searchList[item]=searchList[item]+1
                searchList["count"]=searchList["count"]+1
        return searchList
    except Exception, e:
        print "An error has occurred while parsing page " +str(link)+"."
        log.error(str(datetime.datetime.now())+" "+str(e))

def searchUrl(search):
    try:
        f = urllib2.urlopen(search)
        data = f.read()
        pattern = r"/wiki(/\S*)?$" #regular expression to find url
        links = re.findall(pattern, data)
        return links
    except Exception, e:
        print "An error has occurred while trying to reach the site."
        log.error(str(datetime.datetime.now())+" "+str(e))

def main():
    try:
        parser = OptionParser() #Help menu options
        parser.add_option("-u", "--url", dest="search", help="String containing URL to search.")
        parser.add_option("-f", "--file", dest="file", help="File containing search terms.")
        (options, args) = parser.parse_args()
        if not options.search or not options.file:
            parser.error('Term file or URL to scrape not given')
        else:
            urls = searchUrl(options.search)
            f = open(options.file, 'r')
            terms = f.readlines()
            for url in urls:
                parsePage(url, terms)
            print "Results:"
            print searchList
    except Exception, e:
        log.error(str(datetime.datetime.now())+" "+str(e))

if __name__ == "__main__":
    log = logging.getLogger("error") #create error log
    log.setLevel(logging.ERROR)
    formatter = logging.Formatter('[%(levelname)s] %(message)s')
    handler = logging.FileHandler('error.log')
    handler.setFormatter(formatter)
    log.addHandler(handler)
    try:
        main()
    except Exception, e:
        print "An error has occurred, please review the error.log for more details."
        log.error(str(datetime.datetime.now())+" "+str(e))

Python TypeError: expected a character buffer object

By K4Paul 0 Development, Python , , , , , , ,

Error

I received the following python error:

TypeError: expected a character buffer object

The below screenshot shows my code.

Python Error

Solution

I was using the replace method, while the better method for my situation would be to use the re.sub(patter, replace, string) method, the new line became:

my_text = re.sub(comp, '#undef SEEK_SET\n#undef SEEK_END\n
#undef SEEK_CUR\n#include ', f)

The replace method expects string parameters (looks for a string within a string), while I wanted to use a regular expression to search a string.

SSL Encryption for Django’s Local/Native Server

By K4Paul 0 Development, Linux, Python , , , , , , , , , , ,

Django comes packaged with a lightweight python server. It is not intended to be a production server but more a testing/development host. Running the server is as easy as running the following command within a Django project:

python manage.py runserver

Since it’s so lightweight, it doesn’t come with the same abilities as other servers like Apache or Nginx. It can’t perform encryption, however, there’s a nifty tool called stunnel that can do it for you!

“Stunnel is an open-source multi-platform computer program, used to provide universal TLS/SSL tunneling service” (Wikipedia).

Environment

The following steps were performed on my iMac running OS X Mavericks with a Django 1.5 installation. I believe my instructions should still work for different versions (most) and Linux distributions.

Steps

Initially, I downloaded the latest version of Stunnel, however I ran into numerous compiling issues. One of them being: “ld: warning: directory not found for option ‘-L/usr//lib64.’” The error indicated I did not have the necessary 64x library. When I downloaded version, 4.54, everything compiled nicely.

  • Download the stunnel-4.54.tar.gz source code.
  • Open a terminal window and run the following command to untar (unzip) the file.
 tar –xvf stunnel-4.54.tar.gz
  • Run the following commands to enter the directory and install the tool (credit).
cd stunnel-4.54
./configure && make && make check && sudo make install
  • During the install stage, you will be required to enter in certificate data. Stunnel will conveniently make a self-signed SSL certificate for you and save it to /usr/local/etc/stunnel/stunnel.pem. Thanks Stunnel!
  • Create a configuration file for Stunnel (credit). I put the file inside my Django project to keep things organized.
vim dev_https
  • Edit the file and add the following lines in order to manipulate Stunnel to work with your environment.
pid=
cert=/usr/local/etc/stunnel/stunnel.pem
foreground=yes
debug=7
[https]
accept=<HTTPS ACCEPTING PORT>
connect=<LOCAL PORT YOUR DJANGO SERVER IS USING>
TIMEOUTclose=1
  • Save the file (For vim: ESC ‘:wq’ ENTER).

config

  • Start the Stunnel HTTPS tunneling service.
sudo stunnel <PATH OF dev_https>

stunnel
  • Next, start your Django server.
python manage.py runserver 127.0.0.1:< LOCAL PORT YOUR DJANGO SERVER IS USING>

django

Note – I used 127.0.0.1 purposefully as my hosting IP address, I only want Django to run locally. I do not want the server to run on a public/accessible IP.  Only stunnel will receive web requests.

That’s it! Now stunnel is listening for all encrypted, incoming messges on whatever port you specified. When a request comes in, it will decrypt it and send it locally to your Django server. Following, Django will then respond through the tunnel to the requesting client with the proper data.

PIR Sensor on the Pi

By K4Paul 3 Hardware, Kali, Linux, Python , , , , , , , , , , , ,

Today I soldered a PIR sensor to my Pi! Basically, I want it to detect movement and turn on a LCD screen, then turn the screen off again after a minute of no movement. So when I walk into a room, the screen turns on and when I leave, the screen turns off.

Equipment

Solder

First thing, I looked up the pinout for the Raspberry Pi. The below diagram comes from elinux.org.

We care about one of the 5V, ground and GPIO25 pins.

  • Solder the sensor red cable to either 5V.
  • Solder the black cable to ground.
  • End by soldering the yellow line to GPIO25.

Your results should be similar to my picture below.

back

Next, I used this guy’s pir.py script. The script requires the Python library RPi.GPIO. I installed this by downloading the library from here, the direct link is here. To untag or unzip the file I used the following command:

tar -xvf RPi.GPIO-0.5.4.tar.gz

Before installing it, make sure you have python-dev installed.

apt-get install python-dev

With that necessary package, install RPi.GPIO.

cd RPi.GPIO-0.5.4
python setup.py install

Now you can run the pir.py script. I made some slight changes to his code. I didn’t feel the need to call separate scripts to run a single command so I made the following edits.

import subprocess

to

import os

and

def turn_on(): 
    subprocess.call("sh /home/pi/photoframe/monitor_on.sh", shell=True)
def turn_off(): 
    subprocess.call("sh /home/pi/photoframe/monitor_off.sh", shell=True)

this

def turn_on(): 
    os.system("chvt 2")
def turn_off(): 
    os.system("chvt 2")

Run the script and test it out! The sensor will turn off after a minute of no movement and on again once it detects something. I ended by setting my script to run on startup.

2014-01-30 20.33.51

I need to put a picture in the frame to act as background to the pi…