Yak Shaving

So through a series of interesting events I have code in the app that runs Scraper Wiki. This is a very cool project and if you have some programming skills I recommend you head over and pitch in. You can wither write your own scraper or help with someone elses.

I have playing with Foursquare’s new API v2 to hopefully integrate it into gargoyle.me. Authentication is now a easier than ever, the code below explains how to get a usable auth token to use the API to get data out of Foursquare.

auth.py

foursquare_creds = {
    'key' : 'YOUR_FOURSQUARE_KEY',
    'secret' : 'YOUR_FOURSQUARE_SECRET',
    'return_url' : 'http://redirect_url.appspot.com',
}

foursquare_auth.py

from google.appengine.dist import use_library
use_library('django', '1.2')

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
from django.utils import simplejson
from google.appengine.ext.webapp import template
from auth import foursquare_creds

class Index(webapp.RequestHandler):

    def get(self):
        path = os.path.join(os.path.dirname(__file__), 'templates/index.html')
        self.response.out.write(template.render(path, {}))

    def post(self):
        """Handle data posted from main form"""
        pass

class FourSquareOAuthRequest(webapp.RequestHandler):
    """Handle the oauth request out to foursquare"""

    def get(self):
        self.redirect("https://foursquare.com/oauth2/authenticate?client_id=%s&response_type=code&redirect_uri=%s" % (foursquare_creds['key'], foursquare_creds['return_url']))

class FourSquareOAuthRequestValid(webapp.RequestHandler):
    """Handle the oauth reply from foursquare"""

    def get(self):
        code = self.request.get('code')
        url = "https://foursquare.com/oauth2/access_token?client_id=%s&client_secret=%s&grant_type=authorization_code&redirect_uri=%s&code=%s" % (foursquare_creds['key'], foursquare_creds['secret'], foursquare_creds['return_url'], code)
        auth_json = urlfetch.fetch(url, validate_certificate=False)
        access_token = simplejson.loads(auth_json.content)

        # we now have a valid token
        # this token needs to be included with every API request

        friends_url = "https://api.foursquare.com/v2/users/self/friends?oauth_token=%s" % (access_token['access_token'])
        friends_json = urlfetch.fetch(friends_url, validate_certificate=False)
        friends = simplejson.loads(friends_json.content)

application = webapp.WSGIApplication(
                                     [('/', Index,),
                                       ('/auth', FourSquareOAuthRequest),
                                       ('/authreturn', FourSquareOAuthRequestValid)],
                                     debug=True)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()

For more details see the Foursquare documentation OAuth Docs

last.fm’s API is awesome and for various reasons I have spent a fair amount of time playing with it. I thought it would be neat to see where your most listened to artists or artist of your choosing might be playing so I created cassandra.appspot.com. This is a dead simple rendering of last.fm’s upcoming events for an artist on a simple google map.

I wrote this app about 2 years so there might be something similar out there now. There are few things that I hope to add in as and when I get chance. Go give it a try with either an artist name or your last.fm username.

Recently I have found myself with more ideas than time. So I set about limiting my ideas to time slots and using that time to create some working code. Recently I created githubfm in a spare hour. You enter your github.com username and your last.fm username and with a little bit of cleverness it tells you what you were listening too when you committed your code. It only works for public repositories as there is no oauth taking place. I thought asking people to authorise it might be a step too far.

It’s hosted on Google App Engine and written in Python. It is slightly slow as does quite a few requests to both githubs’s and last.fm’s apis. At some point I will get round to adding some markup as it’s as ugly as it’s possible to be.

The code is available of course at github.com/bassdread/lasthub, go fork it!

Recently I unfollowed a few people on twitter. It was nothing personal I was just following a few too many people and wanted to make my timeline more “tech” focused. Having removed these people I was surprised to see some of their tweets appearing in my timeline. People who I still followed where retweeting those people who I had stopped following. This got me thinking.

Now I can see the tweets that I might have missed following these people when people I follow deem it interesting. I have created a personal filter using my twitter friends by accident.

More and more I get my news and links from twitter (not so much Facebook). All these links are personally curated by people who I value based on their reputation and interests similar to mine.

The other side to this personal service is that it can be seen as a “filter bubble”. Only receiving links from people in a small group with similar view points and opinions. I am not entirely sure how this makes me feel. There is a pretty good reason I only follow people I agree with. On the other hand it’s nice learn other view points once in a while.

I guess the conclusion is to follow a few people who make you feel slightly uncomfortable.

So in my pursuit of the RoR I thought I would share the process for debugger using Ruby 1.9.*

Running/Installing the Debugger

Start the server:

$ rails server --debugger

The following might appear:

$ rails server --debugger

=> Booting WEBrick
=> Rails 3.0.5 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
You need to install ruby-debug to run the server in debugging mode. With gems, use 'gem install ruby-debug'
Exiting

If so install the debug gem thus:

sudo gem install ruby-debug19

The 19 is important as without it you will receive the following error:

$ sudo gem install ruby-debug
Building native extensions.  This could take a while...
ERROR:  Error installing ruby-debug:
	ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.9.1 extconf.rb
Can't handle 1.9.x yet
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/bin/ruby1.9.1

Gem files will remain installed in /usr/lib/ruby/gems/1.9.1/gems/linecache-0.43 for inspection.
Results logged to /usr/lib/ruby/gems/1.9.1/gems/linecache-0.43/ext/gem_make.out

Now enable the 1.9 gem, by uncommenting line 18 (at least in my file):

gem 'ruby-debug19', :require => 'ruby-debug'

To use the newly installed debugger fire up the server:

$ rails server --debugger

Debugging

To pause the executing code add debugger to pause the running server and drop you into an interactive shell.

class HomeController < ApplicationController
  def index
    @events = 'meh!'
    debugger
  end
end

Causes the following in the terminal running the server:

$ rails server --debugger
=> Booting WEBrick
=> Rails 3.0.5 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
=> Debugger enabled
[2011-03-05 22:46:53] INFO  WEBrick 1.3.1
[2011-03-05 22:46:53] INFO  ruby 1.9.2 (2010-08-18) [x86_64-linux]
[2011-03-05 22:46:53] INFO  WEBrick::HTTPServer#start: pid=4461 port=3000
/usr/lib/ruby/gems/1.9.1/gems/actionpack-3.0.5/lib/action_controller/metal/implicit_render.rb:5
default_render unless response_body
(rdb:2)

You can then call methods to inspect the current code.

This is a lovely solution and compared to having to buy an IDE to get a similar debugging functionality (yes, yes pdb, but meh) in Python this is pure win. As usual Ruby Guides have an excellent run down tutorial at http://guides.rubyonrails.org/debugging_rails_applications.html.

This is more a reminder for me The code below allows you to talk to ZODB in an interactive way.

$ bin/instance debug

from Acquisition import aq_parent
from zkaffold.consolehelpers import login
from Products.CMFCore.utils import getToolByName
app = login(app, 'zopeadmin')
pc = getToolByName(app.portal, 'catalog')
content_fragments = pc({'getId': 'id', 'portal_type': 'ContentFragments'})
content_fragments
[, ]

Login allows you to query the catalog otherwise no results are returned.

The code below will refresh a web page when the file is updated. I was using it to make syslog available over http.

http = require('http');
var spawn = require('child_process').spawn;
var sys = require('sys')

// log to monitor
var filename = process.ARGV[2];

if (!filename)
  return sys.puts("Usage: node monitor.js filename");

// fire up tail on the log file
var tail = spawn("tail", ["-f", filename]);

http.createServer(function (req, update) {
  update.writeHead(200, {'Content-Type': "text/plain;"});
  tail.stdout.on("data", function (data) {
    update.write(data);
  });
}).listen(8000);

sudo apt-get install build-essential libssl-dev

Should be enough to make ./configure happy.

For my latest CV click here.