Yak Shaving

channam@pearl:~/Code/Locker$ npm install
npm WARN mongodb@0.9.6-15 package.json: bugs['web'] should probably be bugs['url']
npm WARN xmlbuilder@0.1.2 package.json: bugs['web'] should probably be bugs['url']
npm WARN wrench@1.1.0 package.json: bugs['web'] should probably be bugs['url']
npm WARN async@0.1.9 package.json: bugs['web'] should probably be bugs['url']
npm WARN request@2.1.1 package.json: bugs['web'] should probably be bugs['url']
npm WARN htmlparser2@1.0.0 package.json: bugs['web'] should probably be bugs['url']

> daemon@0.3.0 preinstall /home/channam/Code/Locker/node_modules/daemon
> node-waf configure build

> clucene@0.2.3 install /home/channam/Code/Locker/node_modules/clucene
> node-waf configure build

Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for g++                         : ok
Checking for node path                   : not found
Checking for node prefix                 : ok /usr/local
'configure' finished successfully (0.020s)
Waf: Entering directory `/home/channam/Code/Locker/node_modules/daemon/build'
no such environment: default
Traceback (most recent call last):
  File "/usr/local/bin/node-waf", line 16, in 
    Scripting.prepare(t, os.getcwd(), VERSION, wafdir)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 145, in prepare
    prepare_impl(t, cwd, ver, wafdir)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 135, in prepare_impl
    main()
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 188, in main
    fun(ctx)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 386, in build
    return build_impl(bld)
  File "/usr/local/bin/../lib/node/wafadmin/Scripting.py", line 399, in build_impl
    bld.add_subdirs([os.path.split(Utils.g_module.root_path)[0]])
  File "/usr/local/bin/../lib/node/wafadmin/Build.py", line 981, in add_subdirs
    self.recurse(dirs, 'build')
  File "/usr/local/bin/../lib/node/wafadmin/Utils.py", line 634, in recurse
    f(self)
  File "/home/channam/Code/Locker/node_modules/daemon/wscript", line 17, in build
    obj = bld.new_task_gen("cxx", "shlib", "node_addon")
  File "/usr/local/bin/../lib/node/wafadmin/Build.py", line 335, in new_task_gen
    ret = cls(*k, **kw)
  File "/usr/local/bin/../lib/node/wafadmin/Tools/ccroot.py", line 162, in __init__
    TaskGen.task_gen.__init__(self, *k, **kw)
  File "/usr/local/bin/../lib/node/wafadmin/TaskGen.py", line 118, in __init__
    self.env = self.bld.env.copy()
AttributeError: 'NoneType' object has no attribute 'copy'
npm ERR! error installing daemon@0.3.0 Error: daemon@0.3.0 preinstall: `node-waf configure build`
npm ERR! error installing daemon@0.3.0 `sh "-c" "node-waf configure build"` failed with 1
npm ERR! error installing daemon@0.3.0     at ChildProcess. (/usr/local/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing daemon@0.3.0     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing daemon@0.3.0     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! daemon@0.3.0 preinstall: `node-waf configure build`
npm ERR! `sh "-c" "node-waf configure build"` failed with 1
npm ERR!
npm ERR! Failed at the daemon@0.3.0 preinstall script.
npm ERR! This is most likely a problem with the daemon package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-waf configure build
npm ERR! You can get their info via:
npm ERR!     npm owner ls daemon
npm ERR! There is likely additional logging output above.
npm ERR!
npm ERR! System Linux 2.6.32-35-generic
npm ERR! command "node" "/usr/local/bin/npm" "install"
npm ERR! cwd /home/channam/Code/Locker
npm ERR! node -v v0.4.9
npm ERR! npm -v 1.0.106
npm ERR! code ELIFECYCLE
Setting srcdir to                        : /home/channam/Code/Locker/node_modules/clucene
Setting blddir to                        : /home/channam/Code/Locker/node_modules/clucene/build
Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for g++                         : ok
Checking for node path                   : not found
Checking for node prefix                 : ok /usr/local
Checking for library clucene-core        : no such environment: default
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/channam/Code/Locker/npm-debug.log
npm not ok
Traceback (most recent call last):

This seems to be caused by having installed a new version of Node (0.5 or 0.6 instead of 0.4) at some point. Find the directory where you built the newer Node version and run:

sudo make uninstall

This will remove all the files installed by the newer version. Now reinstall your 0.4 version and everything should be fine. I got the idea from here https://github.com/joyent/node/issues/1716.

So through a series of interesting events I have code in the app that runs Scraper Wiki. This is a very cool project and if you have some programming skills I recommend you head over and pitch in. You can wither write your own scraper or help with someone elses.

I have playing with Foursquare’s new API v2 to hopefully integrate it into gargoyle.me. Authentication is now a easier than ever, the code below explains how to get a usable auth token to use the API to get data out of Foursquare.

auth.py

foursquare_creds = {
    'key' : 'YOUR_FOURSQUARE_KEY',
    'secret' : 'YOUR_FOURSQUARE_SECRET',
    'return_url' : 'http://redirect_url.appspot.com',
}

foursquare_auth.py

from google.appengine.dist import use_library
use_library('django', '1.2')

from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import urlfetch
from django.utils import simplejson
from google.appengine.ext.webapp import template
from auth import foursquare_creds

class Index(webapp.RequestHandler):

    def get(self):
        path = os.path.join(os.path.dirname(__file__), 'templates/index.html')
        self.response.out.write(template.render(path, {}))

    def post(self):
        """Handle data posted from main form"""
        pass

class FourSquareOAuthRequest(webapp.RequestHandler):
    """Handle the oauth request out to foursquare"""

    def get(self):
        self.redirect("https://foursquare.com/oauth2/authenticate?client_id=%s&response_type=code&redirect_uri=%s" % (foursquare_creds['key'], foursquare_creds['return_url']))

class FourSquareOAuthRequestValid(webapp.RequestHandler):
    """Handle the oauth reply from foursquare"""

    def get(self):
        code = self.request.get('code')
        url = "https://foursquare.com/oauth2/access_token?client_id=%s&client_secret=%s&grant_type=authorization_code&redirect_uri=%s&code=%s" % (foursquare_creds['key'], foursquare_creds['secret'], foursquare_creds['return_url'], code)
        auth_json = urlfetch.fetch(url, validate_certificate=False)
        access_token = simplejson.loads(auth_json.content)

        # we now have a valid token
        # this token needs to be included with every API request

        friends_url = "https://api.foursquare.com/v2/users/self/friends?oauth_token=%s" % (access_token['access_token'])
        friends_json = urlfetch.fetch(friends_url, validate_certificate=False)
        friends = simplejson.loads(friends_json.content)

application = webapp.WSGIApplication(
                                     [('/', Index,),
                                       ('/auth', FourSquareOAuthRequest),
                                       ('/authreturn', FourSquareOAuthRequestValid)],
                                     debug=True)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()

For more details see the Foursquare documentation OAuth Docs

last.fm’s API is awesome and for various reasons I have spent a fair amount of time playing with it. I thought it would be neat to see where your most listened to artists or artist of your choosing might be playing so I created cassandra.appspot.com. This is a dead simple rendering of last.fm’s upcoming events for an artist on a simple google map.

I wrote this app about 2 years so there might be something similar out there now. There are few things that I hope to add in as and when I get chance. Go give it a try with either an artist name or your last.fm username.

Recently I have found myself with more ideas than time. So I set about limiting my ideas to time slots and using that time to create some working code. Recently I created githubfm in a spare hour. You enter your github.com username and your last.fm username and with a little bit of cleverness it tells you what you were listening too when you committed your code. It only works for public repositories as there is no oauth taking place. I thought asking people to authorise it might be a step too far.

It’s hosted on Google App Engine and written in Python. It is slightly slow as does quite a few requests to both githubs’s and last.fm’s apis. At some point I will get round to adding some markup as it’s as ugly as it’s possible to be.

The code is available of course at github.com/bassdread/lasthub, go fork it!

Recently I unfollowed a few people on twitter. It was nothing personal I was just following a few too many people and wanted to make my timeline more “tech” focused. Having removed these people I was surprised to see some of their tweets appearing in my timeline. People who I still followed where retweeting those people who I had stopped following. This got me thinking.

Now I can see the tweets that I might have missed following these people when people I follow deem it interesting. I have created a personal filter using my twitter friends by accident.

More and more I get my news and links from twitter (not so much Facebook). All these links are personally curated by people who I value based on their reputation and interests similar to mine.

The other side to this personal service is that it can be seen as a “filter bubble”. Only receiving links from people in a small group with similar view points and opinions. I am not entirely sure how this makes me feel. There is a pretty good reason I only follow people I agree with. On the other hand it’s nice learn other view points once in a while.

I guess the conclusion is to follow a few people who make you feel slightly uncomfortable.

I have been interested in social networks for a while and have been struck more than once that my updates (and those of my friends) are stuck in silos.

These silos are perfectly understandable, if you use foursquare its only logical they want your data to themselves. Nearly every big social network now has an API to provide access to this data and this got me thinking. A list of Facebook updates isn’t that interesting on it’s own but when combined with other updates to other services say your last.fm scrobbles suddenly a world of data mining and analysis opens up.

I loved the idea of being able to mine my updates and find out new things by playing with the raw data. This idea took and root and with the help of @1stvamp www.projectcausal.com was born.

Working with @1stvamp and our ever patient and brilliant designer for hire Matt (you should hire him to do stuff) we created Gargoyle Me.

This is a hosted version of Project Causal code which you can find over at GitHub. We made the decision to open source the underlying platform to allow people to help and improve how we fetch and work on the data. Gargoyle Me is a hosted version of this code base, created mainly top show off the platform.

Go and give it a try and if you like it follow us on @gargoylme, @projectcausal. All comments and patches gratefully accepted.

There are number of similar projects that are kicking off in a similar vein: Locker Project and ThinkUp App. These projects are aimed giving the user the ability to backup their history on these social services.

So in my pursuit of the RoR I thought I would share the process for debugger using Ruby 1.9.*

Running/Installing the Debugger

Start the server:

$ rails server --debugger

The following might appear:

$ rails server --debugger

=> Booting WEBrick
=> Rails 3.0.5 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
You need to install ruby-debug to run the server in debugging mode. With gems, use 'gem install ruby-debug'
Exiting

If so install the debug gem thus:

sudo gem install ruby-debug19

The 19 is important as without it you will receive the following error:

$ sudo gem install ruby-debug
Building native extensions.  This could take a while...
ERROR:  Error installing ruby-debug:
	ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.9.1 extconf.rb
Can't handle 1.9.x yet
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/bin/ruby1.9.1

Gem files will remain installed in /usr/lib/ruby/gems/1.9.1/gems/linecache-0.43 for inspection.
Results logged to /usr/lib/ruby/gems/1.9.1/gems/linecache-0.43/ext/gem_make.out

Now enable the 1.9 gem, by uncommenting line 18 (at least in my file):

gem 'ruby-debug19', :require => 'ruby-debug'

To use the newly installed debugger fire up the server:

$ rails server --debugger

Debugging

To pause the executing code add debugger to pause the running server and drop you into an interactive shell.

class HomeController < ApplicationController
  def index
    @events = 'meh!'
    debugger
  end
end

Causes the following in the terminal running the server:

$ rails server --debugger
=> Booting WEBrick
=> Rails 3.0.5 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
=> Debugger enabled
[2011-03-05 22:46:53] INFO  WEBrick 1.3.1
[2011-03-05 22:46:53] INFO  ruby 1.9.2 (2010-08-18) [x86_64-linux]
[2011-03-05 22:46:53] INFO  WEBrick::HTTPServer#start: pid=4461 port=3000
/usr/lib/ruby/gems/1.9.1/gems/actionpack-3.0.5/lib/action_controller/metal/implicit_render.rb:5
default_render unless response_body
(rdb:2)

You can then call methods to inspect the current code.

This is a lovely solution and compared to having to buy an IDE to get a similar debugging functionality (yes, yes pdb, but meh) in Python this is pure win. As usual Ruby Guides have an excellent run down tutorial at http://guides.rubyonrails.org/debugging_rails_applications.html.

So thought it was time to upgrade and start Ruby on Rails on the latest stable version. First problem I hit was the latest Rails packaged for Ubuntu Maverick is 1.8 Ruby. So if you don’t mind being behind its fine.

What follows is how to get the latest greatest Rails on 1.9.2 Ruby.

sudo apt-get install ruby1.9.1-full

This gets you latest packaged Ruby, note its 1.9.1. After install you will have no “ruby” command so symlink up ruby to the actual binary thus:

$ ruby --version
The program 'ruby' is currently not installed.  You can install it by typing:
sudo apt-get install ruby
$ ruby1.9.1 --version
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
$ sudo ln -s /usr/bin/ruby1.9.1 /usr/bin/ruby
$ ruby --version
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]

Now we are rocking the latest Ruby using the command ruby.

Next install gems from rubygems-1.6.0.tgz. Download and run the installer:

$ wget http://production.cf.rubygems.org/rubygems/rubygems-1.6.0.tgz
$ tar xvzf rubygems-1.6.0.tgz
$ cd rubygems-1.6.0
$ sudo ruby setup.rb
$ sudo ln -s /usr/bin/gem1.9.1 /usr/bin/gem

Yay all installed. Installer the updater as per the instructions printed in the installer bit:

$ sudo gem install rubygems-update
Fetching: rubygems-update-1.6.0.gem (100%)
Successfully installed rubygems-update-1.6.0
1 gem installed
Installing ri documentation for rubygems-update-1.6.0...
Installing RDoc documentation for rubygems-update-1.6.0...
$ sudo update_rubygems
RubyGems 1.6.0 installed

Finally install rails:

$ sudo gem install rails

Before building your project chances are you need to install sqlite3 deps to head off any bundle errors.

$ sudo apt-get install libsqlite3-dev

This will stop:

checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 +universal'

Now you are ready to roll.

Recently I was trying to iterate over a dictionary in a template, this did not end well. By default App Engine is using the Django 0.96 templating engine. Attempting to use

{% for key, value in dict.items %}

gave the cryptic error ‘for’ statements with five words should end in ‘reversed’ a bit of googling revealed this was due to using Django 0.96.

Later versions of App Engine now support Django templating using 1.2 as well as 0.96. You can use 1.0 and 1.1 but you will to install them yourself. More info at http://code.google.com/appengine/docs/python/tools/libraries.html#Django. When using the imports make sure the library part goes before your other imports:

from google.appengine.dist import use_library
use_library('django', '1.2')

from lastfm.helpers import get_events_for_artist, get_events_for_user
from google.appengine.api import users
from google.appengine.ext import webapp

Otherwise you see: django 1.2 was requested, but 0.96.4.None is already in use errors.