Planet Cleanstick

Shared by Jason Scheirer I DO THIS!

Shared by Jason Scheirer
I DO THIS!

Just a visiting object notation model, you know.

Just a visiting object notation model, you know.

Alela Diane - White As Diamonds

Alela Diane - White As Diamonds

Shearwater - Black Eyes Daaaaang.

Shearwater - Black Eyes

Daaaaang.

asbo.org.uk, CakePHP and data.gov.uk

Yesterday, I wrote asbo.org.uk, a site which provides really basic visualisation of the UK's anti-social behaviour order data from 1999-2007. This data was recently released by data.gov.uk.

I wrote the whole site, wrangled the data, and deployed it yesterday afternoon, in about six hours. I've got some contract work coming up using CakePHP, so I wanted to try it out, and I wanted to see what I could do in such a short space of time. I'm quite pleased with the results. There's no analysis of the data, just presentation, but I was trying to see what I could do in the time I had, rather than develop features.

Originally, I wanted to write a 'how safe am I' sort of application, which could offer data about the types of crime most likely to occur in a given area, but this idea was pretty much killed by the time I saw the actual data available. Maybe I don't know enough about Excel-scraping, but I considerably reduced the scope of this project because the data was such a mess - it's just not worth the time to extract information from arbitrarily formatted spreadsheets. Formatting the single table of data used on asbo.org.uk took about three hours, which seems like a waste.

I was generally quite impressed with CakePHP; it's laid out in a sane way, though coming from Python some of the automatic discovery of models in the controllers feels a little bit magic, and I'm not sure I'm comfortable passing around arrays of data rather than data objects themselves, but it's not a deal breaker. I do like the layout system, something I commonly implement with blocks in Django, but seeing it formalised as part of the recommended approach to application templating is nice.

I put the source on Github,if anyone is interested.

Sikuli desktop automation

There's quite a lot of buzz around Project Sikuli at the moment, so I spent time today playing with it.

Sikuli is a GUI automation engine which uses a vision engine to identify elements on screen. In practise, it works well as a quick way to script repetitive desktop actions, without having to learn the AppleScript actions an application provides, or how to hook into a desktop accessibility framework to manipulate applications. Instead, you tell the Sikuli engine how you expect regions of the screen to look, and then how to further manipulate them with clicks, key-presses and so on.

That is quite an abstract explanation, so here is a practical example.

Every morning, when I sit down at my desk, I do two things: I do some basic maintenance on my Mac, and I print out my day planner sheet. This only takes maybe 5 minutes, but I need to wait in front of the screen and click the right buttons at the right time. That's pretty dull, so here's how I'm using Sikuli to automate that.

First of all, the maintenance script. I use CleanMyMac to perform one big system clean every morning, which requires a few steps; launching the application, scanning for files to clean, approving the list of files, running the cleaning process, then closing the application.

My Sikuli script for doing this looks like this:

The script is fairly self-explanatory. First, it switches to (or opens) the CleanMyMac application, then runs the scan process. The script sleeps for five seconds, the searches for the 'scan finished' message, and goes to sleep again until that message is displayed. Then it performs the clean operation, and again waits for the 'clean finished' message. After this, it closes the application.

At no point is this script passing events directly to the application, nor is it querying the application to get information about its state; it's just examining the frame-buffer and looking for patterns which are similar to those specified in the script. On the Sikuli website, they claim their vision engine is smart enough to still work even if an application slightly changes it's visual style, but I haven't been able to test this.

This is the script for printing my day planner:

First, it closes then re-opens Pages so that it is in a predictable state. Then, the script manipulates the 'recent documents' drop-down, to open my 'Day Sheet 2' document. Notice that the script uses the 'wait' function frequently so that the vision engine isn't searching for an image before it has been drawn by the operating system. Once the document is open, we pass the operating system the ⌘P keyboard shortcut to open the print dialogue, then click 'print'. Finally, Pages is closed, and I check my printer tray. Here's a screen-cast of this in action. Note that I use the 'Run and show each action' button to start the script. This way, you can see the vision engine matching and highlighting each element on the screen:

Sikuli does have some limitations; You can't copy and paste between scripts, which I think is due to how the IDE stores image region data on the filesystem. Also, scripts seem wedded to the IDE, so you can't launch a script without needing to click the 'run' button in the IDE, but as the IDE is really just a thin wrapper over an underlying Jython instance, I'm sure this would be possible with a little more digging.

The obvious next step is to properly test how well Sikuli does deal with visual changes; using Sikuli to test web applications would be a great addition to the toolbox (and it would eliminate the problems with brittle tools like Selenium), and the IDE and language is simple enough for non-programmers to take some of the burden of writing tests.

I'm quite looking forward to the future of Sikuli. I'd like to see this visual search technology make it into more traditional scripting environments like AppleScript, though I'm not sure that'll happen any time soon, but anything which reduces reliance on bending traditional accessibility frameworks to perform in this role is a step forward.

Django Project Base

"This is my project base. There are many like it, but this one is mine."

Today I finally got around to putting my Django project base on Github. I've been using this base for about six months now, and after a lot of rewrites and different approaches, it's now reasonably stable. I've been starting a lot of new projects recently, and repeatedly fixing the same small bugs in this project template, so I decided to spend a few hours this afternoon cleaning it up and making it public.

I'm not totally satisfied with my use of shell scripts to do some of the bootstrap actions (ideally I'd use Fabric for all of these tasks), and longer-term I want to make it easier to rename the django project rather than using search/replace in TextMate.

I know there are a lot of similar projects to this on Github, but none of them worked exactly like I wanted (I think too many depend on zc.buildout and similar), and while I'm not crazy about re-inventing the wheel, I hope there are enough other people out there who share my preferences who will find this useful.

Headless Virtualbox on OSX

When I'm developing, I try to continuously deploy to a realistic environment as often as possible. This means a Debian server running a stack as close to production as I can get. Of course, I don't deploy to the actual production servers (a lesson I've learnt many times over), so I virtualise a Debian box and clone package set from the production server. Combined with bridged networking and a quick hosts file change, and I've got a production-equivalent server always available at server.local.

Hence, I use VirtualBox all the time on my Mac, and I've got at least one server instance running all the time. But the Virtualbox.app clutters up the dock, and I really just want the instance available via SSH, I don't care about actually seeing its framebuffer.

Fortunately, you can run headless Virtualbox instances on OSX. It just takes a little extra work. Assuming you installed Virtualbox in the default location, headless Virtualbox binaries live in:

$ ls /Applications/VirtualBox.app/Contents/MacOS | grep "VBoxHeadless"
VBoxHeadless
VBoxHeadless-amd64
VBoxHeadless-x86
VBoxHeadless.dylib

You can easily run an instance in headless mode by specifying the unique name of the VM on the command line (replace "Debian Server" with the name of your VM):

/Applications/Virtualbox.app/Contents/MacOS/VBoxHeadless --startvm "Debian Server"

Of course, it's a pain to have to start these instances each time you boot the host system, but you can have the VM start on login by adding a LoginHook. Here's how:

sudo defaults write com.apple.loginwindow LoginHook /Applications/Virtualbox.app/Contents/MacOS/VBoxHeadless --startvm "Debian Server"

If you want to stop the VM, just use whatever ACPI shutdown method the guest instance provides (for a Debian server, sudo /sbin/init 0 will suffice), and the VBoxHeadless process will silently quit. If you're virtualising a graphical OS, like Windows, you should probably check the documentation to use with VBoxHeadless; you can easily specify a VRDP address so you can connect to the instance via any RDP client.

New Years Resolution: Stop switching tools

Like nearly everyone else on the planet, I've spent some of today thinking about how I can do better next year. My resolutions are roughly split into two categories; tech stuff, and everything else, which makes sense considering I spend at least half of my waking hours in front of a computer.

Since I can remember, I've compulsively tinkered with the computer in front of me; changing hardware, changing operating system, experimenting with different editors, different languages and different organisational systems. I'm a classic computing dilettante, I have a very broad knowledge and very shallow of many areas of programming and administration, but no real expertise anywhere. This bleeds into my choice of environment; I'm forever switching tools and languages trying to find the perfect tool. After years of searching, I've not come any appreciable distance nearer to that goal, so it's time to stop trying.

From what I can tell, this is a fairly common disease among programmer types. Some may tinker with software, some with their time management system, but the problem remains the same; we are wasting time trying to perfect a system which will never be perfect.

So, the main tech resolution I've made is to stop trying new things. This has been happening naturally over the last few years anyway, it's probably a combination of age and habit, but I'm going to endeavour to ignore new software, new methodologies and new trends in the coming 12 months. For the next year, I'm going to use the following tools, without deviation:

• OSX Snow Leopard on my laptop, Debian Lenny on the servers. I already know these platforms, they're both stable, and I have no reason to try anything else. I'm often tempted to try the latest releases of operating systems, which inevitably results in a week of lost productivity. As far as I know, there is no scheduled OSX release in 2010, and no release date has yet been set for Squeeze.
• TextMate. I've spent about half of 2009 using Emacs, but the power of Emacs comes at the expense of distraction; the hours I've spent writing elisp to try to save myself 10 seconds here and there far outweigh the actual time saved. TextMate is stable, the next release is a long way off (in fact, the TextMate 2 FAQ says it's coming when we get Duke Nukem Forever, so I guess that means never), and the lack of customisability compared to Emacs means there's just less to tinker with.
• IPython. I've spent far too long investigating other Python Repls, and while bpython is nice, IPython and ipdb.set_trace() have become so ingrained in my muscle memory that changing to anything else isn't worthwhile.
• Django. I'm not sure exactly how the roadmap for Django is laid out, but I suspect the 1.2 release will be with us for most of 2010, and Django is adequate for all the work I'll be doing with 84labs in the next year. I've toyed with Kohana, Rails and Seashore in the last year, and none of those brought any significant benefits over Django that would be worth holding on to.
• Safari. Safari is the most pleasant browsing experience to use on OSX. 2009's development of the Web Inspector removed any real dependency I had on Firebug, and combined with ClickToFlash and /etc/hosts level ad blocking, I've no need to use any other browser. Thankfully, I very rarely need to test my work in other browsers, so while I'll probably keep around a Firefox installation, it's very unlikely that it'll become my browser of choice again.

So, that's it. We'll see in 2011 if I'm still using this software (that's where the 'resolve' comes in), but I'm looking forward to gaining at least a week or two of productive work simply by keeping a static environment.

Will anyone else commit to joining me?

I don't really use OSX anymore

I have two desktop systems, side-by-side: an Intel Mac Mini and an Intel 21” iMac. The Mini runs Leopard and the iMac runs Ubuntu Karmic Koala, and I find myself completely satisfied with the Linux desktop, and switch back to OSX as an auxiliary rather than as my primary.

I started on Debian back in 1999. I wanted to get into Linux, but both Red Hat and SuSE were a little hard to get going for beginners, and the packages supplied were always a weird grab bag of old and new. I remember KDE 2 was awesome, but I don’t remember much past that. A friend and coworker introduced me to Debian and I was amazed at how much more intuitive it was to use. It didn’t leave me out in the cold to fend for myself, but it automated enough of the low-levels of Linux that it made the rest seem approachable. And its default configuration was super lightweight and not memory hungry at all. I was sold.

I started using Apple hardware again six years ago, on my 21st birthday. Right at the “Apple Renaissance” for developers I bought myself a 12” Aluminum Powerbook and it got me through my last two years of college and the year and a half after it of doing research and my first full-time job. It was probably the most innovative computer I’d ever owned to that point — it had a Mini-DVI port that I could use to have a dual headed system (which was the coolest thing ever) and Bluetooth, which meant I could sync my contacts and calendars with my Sony Ericsson cell phone. I can’t tell you how many times I made it to class only because my phone beeped at me 5 minutes before lecture started. It was self contained: I could do all my C++/Python/Java work for CS courses and contract work without installing anything, it had word processors and diagramming software that handled international text for my linguistics courses, it was a perfect computing device for me at the time.

My love affair continued because of OSX desktop software — it was a pretty face with usable apps on a standard UNIX system: cron and friends were all there for the using. It even had some new technologies that were really exciting: I still consider MDNS-SD one of the greatest consumer networking advancements ever. I could install software in an apt-like way with fink or ports. It always felt like there was a little friction there with a near-parallel UNIX system on a UNIX system when I ran these utilities, but it was good enough.

As OSX aged, it began to diverge from the standard (notably, launchd made me take notice) and felt less open. What was once a pretty face on a standard UNIX became a slightly prettier face on an increasingly non-standard operating system. I had to re-learn the same things over and over from release to release as they began to deprecate the old tools and methods for doing things. Again, the notable example here is launchd, which replaced init.d and cron. While the Quartz window manager and rendering seemed great in 2001, the advancements in X like Cairo and Compiz have caught up feature-and-performance-wise and are open and cross-platform. Apple’s slowly adding not-so-open utilities to scrape up edge cases like GrandCentral, which is kind of an indicator that the desktop environment itself is pretty much stagnant and Apple’s innovation is going to go into other semi-proprietary technologies on top of their exisitng stack.

Installing software is obviously much easier in OSX (discounting apt): I drag an icon to /Applications. If a title isn’t in apt, or is outdated in the Ubuntu repositories, it’s a pain to get going. But more and more apps are providing .debs (with daily builds!), or better yet, are showing up in the Ubuntu PPAs and this is becoming less of an issue. I’m also learning to live with 6 month old software: it’ll get in when it gets in, I’ll live with the features it has now. Usually the bleeding edge has APIs that are so in flux that it makes absolutely no sense to develop against them.

So I find myself on Ubuntu. The UI is familiar and at this point it’s easier to test and deploy apps onto a Debian server using the system that is more Debian-like than the one that is not. It’s fast, it’s responsive, and while it can be idiomatic in its own right for a developer who mostly uses vi, Python and C, it’s great. I can’t really heap too much praise or criticism on it, it just is there and gets out of the way. Ubuntu sucked for stability in 2006, but it works well now. It lets me get stuff done. The new “features” like Ubuntu One that are being constantly tacked on stay out of the way and can even be uninstalled. I had to install a third-party hack to get rid of the Mobile Me account thing in the Finder.

This isn’t to say I’ve had a falling out with the Mac: just having used it for 6 years now, it seems a lot less exciting and a lot more of a hassle to use than it used to. I’m slowly using it less and less and while I will likely always have OSX and Windows as a desktop install somewhere, it won’t necessarily be my go-to platform to get things done. So from a highly subjective point of view, I seem to be unconsciously preferring Linux to OSX to do my daily computing chores, and the trend seems to be strengthening every day.

bunnyrockstar: party time with arctic fox, cupcake, and sir....

bunnyrockstar:

party time with arctic fox, cupcake, and sir. hat

Selleck Waterfall Sandwich

Happy New Ear

Happy New Ear

"That dog is insane for lasers!"

For your convenience

For your convenience

Hi

I am way late in getting a Tumblr, I have been reading others’ for years now but had no reason to get my own. Now here I am. And this time it’s personal.

On Cheapness

I love lamp

Jason Scheirer posted a photo:

At LACMA

OFF

Jason Scheirer posted a photo:

Ante show

Jason Scheirer posted a photo:

Azure Ray

Jason Scheirer posted a photo:

If You Fall

Jason Scheirer posted a photo:

Fuzzy couch

Jason Scheirer posted a photo:

Blown out couch

Jason Scheirer posted a photo:

I should have used a better camera couch

Jason Scheirer posted a photo:

Toulouse

Jason Scheirer posted a photo:

Better focus

Jason Scheirer posted a photo:

A mantis nymph somehow ended up on my handlebars on my ride home

Mantis

Jason Scheirer posted a photo:

Mantis

Jason Scheirer posted a photo:

Yeah Yeah Yeahs

Jason Scheirer posted a photo:

At the Pomona Fox