Twitter bot experiments

27 December 2013 | twitter, Code | 0 comments

The things one does on holiday.

I've set up two experimental twitter bots as a bit of a playground for making beacons transmitting slightly poetic, somewhat random garbage in the spirit of the UVB-76 buzzer.   Right now they are working but unremarkable.

@uc tweets a selection of two words phrases with the first word starting with u and the second starting with c.  It doss this hourly, with the words extracted on a summary of current US news. 


@d8 does something similar.  It does a frequency count of words in a summary of UK news and produces the eight most frequent words starting with the letter d.   This always ends up with death or dies or dead in it without fail.  I guess that's the way we write the news.

 

Not poetic enough

Simple frequency counting and a sort isn't enough to make these really interesting or poetic. Over time I want to mess around with text processing and some language processing to make the results a bit more exciting.

But feel free to follow them if the mood takes you.

Using a Github service hook and mod_wsgi for auto-deployment

2 August 2012 | blog, Code | 0 comments

Github has post-receive hooks that are called on each push to a repo.   And say on each push to a certain branch you'd like to automatically deploy or run other actions.  This describes how I did that with Apache, mod_wsgi and fabric.

The steps are something like this:

  1. Configure the hook in Github to fire a POST to a given URL.
  2. Configure Apache/mod_wsgi to catch the hook
  3. Add a script that runs whenever the URL is POSTED to.
  4. Check that it is a reasonable real hook from Github
  5. Check the hook is for the correct branch
  6. Run a deployment script or something
So let's go through these in detail:

1. Configure the hook

Either follow the github help on the subject, or go to Admin on your github repo, select Service Hooks, then Webhook Urls, and add a URL that you control. Let's say I put http://myexample.com/myexamplewebhook

2. Configure Apache/mod_wsgi

I'm staging a Django app, so I already have Apache and mod_wsgi set up.  I need to make sure that I'm using a WSGI daemon process so I can set the user and group.  It ends up looking something like this in the appropriate myexample file in /etc/apache2/sites-available.

WSGIDaemonProcess myexample user=ubuntu group=ubuntu display-name=%{GROUP}
WSGIProcessGroup myexample

WSGIScriptAlias /myexamplewebhook "/home/ubuntu/myexampleapp/app/posthookwsgi.py" 

So, here we've made sure that the ubuntu user and group is used for the wsgi process, so our deployment actions are done with the user and group we expect, and we've added an alias for /myexamplewebhook to run our hooks script. You should have something like the last line already, that's calling the normal Django wsgi.py.

And we have to allow Apache to access our posthookwsgi.py:

<Directory /home/ubuntu/myexampleapp/app>
<Files "posthookwsgi.py">
Order deny,allow
Allow from all
</Files>
</Directory>

Reloading/restarting apache will make that available.  At this point, pushing  to the repo should cause an error to go into your error.log.

3-6. Add the Script

Now, I need a wsgi script to call that does what I want.  Here's one I prepared earlier:

 

 

This script is a wsgi application all of itself, so you need to have the application method as it shows.  There's an example debug printing line right under the function def.

So, what this script does:

  1. Read the body of the post request, which is JSON, and convert it to a dict.
  2. Match the conditions in the conditions array. Basically here you put in the conditions that you want the hook to meet, and they will be tested.  An error response means nothing else happens. In this example, I'm checking that the ref is the branch I expect.  If any other branch is pushed, we'll not deploy.
  3. Work out the pathname for the base of our deploy
  4. Run the auto_deploy fabric task
  5. Respond OK

 Note that if anything exceptions during the script or the fabric run, we'll have a note in the error log.

Notes and thoughts:

  • This is meant for refreshing a staging server, but you could use similar for all sorts of actions.  The hook POST tends to  arrive within a few seconds of the push, so that's perfect for the use case I need, where one or more html developers can push HTML/CSS changes and wanting to see them live in seconds.
  • If you modify the script to do fancy things, beware you don't expose a security vulnerability by script injection.  The incoming JSON data "From Github" may actually come from an attacker trying to own your server.  Don't use that data in script paramters without sanitation.
  • This isn't a replacement for serious production deployment.

Bringing the Blog Home

1 August 2012 | blog | 0 comments

Posterous was acquired by Twitter. They keep telling me. As though that's going to make me happy.  Or is it a warning?

Actually, it does make me happy. Here's my newly migrated blog, made more of the raw materials I deal with daily than rarified startup hot-air OMG-you-can-blog-by-email-ness.

There will be a Colophon that talks about the details at some point. For now, a quick statement of what this is made of.  The stack is like this:

  • Amazon Route53,
  • Amazon EC2
  • Ubuntu 
  • Apache
  • mod_wsgi
  • Python
  • Django 1.4 -- MySQL
  • feincms
  • feincms-elephantblog 
  • Twitter Bootstrap

There's bit of HTML in there somewhere too, plus some Javascript tidbits.

-OO-

So, this is me bringing my blog back home to my own infrastructure. May it continue with as much delight and promise as I've experienced so far.

 

who.theskiff.org gives presence and profile info

9 February 2012 | 0 comments

As a result of the Skiff Hack, we now have Skiff member profiles and presence info available for the skiff at http://who.theskiff.org.  As in, you can see who is currently in the Skiff or who has been around recently, and a bit about them in the process.

Yes, that is another profile to maintain, but this helps.  We can find out who is around us at the Skiff on a day to day basis.  Lovely work Tom and Simon and I don't know who else made it.

My skiff profile is here.

Adding a sepia filter to django-imagekit

26 August 2011 | 0 comments

I'm rather fond of django-imagekit -- a useful django tool that organises a bunch of image processing on images saved in django models.  It comes with a bunch of useful scale, crop etc tools based on PIL. 

However, I was looking for a way to make a sepia images from my uploaded image.  I found Fredrik Lundh's PIL code for sepia in his post Sepia Toning with PIL and adapted that into the form needed for a django-imagekit image processor.  Here's how  that looks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
"""
Adds a sepia filter processor for django-imagekit which
can be used in an imagekit specification.

https://github.com/jdriscoll/django-imagekit

See http://effbot.org/zone/pil-sepia.htm for the original
code for the sepia filtering in PIL

"""
from PIL import ImageOps
from imagekit.processors import ImageProcessor


class Sepia(ImageProcessor):
    """
Turn image sepia
"""
    @classmethod
    def process(cls, img, fmt, obj):

        def make_linear_ramp(white):
            # putpalette expects [r,g,b,r,g,b,...]
            ramp = []
            r, g, b = white
            for i in range(255):
                ramp.extend((r*i/255, g*i/255, b*i/255))
            return ramp

        img = img.convert('RGB')
        sepia = make_linear_ramp((255, 240, 192))
        if img.mode != "L":
            img = img.convert("L")
        img = ImageOps.autocontrast(img)
        img.putpalette(sepia)
        img = img.convert('RGB')
        fmt = "JPEG"
        return img, fmt

 

 

 

How to remap the # (hash) key on Apple UK keyboards

3 November 2010 | 0 comments

Given how useful the # key is, it has always annoyed me that on the standard UK Apple keyboard, getting a # key requires keying Alt-3 to get it to appear. 

If you are doing web development with jquery, or writing shell scripts or python, you end up using this key a lot and Alt-3 just doesn't work for me.  However, on the left of the recent Apple keyboard, there is the § key.  I've never used §, I guess somebody must.  

Anyway, so I went and found Ukelele, a keyboard layout editor, and modified the keyboard map to put the # where the § used to be, and the result is what I call the BritishLHash keyboard layout.

How to use this:

  1. Download the file, and put it in your ~/Library/Keyboard Layouts folder.
  2. Open Language and Text in System Preferences
  3. Select the Input Sources tab
  4. Tick next to the BritshLHash keyboard.  This makes it available as an input source for you.  And also Tick Show Input in menu bar
  5. Find the little flag in the menu bar.  Pop it up and change it to BritishLHash.

That's all there is to it.  I've been using this mapped file for a year or so, and it works fine in OS X 10.5 and 10.6.

Or if you want to meddle with keymaps and propose changes, have a look at the github repo grasuth/britishlhash.

Updated: 13 July 2013:  The file on posterous had gone, so now the keylayout file is hosted on github. 

Hiding Promoted Tweets in Twitter Searches

12 October 2010 | 0 comments

I'm getting rather fond of a full mutiple-window experience when watch the Formula 1, and one of those windows is open on a search for f1 or #f1 to keep an eye on the discussion.

That works, except there is often a repeated promoted tweet that stays at the top of the search results. This confuses and annoys me, as I have to cast my eyes past this promoted tweet again and again and again to see the real search results.  Once, on the initial search, is enough, after that, these things are unnecessary noise. Like this:

Screen_shot_2010-10-12_at_16

So, I've made a Google Chrome Extension to take away promoted tweets after the first time you see them.  When there are more search items, and 'x new tweets'  shows, clicking on this renders the search results minus the promoted tweets.   You'll still see Top Tweets, though, I might take them on next.

Get the extension from here:  Hide Repeated Promoted Tweets.  Read more info here.  I hope it is useful.  And let me know what you think.

 

Unlogo.org -- excellent idea

6 October 2010 | 0 comments

Unlogo is a web service that eliminates logos and other corporate signage from videos. On a practical level, it takes back your personal media from the corporations and advertisers. On a technical level, it is a really cool combination of some brand new OpenCV and FFMPEG functionality. On a poetic level, it is a tool for focusing on what is important in the record of your life rather than the ubiquitous messages that advertisers want you to focus on.

Very nice piece of work, this. Unlogo removes logos and corporate signage from video. What a great experiment. I haven't had a look at the videos on the site yet, but I have great hopes.

One of the things that I notice particularly about the UK countryside is that there is a complete absence of billboards, and that's almost worth a sigh of relief every time there's a stretch of road with green either side and no advertising signage.

Why you don't need a new 'fridge.

15 September 2010 | 0 comments

For some years now, I've been, on occasions, having a good rave about wanting to start a (print) magazine that had neutral or balanced advertising.   I'd want a magazine that devoted equals space and effort to advertising to sell things to people, and advertisments to encourage people not to buy things.

So, on one page you'd have an ad for a new 'fridge, and on the other side, a similarly high-quality ad encouraging you to maintain and keep the old one or share the neighbours.  There are two sides of every buying decision, we just don't get to see them very often.  And when we are trying to save carbon, not buying appliances does help reduce the CO2 in the atmosphere.

So, you get the idea:  A nicely designed magazine with rules that mean that advertisers need to somehow ensure or allow or fund the for and against cases for their product.  Would anybody care to advertise? Would anybody dare to advertise?  It'd be an interesting experiment.

It would be mad to start a new print magazine right now.  So, what about something online -- and this is the new bit of my thinking.  Presumably, once can buy banner ads with the keyword 'fridge' or whatever and promote not buying a new fridge.  It's be easy enough to make ads like that, and they'd get nicely delivered onto pages where people were looking to buy fridges or were searching out a new 'fridge to buy.  Perhaps the click through rates wouldn't be that high, so it wouldn't cost too much, but who knows?

This has to be against ad placement terms and conditions, you think?  Well, I've just had a look at the google adwords policies, and I can't see anything in there that make it wrong to suggest not buying something might be a good option. Actually, under the 'Anti' and Violence section of the Content Guidelines, there is a specific allowance for "Consumer awareness ads (positive and negative) that don't violate the basics of the policy".   That sounds ok for now, but there's a lot of details to work on.

Okay, so, to make this thing fly, we need:

  • funding for some ads
  • destination website(s) with the reasons for all of this, or useful consumer information on 'sharing the neighbours fridge' or 'how to survive without a car' or similar.

And that probably means a charity or consumer organisation that wants to try it out. An initial experiment would be very interesting because it would yield click-through information about how consumers react to these messages, and could give a charity or similar a quite high profile.

Anybody interesting in trying this out?