Convert XML to JSON in Clojure

Mon, 26 Dec 2016 00:00:00 +0000

Mainly as an exercise in recursion and because I needed something quick I decided to implement my own XML/JSON converter. Here follow the code with the explanation.

Parse a XML string

The standard clojure.xml/parse is excellent for parsing, as explained in xml for fun and profit it can be done with this simple one-liner:

(defn parse [s] (clojure.xml/parse (java.io.ByteArrayInputStream. (.getBytes s))))

However clojure.xml/parse returns a clojure hash with the following structure:

:attrs contains the attributes
:content contains the nested structure of the XML
:tag is the name of the XML tag

For example, if we have the following XML:

<hello>bravo</hello>

The result of calling (parse "<hello>bravo</hello>") would be:

{:attrs nil, :content ["bravo"], :tag :hello}

Instead I wanted to produce the following output:

{:hello "bravo"}

After that, it’s very easy to convert the previous Clojure map in JSON using cheshire

The base case

When writing a recursive function, it’s good practise to start with the base case: ie. when the function is defined for well known cases. If a recursive function operates on numbers the typical base case would be a… yes, just a number. If the function operates on lists, the base case is typically the empty list. Otherwise if the function return hashes (or maps or Objects if you program in Javascript) the common base case is the empty object.

We’ll define function called xml->json and consider the following base cases:

(defn xml->json [element]
      (cond
        (nil? element) nil
        (string? element) element
        (and (map? element) (empty? element)) {}
        :else nil))

Recursion

The simplest XML to convert is <hello>bravo</hello>.

(def simple-input (parse "<hello>bravo</hello>"))
    ;{:attrs nil, :content ["bravo"], :tag :hello}

Let’s add recursive condition to support that input

(defn xml->json [element]
      (cond
        (nil? element) nil
        (string? element) element
        (and (map? element) (empty? element)) {}
        (map? element) {(:tag element) (xml->json  (:content element))}
        :else nil))

The previous code says: If the element is a map then it returns a new map with key equals to (:tag element) and the value equals to xml->json applied to the (:content element).

If we try to run the previous code, we will get:

(xml->json simple-input)
    ;{:hello nil}

We get back a nil value because (:content element) is a list of elements, and we don’t have a case defined for that (the code reach the :else condition). Let’s fix this problem, adding a condition when there is a list of elements:

(defn xml->json [element]
      (cond
        (nil? element) nil
        (string? element) element
        (and (map? element) (empty? element)) {}
        (sequential? element) (map xml->json element)
        (map? element) {(:tag element) (xml->json  (:content element))}
        :else nil))

The previous code (sequential? element) (map (partial xml->json) element) means:

If the element is a sequence, we want to apply xml->json to every items.

If we try again to run the modified version, we will see that:

(xml->json simple-input)
    ; {:hello ("bravo")}

That’s better, we eventually get the value "bravo", but it is not right yet. In JSON we want to get this:

{"hello": "bravo"}

The previous code instead returns the following JSON:

{"hello": ["bravo"]}

Note: We actually get a clojure map that would be converted into JSON using Cheshire.

It seems that the code (at it stands) works well for those element having a list of nested sub-elements. Just for testing purposes, we can try to run the following:

(def simple-nested (parse "<div><h1>hello</h1><h1>ciao</h1></div>"))
    (xml->json simple-nested)
    ; {:div ({:h1 ("hello")} {:h1 ("ciao")})}

The map is actually working, but we still have a problem there. If the sequence has only one element we do not want to return the list, instead we want to return the element itself.

Let’s fix the code again, considering that:

(defn xml->json [element]
      (cond
        (nil? element) nil
        (string? element) element
        (and (map? element) (empty? element)) {}
        (sequential? element) (if (> (count element) 1)
                        (map xml->json element)
                        (xml->json  (first element)))
        (map? element) {(:tag element) (xml->json  (:content element))}
        :else nil))

The previous code does the following:

Assuming the element is a sequence, if there are more than one items then we apply the map to every items, otherwise we call xml->json on the first item.

If we try now to run the previous code, it will produce the following output:

(xml->json simple-nested)
    ; {:div ({:h1 "hello"} {:h1 "ciao"})}

Special case: Array or Object?

There are few more special cases to consider. For example the following XML:

<div>
    <h1> hello </h1>
    <h1> ciao </h1>
</div>

would be converted into this JSON

{
    "div": [
        {"h1": "hello"},
        {"h1": "ciao"}
    ]
}

It’s an Array of Objects! Instead, this XML:

<person>
    <fullname> Salvatore </fullname>
    <address> Catania </address>
</person>

It’s better represented by the following JSON:

{
    "person": {
        "fullname": "Salvatore",
        "address": "Catania"
    }
}

Instead, the function xml->json would produce:

{
    "person": [
        {"fullname": "Salvatore"},
        {"address": "Catania"}
    ]
}

Array of Objects! Although both XML contains a sequence of inner elements, the JSON should be different. Why?

It’s mostly personal taste, both JSON representations are valid, but I think that’s easier to have an Object instead of an Array. For example, it’s easier to access the property fullname like this:

// Javascript code ahead!
var data = {
    "person": {
        "fullname": "Salvatore",
        "address": "Catania"
    }
}
console.log(data.person.fullname); // We can easily access the fullname property

If we would have instead an array, it won’t be possible to use the dot notation.

Having explained that, I’ll make the following arbitrary decision:

If the list of elements contains the same tag (a list of <h1> for example) then it’s fine to produce JSON Array. Otherwise, if the list of elements contains different tags (like in the previous example, we had <fullname> and <address>) then we produce an JSON Object.

Let’s defined a helper function that checks if a sequence of elements are different:

(defn different-keys? [content]
      (when content
        (let [dkeys (count (filter identity (distinct (map :tag content))))
              n (count content)]
          (= dkeys n))))

Now, we can use the different-keys? in our recursive xml->json:

(defn xml->json [element]
      (cond
        (nil? element) nil
        (string? element) element
        (and (map? element) (empty? element)) {}
        (sequential? element) (if (> (count element) 1)
                                  (if (different-keys? element)
                                    (reduce into {} (map (partial xml->json ) element))
                                    (map xml->json element))
                                  (xml->json  (first element)))
        (map? element) {(:tag element) (xml->json  (:content element))}
        :else nil))

The following line (reduce into {} (map (partial xml->json ) element)) means:

Apply xml->json to every items, and reduce the list into a hash. In other words, the function map returns a list that’s why we convert it back into a hash.

Finally we have a xml->json that works for most the cases, and parses into Array or Object depending on the tags.

Are we there yet?

Not yet! There’s one more thing: XML attributes! We want the ability to support attributes, for example the following XML:

<person gender="female">
    <name> Carmela </name>
    <address> Siracusa </address>
</person>

The attribute gender needs to be converted into a meaningful JSON object. Again, the following decision is arbitrary, and surely can be implemented in other ways. I wanted to group attributes together inside one JSON keyword (other implementations might want to use the “@” chars for example):

The final code is:

(defn xml->json [element]
  (cond
    (nil? element) nil
    (string? element) element
    (sequential? element) (if (> (count element) 1)
                           (if (different-keys? element)
                             (reduce into {} (map (partial xml->json ) element))
                             (map xml->json element))
                           (xml->json  (first element)))
    (and (map? element) (empty? element)) {}
    (map? element) (if (:attrs element)
                    {(:tag element) (xml->json (:content element))
                     (keyword (str (name (:tag element)) "Attrs")) (:attrs element)}
                    {(:tag element) (xml->json  (:content element))})
    :else nil))

This line of code (keyword (str (name (:tag element)) "Attrs")) (:attrs element) means:

Convert the value of (:tag element) into another keyword with a postfixed “Attrs”, the (:attrs element) will follow.

Conclusion

XML to JSON is a very common procedure. Clojure core support xml via clojure.xml, and the de-facto method to query/access and manipulate XML is using zippers.

I needed a simpler way to just transform XML into JSON, and came up with a recursive and simple solution. I don’t think it’s optimised to handle large XML, but it shouldn’t be that bad either.

Feel free to use the code: https://github.com/rosario/json-parse.

Tartu: an iOS game in Swift/SpriteKit

Sat, 24 Dec 2016 00:00:00 +0000

It’s now time to clean up some of my old projects and publish this little iOS game I made last year. I had two main objective: 1) Make some money and 2) Learn the basic of Swift.

Here’s a description of the main parts and some explanation how to glue things together when developing a mobile game of iOS.

Tartu is getting ready to start. Candies…

Why learning Swift

It’s well explained by Martin Fowler on his blog:

For many developers, the one-language notion is a sign of lack of professionalism. This is best exemplified by the Pragmatic Programmers’ advice to learn a new language every year. The point here is that programming languages do affect the way you think about programming, and learning new languages can do a lot to help you think about solving problems in different ways. (It’s important to learn languages that are quite different in order to get the benefit of this. Java and C# are too similar to count.)

Since I knew nothing of iOS programming I thought that making a game would be more fun that programming yet another calculator. This online iOS Swift course from Stanford University is a must (apart from the calculator).

Obviously, I also wanted to get rich as game developer, and I was quite surprised I even made money with it. To be precise I made $0.01 (and counting…) in 1 year.

The game mechanics is simple: there’s a turtle riding a unycicle collecting candys and trying to avoid spikes. It’s a typical one tap scroller, but complex enough to learn several aspect of iOS/Swift programming.

Code overview

Here’s a brief description of the things I’ve learnt during the development of this iOS game. I’d recommed to follow some good gamedev videos here: CartoonSmart. (Note: I am not affiliated and I genuinely think they have good videos).

Optional Type in Swift

Before looking at the Game, a quick comment on Swift syntax. Some types have a postfix ? char. That’s just syntactic sugar for Optionals Type.

An quick example is :

    var str:String? = "hello"

    if let a = str {
        print(str)
    } else {
        print("not defined!")
    }

In Scala, we could do something similar with:

    val str:Option[String] = Some("hello")

    str match {
        case Some(value) => print(value)
        case _ => print("not defined")
    }

The GameViewController

The GameViewController is the class the initialise the game:

class GameViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()
        // Configure the view.
        let skView = self.view as! SKView

        skView.showsFPS = true
        skView.showsNodeCount = true
        if #available(iOS 8.0, *) {
            skView.showsPhysics = true
        } else {
            // Fallback on earlier versions
        }

        /* Sprite Kit applies additional optimizations to improve rendering performance */
        skView.ignoresSiblingOrder = true

        let scene = GameScene()
        scene.scaleMode = SKSceneScaleMode.ResizeFill
        scene.size = skView.bounds.size

        if scene.size.width < 480 {
            scene.size.width = 480

        } else if scene.size.height < 320 {
            scene.size.height = 320
        }
        skView.presentScene(scene)
    }

The GameScene is loaded and the presented to the screen. While debugging is useful to display FPS and physics borders (for detecting problems with sprite collisions).

The GameScene loads the background, the sprites, and background layers. The game state is saved in the global variable currentGameState, and it represents the possible states of the game (Active, Death, Completed).

class GameScene: SKScene, SKPhysicsContactDelegate {

    var currentLevel = 1
    let layerBackgroundSlow = LayerBackground()
    let layerBackgroundFast = LayerBackground()
    let layerGameWorld = LayerWorld(
        tileSize: CGSize(width: 20, height: 20),
        gridSize: CGSize(width: 200, height: 15))

    let player = Player(size: CGSize(width: 30 , height: 45))
    var currentGameState = GameState.Intro
    var lastUpdateTime:NSTimeInterval = 0
    var dt:NSTimeInterval = 0
    var playerStartPoint:CGPoint?

    override func didMoveToView(view: SKView) {
        physicsWorld.contactDelegate = self
        physicsWorld.gravity = CGVector(dx: 0, dy: -7)
        anchorPoint = CGPoint(x: 0.5, y: 0.5)

        if let level = settings.levelData(currentLevel) {
            addSun()
            addSea(images: level.sea)
            addSky(color: level.skyColor)
            addBackground(parallax: level.parallax)
            addGameWorld(tmxfile: level.tmxfile)

            if currentGameState == .Intro {
                displayTapToStart()
            } else {
                addPlayer()
            }
        }
    }

The Game loop is the main procedure that continuosly run and update the game status. It’s also responsible of listening to events and firing the correspondent actions. When the game is Active, the player position and the backgrounds layer are updated. We also check if the player collided with a spike or if the player reached the end of the level.

    override func update(currentTime: CFTimeInterval) {
        // Delta Time
        if lastUpdateTime > 0 {
            dt = currentTime - lastUpdateTime
        } else {
            dt = 0
        }
        lastUpdateTime = currentTime

        //Update Game
        switch currentGameState {
        case .Active:
            layerBackgroundSlow.update(dt, affectAllNodes: true, parallax: true)
            layerBackgroundFast.update(dt, affectAllNodes: true, parallax: true)
            player.update(dt)

            if player.spikeTouched {
                currentGameState = .Death
            } else if player.completed {
                currentGameState = .Completed
            }

        case .Death:
            player.deathAnimation()
            displayPostScreen()
            currentGameState = .Paused
        case .Completed:
            player.happyAnimation()
            displayNextScreen()
            currentGameState = .Paused
        default:
            break
        }
    }

Player and Collisions

Collisions represent how the player interacts with the scene. For example touching a spike or collecting a candy.

The player is represented with a class, and the collision part is easily managed by SpriteKit Physics:

    func collidedWith(body:SKPhysicsBody, contact:SKPhysicsContact) {
        if body.categoryBitMask == ColliderType.Wall {
            currentState = State.OnGround
            playGroundSound()
        } else if body.categoryBitMask == ColliderType.Spike {
            spikeTouched = true
            playSpikeSound()
        } else if body.categoryBitMask == ColliderType.Edge {
            spikeTouched = true
            playWaterSound()
        } else if body.categoryBitMask == ColliderType.Trigger {
            completed = true
        }
    }

In this example, two bodies collide and we use bitmask to check the possibile combinations.

Sequence of Animations in SpriteKit

While the player is happily jumping and avoiding spikes, there are few animations in the background. The sun has a smooth shine effect, and two backgrounds are scrolling at different speed with a simple parallax effect.

Sun is shining! The Sun animation is made of four animations running in sequence

Animations are quite easy to implement in SpriteKit, here’s an example of how to draw the shining sun:

    func runAnimation(){

        let scaleUp = SKAction.scaleBy(1.2, duration: 1)
        let scaleDown = SKAction.reversedAction(scaleUp)()
        let fadeOut = SKAction.fadeOutWithDuration(1)
        let fadeIn = SKAction.reversedAction(fadeOut)()

        let innerBorder = circleOfRadius(size.width/2)
        innerBorder.strokeColor = SKColor.whiteColor()
        innerBorder.lineWidth = 5
        addChild(innerBorder)

        let outBorder = circleOfRadius(size.width/2 + size.width*0.1)
        outBorder.fillColor = SKColor.whiteColor()
        outBorder.alpha = 0.2
        addChild(outBorder)

        let outBorderAnim = SKAction.sequence([scaleUp, fadeOut, scaleDown])

        let explosion = SKAction.runBlock({
            outBorder.runAction(outBorderAnim)
            outBorder.alpha = 0.2
        })
        let wait = SKAction.waitForDuration(5)

        let innerBorderAnim = SKAction.sequence([
            scaleUp,
            SKAction.group([fadeOut, explosion,wait]),
            scaleDown,
            fadeIn
            ])

        innerBorder.runAction(SKAction.repeatActionForever(innerBorderAnim))
    }

There are four animations: to scale the sun Up and Down and to fade the shine In and Out. The animation is made of two sequences, the outBorderAnim and the innerBorderAnim. Chaining of sequences is donw with SKAction.sequence, and if we want to repeat the sequence SKAction.repeatActionForever.

Levels construction

Levels are: (drumrolls) just XML. A Tilemap editor is used to position each tile. I used Tiled for that. After a level is designed it’s necessary to parse the XML. I was using JSTileMap to read the XML (although it’s in Objective-C), and ad-hoc Swift code to create the Sprites.

class LayerWorld:LayerTiles {

    var tileMap:JSTileMap? = nil

    func addTileMap(filename:String) {
        tileMap = JSTileMap(named: filename)
        if tileMap != nil {

            tileMap!.name = "tileMap"
            let mapBounds = tileMap!.calculateAccumulatedFrame()

            addGroup("walls", tilemap: tileMap!) { wall in
                if let point = self.objectPoint(wall), size = self.objectSize(wall){
                    self.tileMap!.addChild(Wall(offset: point, size: size))
                }
            }

            addGroup("spikes", tilemap: tileMap!) { spike in
                if let point = self.objectPoint(spike), size = self.objectSize(spike){
                    self.tileMap!.addChild(Spike(offset: point, size: size))
                }
            }

            addGroup("triggers", tilemap:  tileMap!) {trigger in
                if let point = self.objectPoint(trigger), polyline = self.objectPolyline(trigger){
                        if let name = self.objectName(trigger) {
                            if name == "completed" {
                                self.tileMap!.addChild(
                                    Trigger(offset: point, polyline: polyline, name:name))
                            }
                        } else {
                            self.tileMap!.addChild(Edge(offset: point, polyline: polyline))
                        }
                }
            }
            addChild(tileMap!)
        }
    }
}

Winter is coming

Conclusion

Swift is a fine language with interesting features. I guess I still need to build a calculator though and follow the Stanford Course. Many thanks to Neil North for his online course. Some of the code of Tartu is taken from his excellent tutorials.

The entire code of the game is published on my github. Sounds effect were freely downloaded from… somewhere, I can’t remember (if you recognise your sound get in touch and I’ll add a link here). Graphics, background and tiles were made by myself, so feel free to use them.

I’ve been careful to place any copyright section when required, but if I’ve forgot please get in touch and I’ll fix it.

Should tech recruiters learn to code?

Tue, 07 Jan 2014 00:00:00 +0000

It’s time to up your game tech recruiters. Learn something about programming, at least the very basic terminology and get to talk to developers in a professional way.

Tech recruiters + coding skills?

I run coding workshops in London, teaching people how to code simple websites from scratch. I’ve been trying to find a niche, a group of people that could benefit from knowing the very basics of web development. It’s 2014, surely people working in tech-recruitment agencies should know the difference between HTML and CSS.

I thought that tech recruiters might be interested so I decided to get in touch with them. Properly talking to them on the phone. Here’s how it went.

Hello, I’m a developer

It would have been too easy to send emails. Instead, it was going to be more fun to actually cold calling them on the phone, their preferred way of communication!.

Ouch. It didn’t go well. But it was a fun experience. I would always ask the question:

You are currently recruiting web developers. I can teach you the difference between HTML and CSS? Do you think it could help you?

No. No thanks. No, sorry. No no. No, but… are you currently looking for a job? Ah! No thanks. What? NO! No, no no. Not interested, thanks for the call. No. No.

There’s hope!

I must to improve my marketing skills. Most of the time, the people answering the phone thought I wanted to hire. I adjusted my pitch saying right in the beginning that I wasn’t looking to hire anyone, although it didn’t help much!

Hopefully I’ll be able to convince some tech recruiters to come and attend the coding workshops, sooner or later they’ll also need to up their game, and knowing at least something about software development won’t cause any harm.

Care to help?

If you’d like to help, let your favourite tech recruiters in London know about these coding workshops.

Caffè e Startup

Sat, 04 Jan 2014 00:00:00 +0000

Parliamo di caffè e startups. L’Inghilterra scopre il piacere di bere un buon caffè. Non solo, da bravi inglesi pensano anche a farne un business. Ci sono infatti almeno tre startup che iniziano a vendere caffè online.

Se facciamo una passegiata nella zona East di Londra, possiamo trovare sempre più numerosi dei caffè a gestione ‘locale’. Fino a pochi anni fa, gli unici posti dove bere un caffè erano i vari Costa Coffe, Starbucks, e Caffe Nero. Lo chiamano coffee, ma a dirla tutta è un miscuglio nero e bruciacchiato, mischiato con mezzo litro di latte riscaldato fino al punto di ebollizione.

Durante gli ultimi cinque anni invece, l’Inghilterra e specialmente Londra, scopre che cosa significa bere un buon caffè.

Image credits to Tazzina in Cucina

Fresh, better tasting coffee.

Ci sono almeno tre startup, che forti di questa nuova moda, stanno investendo tempo e soldi, per portare a casa degli inglesi un buon caffè macinato fresco, che sappia di caffè. Le startup in questione sono Pact Coffee, Kopi e HasBean.

Vendere caffè, nel paese del tè.

Non li ho ancora provati, ma sono molto curioso di provarli. È ancora troppo presto per abbandonare il caffè Lavazza e Torrisi, ma c’è qualcosa da ammirare in queste idea: la capacità di andare contro i luoghi comuni, e lo spirito l’iniziativa.

Dispiace solo una cosa. Starbucks, Caffe Nero e Costa Coffee sono tutti in mano a stranieri. Peccato che come italiani, stiamo perdendo anche il treno del “miglior caffè, macinato fresco” venduto online.

Pasqua Rosée, Ragusana

La prima coffee house di Londra è stata aperta da Pasqua Rosée nel 1652. Pasqua era di Ragusa, in Sicilia.

Kasper - A theme for Jekyll

Sun, 10 Nov 2013 00:00:00 +0000

Ghost is a very interesting blogging platform written in Node.js and it features a good, simple and modern theme called Casper. I really like Casper and I wanted to port it to Jekyll. The result is Kasper.

Here’s a list of what Kasper currently includes:

Pagination
Rss
Google Analytics Tracking code
Code Syntax Highlight
Author’s profile with picture
Disqus comments

Screenshots

Here you can find a couple of screenshots.

Feel free to fork the repository, the source code is on github.

London Hackers Guide

Wed, 09 Oct 2013 00:00:00 +0000

A short guide for all the hackers willing to live in London.

Image credits to Nietnagel

Living in London

Most of tech companies and startups are based in East London (Shoreditch and around Old Street - the Silicon Roundabout). If you’re a meetup/hackhaton/conferences kind of person that’s where you want to live. You might also choose to live in Hoxton or Haggerston, as long as you’re close to the Overground it’s easy to commute.

West London can be cheaper than East nowdays. The Central line is usually fast but busy during office hours. A typical commute time from West London to East London could take up to 30 mins. Use the Transports for London or City Mapper to find out more about commuting time.

East London: Shoreditch, Bethnal Green, Dalston Kingsland, Hackney - There are tons of bars and clubs in the area. If you work around the “Silicon Roundabout” it could be the most obvious choice to live there. Note that it could be noisy in certain streets (lots of partygoers).

West London: Hammersmith and Fullham, Shepherd’s Bush, Notthing Hill - West London tends to be slightly cheaper than East London and more residential (basically there are not enough cool pubs/bars/cafe!).

South London - Brixton, Southwark and Borough - The area has lots to offer in terms of cafes, restaurants and night life. There are also great markets (Brixton Market and Borough Market).

Hacker map: If you’re interested in London’s Gentrification.

Find accomodation in London

Finding good accomodation in London can be stressful. Keep it mind it’s not an easy task! Here’s where to start looking:

Open Rent (London Startup!)
Zoopla
Right Move

Note: Some online offers might be already gone before you can even click on them! Most of the offers will be from Letting Agents.

Hacker approach:

Use the websites above to find out the price of the area.
Note of the name of the letting agents covering that area.
Go to the Agency in person. You will find out about new properties before they’ll start advertising online.
Double check if the aparment you’ve found is still available (phone the agency).
Be patient. It could take a couple of weeks before you find something nice.
Be fast. If you like it, tell the agents you want to take it. Note: typically you’ll have to give a deposit.
Pro tips If you set an appointment with an agent, call them on the phone a couple of hours before the meeting. Letting agents tend to forget to call you back if the property is already gone. You’ll spare a trip and save time.

I haven’t dealt with private landlords that much, I think it’s a bit more risky (it depends on the landlord) but if you feel brave you can always have a look at Gumtree (it’s the local CraigsList) and you might actually find a good deal.

Accelerators / Incubators

If you’re coming to London because you want to start a business that’s a great time for being in the City. Here’s a list of most important accelerators and incubators in London:

Meetups / Events

There are so many events happening in London that’s really hard to keep track of all of them. Meetup is a good place to start looking, as well as EventBrite. There’s also Attending.io (london startup!) that lists some events happening in London.

A list of recommended events and meetups:

For a longer list, check the London Tech Event Collection

##Coworking Spaces

If you want to work as a freelancer, there are plenty of co-working spaces around the city. The advantage of paying for a desk is the possibility to meet and chat with people working in your area and you might even get to know about new paid gigs.

Looking for Work

Hacker Jobs A good recruiter free job board run by Steve (Peroni on HN)
Work in Startups Mainly startup based (it has a co-founders section).
3 Beards Jobs Startups, digital agencies and established companies advertising their openings.
Silicon Milk Roudabout A really big job fair in London.

Hacker approach: Job boards are ok. A better approach is to join some developers mailing lists. Look for your favourite programming language meetup and group. Searching for “[language] User Group London” on Google will do the trick.

You might also be proactive and check yourself the available openings. It requires a bit of manual work (you’ll have to find the companies and their hiring/jobs page). Few useful websites:

After work drinks

Burger Night: Organized by Rob Fitzpatrick, it’s an informal and friendly meeting for people working on startups.

Silicon Drinkabout: Organized by 3 Beards, it’s a Friday after work meeting in East London from around 6-7pm onwards, at a different venue each week.

Do you have any suggestions?

I’m sure I’ve missed some event, incubator or something important to write (also I don’t know North London!).

If you’d like to suggest a link, just leave it in the comments here or join the discussion on hacker news.

Chatting on Last.fm using websockets

Mon, 29 Oct 2012 00:00:00 +0000

Adding a Chat box on Last.fm is easier than you think. Here’s a quick hack.

The hack

As shown before, it’s relatively easy to embed a chat box on twitter. Doing the same for last.fm is trivial. First, we need to add omniauth_lastfm in the Gemfile of our Sinatra app.

    gem 'omniauth-lastfm'

We need to add also the require in the app file:

    require 'omniauth-lastfm'

Now we have to define the OAuth callback for Lastfm:

  get '/auth/lastfm/callback' do
    auth = request.env["omniauth.auth"]

    user = User.first_or_create({ :uid => auth["uid"]}, {
      :uid => auth["uid"],
      :name => auth["info"]["name"],
      :nickname => auth["info"]["name"],
      :created_at => Time.now })

    session[:user_id] = user.id # User logged in
    redirect 'http://last.fm'
  end

Chrome Extension

We add a Chrome Extension that automatically loads up the javascript and opens the chat box. The manifest.json is:

    {
       "content_scripts": [ {
          "js": [ "qop.js" ],
          "matches": [ "http://*.last.fm/*", "https://*.last.fm/*" ],
          "run_at": "document_end"
       } ],
       "description": "Last.fm Chat",
       "name": "qop.im",
       "version": "1.0",
       "manifest_version": 2
    }

And the qop.js is:

    if (window.top === window) {
        (function(){
            link = document.createElement('link');
            link.href = 'http://YOURSERVER/application.css';
            link.type = 'text/css';
            link.rel = 'stylesheet';
            document.getElementsByTagName('head')[0].appendChild(link);
            var script = document.createElement('script');
            script.type = 'text/javascript';
            script.async = true;
            script.src = 'http://YOURSERVER/application.js';
            document.getElementsByTagName('head')[0].appendChild(script);})();
    }

That’s all. Give it a spin if you’d like, but remember that it’s alpha code. Here’s the Chrome extension: Chat on Last.fm

Lettera aperta agli studenti siciliani

Mon, 02 Jul 2012 00:00:00 +0000

Non andate a lavorare. Dopo la laurea, studiate quello che vi serve per far partire una startup.

Mi chiamo Rosario, e siciliano sono. Mi sono laureato in Informatica a Catania ma vivo da tempo lontano dalla Sicilia. Lavoro come freelancer, la moderna versione del mercenario informatico. Non ho un capo, ma clienti. Non ho un orario di ufficio stabilito, spesso dalle 9 alle 6, ma lavoro la domenica se posso, e rimango a casa il lunedi se ho voglia. Non vivo sempre nello stesso posto, sono nomade per scelta.

Sapete perché vi scrivo? Perché andare a lavorare subito dopo la laurea non è la scelta giusta. Se qualcuno vi offre un lavoro, non accettatelo.

Studiate

Studiate invece! Condividete il codice su github, imparate a usare heroku, pusher, socket.io, websockets, html5, css3, responsive web design, mediaqueries, ruby on rails, node.js, coffescript, backbone.js, redis, mongodb, couchdb, scala, solr, lucene.

Come programmatori abbiamo un vantaggio enorme rispetto ad altri mestieri. Possiamo creare quello che immaginiamo.

Parlate col vostro collega, convincetelo a non lavorare, almeno per un altro anno studiate quelle tecnologie che rendono USA e UK le nazioni che creano i vari facebook, twitter, instagram, foursquare, airbnb, hipmunk ecc ecc.

Perché una startup? Perché potete essere padroni delle vostre scelte, perché siete stati educati per diventare creatori di tecnologia non forza da lavoro da cubicolo.

Fate un favore a voi stessi, non andate a lavorare subito dopo la laurea, fate partire una startup, la vostra startup.

Call Forwarding with Twilio and Sinatra

Tue, 05 Jun 2012 00:00:00 +0000

Say you want to buy a virtual phone number, and you’d like to redirect calls to your mobile number. There could be different reasons why you’d want to have another phone number, without having to carry a physical phone.

A typical case is when you submit your CV to websites. Instead of giving away your mobile number, you could buy a number from Twilio, create a micro application hosted for free on heroku, and let Twilio redirect those calls to your mobile number.

You talkin’ to ME ?!?

Twilio uses TWIML, a markup language that describes operations like DIAL or REJECT.

When somebody calls our virtual phone number, Twilio will make a POST request to our server. We return some TWIML instructing Twilio to forward that call to our personal (mobile or landline) phone:

      post '/:twilio_number' do
        if @account and @account.forward_number
          # Dial phone number
          builder do |xml|
            xml.instruct!
            xml.Response do
                xml.Dial @account.forward_number,
                  :action =>"/call_status/#{params[:twilio_number]}"
            end
          end
        else
          # Reject the call, the number is not active
          builder do |xml|
            xml.Response do
              xml.Reject
            end
          end
        end
      end

Note, this POST request is made by Twilio to our server. We need to configure our Twilio Account and associate an URL with our virtual phone number.

The following action is executed when the phone hangs up, we just instruct Twilio to close the call:

      post '/call_status/:twilio_number' do
        builder do |xml|
          xml.instruct!
          xml.Response do
            xml.Reject
          end
        end
      end

Logging

We can use a database to log calls too, and it’s quite easy to setup with Datamapper and Sinatra. Here’s a simple database model:

      class Account
        include DataMapper::Resource
        property :id,             Serial
        property :email,          String
        property :twilio_number,  String
        property :forward_number, String
        property :created_at,     DateTime
        has n, :calls
      end

      class Call
        include DataMapper::Resource
        property :id,                 Serial
        property :account_sid,        String
        property :to_zip,             String
        property :to_city,            String
        property :to_state,           String
        property :to_country,         String
        property :called_zip,         String
        property :called_city,        String
        property :called_country,     String
        property :called_state,       String
        property :call_status,        String
        property :call_sid,           String
        property :called,             String
        property :from,               String
        property :to,                 String
        property :caller,             String
        property :dial_call_duration, Integer, :default => 0
        property :dial_call_status,   String
        property :dial_call_sid,      String
        property :direction,          String
        property :api_version,        String
        belongs_to :account
      end

Twilio provides the details of who’s calling, from where and other useful infos. We can grab this informations for each request, with a before filter in Sinatra:

      before do
        if params["To"] and params["AccountSid"] and params["CallStatus"]
          call = Call.new({
            :account_sid =>   params["AccountSid"],
            :call_status =>   params["CallStatus"],
            :to_zip =>        params["ToZip"],
            :to_city =>       params["ToCity"],
            :to_state =>      params["ToState"],
            :called =>        params["Called"],
            :to =>            params["To"],
            :to_country =>    params["ToCountry"],
            :called_zip =>    params["CalledZip"],
            :direction =>     params["Direction"],
            :api_version =>   params["ApiVersion"],
            :caller =>        params["Caller"],
            :called_city =>   params["CalledCity"],
            :called_country =>      params["CalledCountry"],
            :dial_call_duration =>  params["DialCallDuration"].to_i || 0,
            :dial_call_status =>    params["DialCallStatus"] || "",
            :dial_call_sid =>       params["DialCallSid"] || "",
            :call_sid =>      params["CallSid"],
            :called_state =>  params["CalledState"],
            :from =>          params["From"]
          })
          account = Account.first(:twilio_number=>call.to)
          account.calls << call
          account.save
          call.save
          @account = account
        end
      end

Conclusion

It’s easy, really easy to setup a simple call forwarding service with Twilio. When we get tired of receiving calls we can always disable the virtual phone number, hassle free.

Source code can be found here: callforwarding

HTML5 Video Streaming over WebSockets using Pusher

Mon, 04 Jun 2012 00:00:00 +0000

Using the new getUserMedia it’s possible to send a stream of images over websockets. An experiment with Node.js and Pusher Pipe.

Overview

There’s a new feature in the latest Chrome browser’s (Chrome 18 and 19, and Canary 21) called getUserMedia. It’s still an experimental feature and it’s hidden behind chrome://flags

Enable Media Source API on <video> elements. Enable experimental Media Source API on the video elements. This API allows JavaScript to send media data directly to a video element.

It’s a trivial task to grab a screenshot using video and canvas, and it’s also very easy to send this image over websockets using pusher.

Express.js and Pusher Pipe

We’ll use Express.js and Pusher Pipe. A quick note on pusher pipe, instead of sending requests using a typical REST API, using the Pipe we can enable our Node server and the browsers to communicate with Pusher services using websockets.

    <div class="center">
      <img src="/images/pusher-logo.png" alt="pusher logo">
    </div>

The following code is written in Coffescript. We tell our server to render two pages, stream is used to send a stream of images, and mirror is used to receive the images and display them:

    # Express.js. Serving /stream and /mirror
    express = require('express')

    app = express.createServer(express.logger())
    app.set('view engine', 'jade');
    app.use('/js', express.static(__dirname + '/js'));

    port = 9000 || process.env.PORT || 3000;
    app.listen port, ->
      console.log("Listening on " + port);

    app.get '/stream', (req, res) ->
      res.render('stream');

    app.get '/mirror', (req, res) ->
      res.render('mirror');

Now we connect to the Pipe. The following code sends to the channel mirror the data received on the channel stream

    # Pusher Pipe. Streaming data over WebSockets
    Pipe = require('pusher-pipe')
    client = Pipe.createClient {
      key: 'your key'
      secret: 'your secret'
      app_id: 'your app id'
      debug: true
    }

    client.subscribe ['socket_message']
    client.channel('stream').on 'event:frame', (socketId, data) ->
      client.channel('mirror').trigger 'frame', {message: 'Sending frame', dataUrl: data.dataUrl}

    client.connect()

Note that the urls /stream and /mirror are served with Express, and they respond to the usual http GET method. The channels ‘stream’ and ‘mirror’ instead are opened with Pusher.

Magic Mirror on the Wall…

Let’s have a look first at the page receiving the stream of images. The user hits /mirror, the server renders this jade template:

      h1 Mirror

      img(id="screenshot", src="")
      canvas(id="screenshot-canvas", style="display:none;")
      script(src="js/player.js")

We use the class Player to connect with the websocket and update the <img> with a stream of data:

      document.addEventListener('DOMContentLoaded', ->
        new Player();
      , false);

      class Player
        constructor: ->
          @img = document.querySelector('#screenshot')
          Pusher.host = "ws.darling.pusher.com"
          Pusher.log = (message) ->
            if (window.console && window.console.log)
              window.console.log(message)

          @pusher = new Pusher('YOURKEY')

          @mirror = @pusher.subscribe 'mirror'
          @mirror.bind 'frame', (data) =>
            @img.src = data.dataUrl;

The idea is to update the src of an img tag with the data sent over websockets, and it’s simply done by binding the @mirror channel to the frame event.

Meanwhile our Hero…

The /stream page contains the following template:

        h1 Video

        video(autoplay)
        img(id="screenshot", src="")
        canvas(style="display:none;")

        button(id="start") Capture
        button(id="stop") Stop

        script(src="js/stream.js")

We use two buttons, one to start the streaming, and one to stop it. The coffescript code is the following:

      document.addEventListener('DOMContentLoaded', ->
        new VideoStream();
      , false);

      class VideoStream
        constructor: ->
          @video = document.querySelector('video')
          @video.addEventListener('click', @snapshot, false)

          @button = document.querySelector('#start')
          @button.addEventListener('click', @snapshot, false)

          @stop = document.querySelector('#stop')
          @stop.addEventListener('click', @stopRefresh, false)

          @canvas = document.querySelector('canvas')
          @ctx = @canvas.getContext('2d')
          @getUserMedia()

          Pusher.host = "ws.darling.pusher.com"
          Pusher.log = (message) ->
            if (window.console && window.console.log)
              window.console.log(message)

          @pusher = new Pusher('YOURKEY')
          @channel = @pusher.subscribe 'stream'

        sizeCanvas: ->
          setTimeout( =>
            @canvas.width = @video.videoWidth;
            @canvas.height = @video.videoHeight;
          , 50)

        refresh: =>
          @ctx.drawImage(@video, 0, 0)
          dataUrl = @canvas.toDataURL('image/webp')
          @channel.trigger 'frame',  dataUrl: dataUrl

        stopRefresh: =>
          console.log('stop timer')
          clearTimeout(@timer)

        snapshot: =>
          @timer = setInterval( @refresh, 250)

        getUserMedia: ->
          version = parseInt(window.navigator.appVersion.match(/Chrome\/(.*?) /)[1].split('.')[0]);
          if navigator.webkitGetUserMedia
            if (version < 21)
                navigator.webkitGetUserMedia('video', (stream) =>
                  @video.src = window.webkitURL.createObjectURL(stream);
                  @sizeCanvas()
                  @button.textContent = 'Take Shot'
                , @onFailSoHard)
            else
                navigator.webkitGetUserMedia({video:true}, (stream) =>
                  @video.src = window.webkitURL.createObjectURL(stream)
                  @sizeCanvas()
                  @button.textContent = 'Take Shot'
                , @onFailSoHard)
          else
           console.log 'Error with getUserMedia()'

The getUserMedia is a wrapper around webkitGetUserMedia. We need it because Chrome 18/19 and Chrome 21 have a slightly different syntax.

After the user clicks the button, we start a timer and we refresh the image taken from the video and send this data to the @channel.

Fin

The first problem lies in the way images are encoded, they are Base64 images. Ideally we could send binary data over a websocket. Pusher also has a rate limit on the number of messages per second.

Can we use websockets for video streaming? Not with this setup, after all it was just an experiment.

Source Code and Credits

The source code can be found here videostream-pusher-pipe

This experiment borrowed code from smartjava.org and from html5rocks.com