log4p

Peter Maas’s Weblog

Selenium2 from Scalatest

March 16th, 2010 | Category: scala, testing

Scalatest offers some very elegant ways to layout your tests. After using Selenium2 at work I started thinking how I could leverage the BDD-like goodness in combination with Selenium2. The combination proves to be very useful.

Some basics
Selenium2 offers a couple of drivers which allow your code to spawn webbrowsers (or a virtual browser) and control and monitor them.

In Scala starting a Firefox instace and opening a url would look like:

PLAIN TEXT
  1. val driver = new FirefoxDriver
  2. driver.get("http://log4p.com/")

After opening the page you can use the driver to browse the actual Dom of the response. Methods for this include stuff like findElementsByTagName, ClassName, Id and XPath. It even allows you to make a PNG image of the page.

Bind pages to classes
Next to the above methods the driver can 'bind' values in the page to bean properties. I discovered that Selenium has no problems with populating annotated Scala classes:

PLAIN TEXT
  1. class BlogPage {
  2.   @FindBy(how = How.ID, using = "cat")
  3.   var categorySelect: WebElement = null
  4.  
  5.   @FindBy(how = How.ID, using = "content")
  6.   var content: WebElement = null
  7.  
  8.   ....
  9. }

The FindBy annotations can also be configure to use classes, XPaths, partial url texts etc. Populating instances can be done via the driver (after having it request a url):

PLAIN TEXT
  1. driver.get("http://log4p.com/")
  2. val indexPage = PageFactory.initElements(driver, classOf[BlogPage])

combined with FeatureSpec
FeatureSpecs allow you to describe scenarios of features, and has options to create 'pending' tests; very useful for test driven development. If we combine Selenium with this type of test we get something like this:

PLAIN TEXT
  1. class Log4pSpec extends FeatureSpec with GivenWhenThen with ShouldMatchers {
  2.   lazy val driver = new FirefoxDriver
  3.  
  4.   feature("A weblog should have a archive based on post categories") {
  5.     info("As a visitor")
  6.     info("I want to be able to select a category")
  7.  
  8.     scenario("we select the 'scala' category to see posts about Scala") {
  9.       given("we open the frontpage page")
  10.  
  11.       driver.get("http://log4p.com/")
  12.       val indexPage = PageFactory.initElements(driver, classOf[BlogPage])
  13.  
  14.       then("we should see a widget containing categories")
  15.       indexPage.categorySelect.isEnabled should equal(true)
  16.  
  17.       given("we select 'scala' using the select widget")
  18.       indexPage.selectCategoryName("scala")
  19.  
  20.       then("the url for that category should be openend")
  21.       driver.getCurrentUrl should endWith("/category/scala/")
  22.  
  23.       then("all posts should have the selected category in their metadata")
  24.       val categoryPage = PageFactory.initElements(driver, classOf[BlogPage])
  25.       categoryPage.verifyPostCategories("scala") should equal(true)
  26.     }
  27.  
  28.   }
  29. }

This test will:

  • Start Firefox
  • Open the frontpage of this blog
  • Select 'scala' in the category box in the menu on the right
  • Verify the opened url
  • Verify that all posts in the category page have the selected category in their metadata

If you run the test, scalatest will create some very nice output:

PLAIN TEXT
  1. Feature: A weblog should have a archive based on post categories
  2.   As a visitor
  3.   I want to be able to select a category
  4.   Scenario: we select the 'scala' category to see posts about Scala
  5.     Given we open the frontpage page
  6.     Then we should see a widget containing categories
  7.     Given we select 'scala' using the select widget
  8.     Then the url for that category should be openend
  9.     Then all posts should have the selected category in their metadata

Keen observers might have noticed some helper methods on the BlogPage class, like 'selectCategoryName' and 'verifyPostCategories'. Using the collection conversions from Scala 2.8 we don't even have to wrap the Selenium API to make them look nice:

PLAIN TEXT
  1. class BlogPage {
  2.   @FindBy(how = How.ID, using = "cat")
  3.   var categorySelect: WebElement = null
  4.  
  5.   @FindBy(how = How.ID, using = "content")
  6.   var content: WebElement = null
  7.  
  8.  
  9.   def selectCategoryName(categoryName: String) = {
  10.     val options = categorySelect.findElements(By.tagName("option"))
  11.     val option: Option[WebElement] = options.find(_.getText.equals(categoryName))
  12.     if (option.isDefined)
  13.       option.get.setSelected
  14.   }
  15.  
  16.   def verifyPostCategories(categoryName: String): Boolean = {
  17.     val entries = content.findElements(By.className("entry"))
  18.     entries.forall {
  19.       entry =>
  20.         val entryMetaData = entry.findElement(By.className("entrymeta"))
  21.         entryMetaData.getText.split("Category:").last.split(",").map(_.trim).contains(categoryName)
  22.     }
  23.   }
  24. }

As you can see translating stories into real integration tests is quite simple; and it should be possible to run the tests in different browsers (Chrome, IE) as well. And since you won't be writing production code it could be a very nice way to give scala into your project!

No comments

Presentation: DSLs in Scala

March 04th, 2010 | Category: scala

At DuSe III I gave a presentation on writing Domain Specific Languages using Scala based on the work I did for my blogposts on internal and external with Scala.

Domain Specific Languages In Scala Duse3
View more presentations from Peter Maas.

After the SQL example I also showed some of the work I did on a custom templating language I wrote as a experiment for Ebay/Markplaats for which I wrote the parser and interpreter in Scala. Maybe I can give some more details on that in the future.

No comments

Introduction to Scala

March 04th, 2010 | Category: scala

Last Tuesday I gave a 'introduction to Scala' presentation during the monthly 'Tech Tuesday' at Ebay/Marktplaats. After going through the slides I did some live test driven development. I think I managed to get some more people interested!

Introduction To Scala
View more presentations from Peter Maas.
No comments

Back on Ubuntu / got a new job.

January 09th, 2010 | Category: linux, ubuntu

Last week was my first week at my new employers office. At eBay / Marktplaats I'll be working on the upcoming version of Marktplaats and various applications/services part of the platform around it.

You can either bring your own hardware or use a corporate laptop. Since I don't actually own a laptop I went for the corporate machine. I got a (huge) Dell Latitude D830. It contains a T7700 core duo 2.4gz, 4gb of memory and a NVidia Quadro NVS140M (business version of the NVIDIA GeForce 8400M G).

I installed the 64- bit version Ubuntu 9.10 (karmic) from a bootable USB stick during my first day. Installation went smooth. Getting X to recognize all display configurations took some work and is still far from perfect: stuff like detaching a display/docking system and attaching another display doesn't really work without manual intervention. I setup VMWare and a WinXP image to be able to access all corporate tools and test in IE.

What I like:

  • Compiz has some nice tricks for windows selection. alt-shift-up for window selection and some exposé-like features which I mapped to F9 and F10 like I was used to on Osx.
  • Gnome Do is a workable replacement for Quicksilver
  • Suspending/Hibernating actually works well and configuration is far more flexible then it is on OSX
  • apt is so much better than macports/fink
  • I like gnome virtual desktops better dan spaces on OSX

What I (still) don't like (compared to OSX):

  • Copying text between applications is still not consistent (at least for me)
  • Input dialogs don't get focus all the time
  • Reconfiguration of X is flaky
  • Still looking for a workable replacement for Omnigraffle, using Dia for the moment
  • I miss Mail.app/Addressbook/iCal
  • Some stuff is harder to setup then it should; skype for instance
  • Various hardware related aspects: multitouch trackpad being the most important one

But overall the experience is good. I got productive within a day; gave a presentation and used the corporate tools through VMWare. I'll try and see if I can improve switching X configs, which would really improve the overall experience.

4 comments

ScotchBuilder with Scala 2.8

December 23rd, 2009 | Category: dsl, scala

While writing my internal SQL DSL example I stumbled upon this blogpost about a Type-safe Builder Pattern in Scala a couple of times. One of the issues I have with that code is the unnecessary use of mutable state in the example. Which are probably there to demonstrate the validation of the result at the end of the article.

The blogpost concludes with some complex implicit solution to have the builder only create valid configurations, but (IMHO) fails to address the fact that it is still possible to create OrderOfScotch variants which are invalid through its' constructor.

With Scala 2.8 it is possible to vastly simplify the example by using the the copy method on the case classes and default arguments:

PLAIN TEXT
  1. sealed abstract class Preparation
  2. case object Neat extends Preparation
  3. case object OnTheRocks extends Preparation
  4. case object WithExtraWater extends Preparation
  5.  
  6. sealed abstract class Glass
  7. case object Short extends Glass
  8. case object Tall extends Glass
  9. case object Tulip extends Glass
  10.  
  11. case class OrderOfScotch(val brand:String , val mode:Preparation = Neat, val double:Boolean = false, val glass:Option[Glass] = None) {
  12.   val invalid: Boolean = double && glass.isDefined && glass.get == Short
  13.   if(invalid) {
  14.       throw new IllegalStateException("Illegal combination")
  15.   }
  16.  
  17.   def prepare(p:Preparation) = copy(mode = p)
  18.   def isDouble(d:Boolean) = copy(double = d)
  19.   def inGlass(g:Glass) = copy(glass = Option(g))
  20. }
  21.  
  22. object OrderBuilder {
  23.   def scotch(brand:String) = OrderOfScotch(brand)
  24. }

Using the above you can write something like:

PLAIN TEXT
  1. val scotchOrder1 = scotch("Glenfoobar") prepare WithExtraWater inGlass Tulip
  2. val scotchOrder2 = scotch("Talisker") inGlass Tall prepare Neat

When using an incorrect configuration (in the example implementation a double scotch, in a short glass) creating the Order throws an exception. And since we in effect only use the constructor to create the Order this works in any case.

Consider the following tests:

PLAIN TEXT
  1. class ScotchBuilderSpec extends Spec with ShouldMatchers {
  2.   import OrderBuilder._
  3.   describe("given two scotches with the same configuration") {
  4.     val constructorResult = OrderOfScotch("Glenfoobar", WithExtraWater, false, Option(Tulip))
  5.     val builderResult = scotch("Glenfoobar") prepare WithExtraWater inGlass Tulip
  6.  
  7.     describe("when created by constructor and by the builder") {
  8.       it("they should return identical results") {
  9.         constructorResult should be (builderResult)
  10.         println("success!")
  11.       }
  12.     }
  13.   }
  14.  
  15.   describe("trying to create an illegal order") {
  16.     describe("like combining a double portion with a short glass") {
  17.       it("should throw an exception, for now") {
  18.         evaluating { OrderOfScotch("Glenfoobar", Neat, true, Option(Short)) } should produce[Throwable]
  19.         evaluating { scotch("Glenfoobar") isDouble(true) inGlass Short } should produce[Throwable]       
  20.       }
  21.     }
  22.   }
  23. }

As you can see the amount of boilerplate code was reduced dramatically, we've got rid of all mutable state and moved validation of the Order to a (IMHO) better location!

One 'drawback' of my solution is of course that it only works on runtime; whereas the solution presented in the catches incorrect configurations created by the builder at compiletime... which was the entire point of that blogpost. I'll try and see if I can factor that into a more compact form as well lator on!

2 comments

external DSLs with scala

November 26th, 2009 | Category: dsl, scala

Following up on my previous post on writing an internal SQL-like DSL for Scala I decided to bite the bullet and implement a 'real' parser for a subset of the SQL language to create the object tree from a SQL-like string. Since I didn't have any experience in Parser/Combinators this proved to be quite an interesting exercise.

At this point the specs for the following queries succeed and produce valid results when I render SQL from the object graph:

PLAIN TEXT
  1. select name from users order by name asc
  2. select name from users where name = "peter"
  3. select age from users where age = 30
  4. select name from users where name = "peter" and age = 30
  5. select name from users where age = 20 or age = 30
  6. select name from users where name = "peter" and age = 20 or age = 30
  7. select name,age from users where name = "peter" and (active = true or age = 30)

The entire parser is just under 54 lines of code!

As you can see in the source code the Parser is basically a collection of small parsers which are combined into a parser for the entire language. Let's have a look at the more basic ones:

The order clause
The order clause is probably the easiest part of the query. The production for the parser looks like this:

PLAIN TEXT
  1. def order:Parser[Direction] = {
  2.   "order" ~> "by" ~> ident  ~ ("asc" | "desc") ^^ {
  3.     case f ~ "asc" => Asc(f)
  4.     case f ~ "desc" => Desc(f)
  5.   }
  6. }

It matches a string which starts with order by, then an identifier and ends with asc or desc. The tilde (~) depicts a separater, and the greater then (>) after the tilde is used to drop the fields on the left side of the operator; just to have less repetition in the match algorithm to bind the value into objects which will form the AST.

Quite straightforward and easy to read.

Typed predicates
I had some more troubles implementing the typed predicates and still feel this could be written a bit more concise:

PLAIN TEXT
  1. def predicate = (
  2.     ident ~ "=" ~ boolean ^^ { case f ~ "=" ~ b => BooleanEquals(f,b)}
  3.   | ident ~ "=" ~ stringLiteral ^^ { case f ~ "=" ~ v => StringEquals(f,stripQuotes(v))}
  4.   | ident ~ "=" ~ wholeNumber ^^ { case f ~ "=" ~ i => NumberEquals(f,i.toInt)}
  5. )

And/Or en parentheses
Implementing and/or in the where clause proved to be more difficult then I expected; mainly due to the precedence rules when using parentheses. I asked around on stackoverflow and read how this is done in this great article by Jim McBeath.

This resulted in the following solution:

PLAIN TEXT
  1. def where:Parser[Where] = "where" ~> rep(clause) ^^ (Where(_:_*))
  2.  
  3. def clause:Parser[Clause] = (predicate|parens) * (
  4.    "and" ^^^ { (a:Clause, b:Clause) => And(a,b) } |
  5.     "or" ^^^ { (a:Clause, b:Clause) => Or(a,b) }
  6. )
  7.  
  8. def parens:Parser[Clause] = "(" ~> clause  <~ ")"

Which effectively is a way of repeating predicates or clauses within parentheses interleaved with and/or. The parens production is just pre- and postfixing the clause with parens.

Optional parts
The select and order clause are optional, this is specified using the opt method:

PLAIN TEXT
  1. def query:Parser[Query] = operation ~ from ~ opt(where) ~ opt(order) ^^ {
  2.   case operation ~ from ~ where ~ order => Query(operation, from, where, order)
  3. }

Since the Query object already accepts optional objects as where and order clause binding is straightforward.

Conclusion
Being fairly new to this game I had some trouble finding out how I was supposed to approach some of the problems I faced; but managed to overcome them quickly. There is a lot of documentation on various blogs and books. Writing a parser in Java, Groovy or Ruby would probably have taken me more time and would probably have resulted in far more code.

Now, off to look for some nails to test my new hammer on!

Full sources can be found in the Scala-SQL-DSL github repository.

3 comments

Internal DSLs with Scala

November 23rd, 2009 | Category: dsl, scala

I've been playing around with Scala again lately. Writing a (internal) DSL or a fluent api was still on todo-list.

Instead of writing some arbitrary language for a made-up domain I decided to pick a language and a domain I know: SQL. Or, a rather small subset.

The first step was creating a model of the language (no, I didn't start with this diagram):

sql_dsl

As you can see a query object consists of:

  • Operation (i.e. select, update, delete)
  • From
  • Where, containing various predicates and ways to combine predicates
  • Optional ordering

In Scala my implementation of the model looks like this:

PLAIN TEXT
  1. case class Query(val operation:Operation, val from: From, val where: Where, val order: Option[Direction] = None) {
  2.   def order(dir: Direction) = this.copy(order = Option(dir))
  3. }
  4.  
  5. abstract class Operation {
  6.   def from(table: String) = From(this, table)
  7. }
  8. case class Select(val fields:String*) extends Operation
  9.  
  10. case class From(val operation:Operation, val table: String) {
  11.   def where(clauses: Clause*): Query = Query(operation, this, Where(clauses:_*))
  12. }
  13.  
  14. case class Where(val clauses: Clause*)
  15.  
  16. abstract class Clause {
  17.   def and(otherField: Clause): Clause = And(this, otherField)
  18.   def or(otherField: Clause): Clause = Or(this, otherField)
  19. }
  20.  
  21. case class StringEquals(val f: String, val value: String) extends Clause
  22. case class NumberEquals(val f: String, val value: Number) extends Clause
  23. case class BooleanEquals(val f: String, val value: Boolean) extends Clause
  24. case class In(val field: String, val values: String*) extends Clause
  25. case class And(val clauses: Clause*) extends Clause
  26. case class Or(val clauses: Clause*) extends Clause
  27.  
  28. abstract class Direction
  29. case class Asc(field: String) extends Direction
  30. case class Desc(field: String) extends Direction

As you can see the code is a straightforward implementation of the model, only using immutable values. I added some utility methods which will be used to 'chain' objects:

  • Query#order - Clones the query object and overrides the order with the specified order
  • Operation#from - creates a from clause from the operation object
  • Clause#and / Clause#or - combines two clauses

Next to the model I created a QueryBuilder object which contains some implicit conversions and utility methods:

PLAIN TEXT
  1. object QueryBuilder {
  2.   implicit def tuple2field(t: (String, String)): StringEquals = StringEquals(t._1, t._2)
  3.   implicit def tuple2field(t: (String, Int)): NumberEquals = NumberEquals(t._1, t._2)
  4.   implicit def tuple2field(t: (String, Boolean)): BooleanEquals = BooleanEquals(t._1, t._2)
  5.  
  6.   /** entrypoint for starting a select query */
  7.   def select(fields:String*) = Select(fields:_*)
  8.   def in(field: String, values: String*) = In(field, values: _*)
  9. }

The implicit conversions allow tuples to be converted to typed case classes. With the above we can write Scala code which resembles SQL. I Wrote some tests (using ScalaTest) which demonstrate how it works. Example inputs for my tests include:

PLAIN TEXT
  1. val q = select ("*") from ("user") where (("name","peter") and (("active", true) or ("role", "admin")))
  2. val q = select ("*") from ("user") where (("name","p'eter"))
  3. val q = select ("*") from ("user") where (("id", 100))
  4. val q = select ("*") from ("user") where (in("name","pe'ter","petrus"))
  5. val q = select ("*") from ("user") where (("name","peter")) order Desc("name")

To generate a SQL String from a Query object I wrote a fairly basic generator and an implicit conversion to convert queries to SQL:

PLAIN TEXT
  1. case class SQL(val sql:String)
  2.  
  3. object AnsiSqlRenderer {
  4.   implicit def query2sql(q:Query):SQL = SQL(sql(q))
  5.  
  6.   def sql(q: Query): String = {
  7.     List(
  8.       expandOperation(q),
  9.       expandFrom(q),
  10.       expandWhere(q),
  11.       expandOrder(q)
  12.     ).mkString(" ").trim
  13.   }
  14.  
  15.   def expandOperation(q:Query):String = q.operation match {
  16.     case Select(fields) => "select %s".format(fields.mkString(","))
  17.     case _ => throw new IllegalArgumentException("Operation %s not implemented".format(q.operation))
  18.   }
  19.  
  20.   def expandFrom(q: Query) = "from %s".format(q.from.table)
  21.   def expandWhere(q: Query) = "where %s".format(q.where.clauses.map(expandClause(_)).mkString(" "))
  22.  
  23.   def expandClause(clause: Clause): String = clause match {
  24.     case StringEquals(field, value) => "%s = %s".format(field, quote(value))
  25.     case BooleanEquals(field, value) => "%s = %s".format(field, value)
  26.     case NumberEquals(field, value) => "%s = %s".format(field, value)
  27.     case in:In => "%s in (%s)".format(in.field, in.values.map(quote(_)).mkString(","))
  28.     case and:And => and.clauses.map(expandClause(_)).mkString("(", " and ", ")")
  29.     case or:Or => or.clauses.map(expandClause(_)).mkString("(", " or ", ")")
  30.     case _ => throw new IllegalArgumentException("Clause %s not implemented".format(clause))
  31.   }
  32.  
  33.   def expandOrder(q: Query) = q.order match {
  34.     case Some(direction) => direction match {
  35.       case Asc(field) => "order by %s asc".format(field)
  36.       case Desc(field) => "order by %s desc".format(field)
  37.     }
  38.     case None => ""
  39.   }
  40.  
  41.   def quote(value: String) = "'%s'".format(escape(value))
  42.   def escape(value: String) = value.replaceAll("'", "''")
  43. }

Most of the beef is in (recursively) expanding the where clause into a String. When using the implicit conversion you can now do the following:

PLAIN TEXT
  1. scala> val q = select ("*") from ("user") where (("name","peter") and (("active", true) or ("role", "admin")))
  2. scala> q.sql
  3. res0: java.lang.String = select * from user where (name = 'peter' and (active = true or role = 'admin'))

Apart from the parentheses (probably I'll figure out how to get rid of some more of them one day) the two look very similar. But the first one creates a typesafe object graph which can be rendered/validated/manipulated in various ways!

The example could be extended to allow 'greater/smaller then' clauses (using operator overloading?) or joins in the from clause; feel free to clone my repository and do so!

Full sources can be found in the Scala-SQL-DSL github repository.

6 comments

A simple couchdb info widget for wordpress

June 09th, 2009 | Category: couchdb, php, wordpress

I wrote a small wordpress widget to display some version details of the CouchDB server I'm running. Writing a wordpress widget wasn't to hard (I used this excellent tutorial to get started), parsing CouchDB JSON responses is even simpler.

The code for the widget (put it in 'wp-content/plugins' and enable the plugin afterwards) looks like this:

PLAIN TEXT
  1. <?php
  2. /*
  3. Plugin Name: Couch-Info
  4. Plugin URI: http://log4p.com
  5. Description: Shows information of the configured couchdb server
  6. Version: 0.1
  7. Author: Peter Maas
  8. Author URI: http://log4p.com
  9. */
  10.  
  11. add_action("widgets_init", array('Couch_Info_Widget', 'register'));
  12. class Couch_Info_Widget {
  13.   function control(){
  14.     $data = get_option('couchdb_info_widget');
  15.     ?>
  16.     <p>Configure CouchDB location</p>
  17.     <p><label for="couchdb_info_widget_host">Host</label><input name="couchdb_info_widget_host" type="text" value="<?= $data['host'] ?>" /></p>
  18.     <p><label for="couchdb_info_widget_port">Port</label><input name="couchdb_info_widget_port" type="text" value="<?= $data['port'] ?>" /></p>
  19.     <?php
  20.     if (isset($_POST['couchdb_info_widget_host'])){
  21.       $data['host'] = attribute_escape($_POST['couchdb_info_widget_host']);
  22.       $data['port'] = attribute_escape($_POST['couchdb_info_widget_port']);
  23.       update_option('couchdb_info_widget', $data);
  24.     }
  25.   }
  26.  
  27.   function widget($args){
  28.     $data = get_option('couchdb_info_widget');
  29.  
  30.     $couchdb_version = json_decode(file_get_contents('http://'.$data['host'].':'.$data['port'].'/'))->version;     
  31.     $num_articles = json_decode(file_get_contents('http://'.$data['host'].':'.$data['port'].'/articles/'))->doc_count;
  32.  
  33.     echo $args['before_widget'];
  34.     echo $args['before_title'] . 'COUCHDB INFO' . $args['after_title'];
  35.     ?>
  36.     Parts of this blog are fed by <a href="http://couchdb.apache.org">CouchDB</a>
  37.     I'm running CouchDB of the <a href="http://svn.apache.org/repos/asf/couchdb/trunk/">current SVN trunk</a> and update once in a while. Current version: <strong><?= $couchdb_version ?></strong>
  38.     My CouchDB instance contains all <strong><?= $num_articles ?></strong> articles of this blog. All numbers in this widget are retrieved from CouchDB as well
  39.     <?php
  40.     echo $args['after_widget'];
  41.   }
  42.   function register(){
  43.     register_sidebar_widget('CouchDB Info', array('Couch_Info_Widget', 'widget'));
  44.     register_widget_control('CouchDB Info', array('Couch_Info_Widget', 'control'));
  45.   }
  46. }
  47. ?>

As you can see it is one class which has two functions which are registered in wordpress. 'widget' retrieves the needed data and renders the actual widget. 'control' creates the control form to edit the widgets' settings. This is what the control form looks like in the admin interface:

Widget controlpanel (right bottom)

The output of the 'widget' function should be visible in the bottem of the right sidebar.

2 comments

3v12 api from Scala

June 02nd, 2009 | Category: 3voor12, api, scala

I rewrote the code in my previous post in Scala, with a minor difference.. I'm not using an RSS api here.. Scala has native XML support... which makes writing basic RSS a breeze:

PLAIN TEXT
  1. package v12_rss
  2.  
  3. import java.net.{URLConnection, URL}
  4. import scala.xml._
  5.  
  6. // define some case classes as a simple model for the rss feed we're going to build
  7. case class Channel(title:String, link:String, description:String, items:List[Item]) {
  8.   def toXML = <channel>
  9.                 <title>{title}</title>
  10.                 <link>{link}</link>
  11.                 <description>{description}</description>
  12.                 {items.map{_.toXML}}
  13.               </channel>
  14. }
  15. case class Item(title:String, link:String, description:String, enclosure:Enclosure) {
  16.   def toXML = <item>
  17.                 <title>{title}</title>
  18.                 <link>{link}</link>
  19.                 <description>{}</description>
  20.                 {enclosure.toXML}
  21.               </item>
  22. }
  23. case class Enclosure(url:String) {
  24.   def toXML = <enclosure url={url} type="image/jpeg"/>
  25. }
  26.  
  27. // Helper for working with URN values from the API
  28. case class Urn(urn:String) {
  29.   def comps = urn.split(":").slice(1, 4).toArray // remove the first value ('urn') not interesting
  30.  
  31.   def src = comps.apply(0)
  32.   def mediaType = comps.apply(1)
  33.   def number = comps.apply(2)
  34. }
  35.  
  36. object Main {
  37.  
  38.   def main(args: Array[String]) = {
  39.     val groupElem = retrieveGroup(41129661)
  40.     println(<rss version="2.0">
  41.               {new Channel(title(groupElem), "http://3voor12.vpro.nl/tv/", shortTitle(groupElem), itemize(groupElem)).toXML}
  42.             </rss>)
  43.   }
  44.  
  45.   def retrieveGroup(num:Int):Elem = {
  46.     XML.load("http://3voor12.vpro.nl/api/media/1/rest/group/"+ num + ".xml")
  47.   }
  48.  
  49.   // short for getting an attribute as string
  50.   def attr(node:Node, name:String) = node.attribute(name).get.text
  51.  
  52.   // retrieve the first title field from the given XML
  53.   def title(elem:Elem):String = (elem \\ "title").first.text
  54.  
  55.   // retrieve the first shortTitle field from the given XML
  56.   def shortTitle(elem:Elem):String = (elem \\ "shortTitle").first.text
  57.  
  58.   // determine the image url for the given media xml
  59.   def imageUrl(elem:Elem):String = "http://images.vpro.nl/images/" + new Urn(attr((elem \\ "image").first, "urn")).number
  60.  
  61.   def itemize(group:Elem):List[Item] = {
  62.     val items = (group \\ "media") slice(0, 15) map{ m =>
  63.       val urn = new Urn(attr(m, "urn"))
  64.       println ("http://3voor12.vpro.nl/api/media/1/rest/"+ urn.mediaType +"/"+ urn.number + ".xml")
  65.       val media = XML.load("http://3voor12.vpro.nl/api/media/1/rest/"+ urn.mediaType +"/"+ urn.number + ".xml")
  66.  
  67.       new Item(title(media), "http://3voor12.vpro.nl/tv/#/41129661/" + urn.number , shortTitle(media), new Enclosure(imageUrl(media)))
  68.     }
  69.  
  70.     items.toList
  71.   }
  72. }

1 comment

Using the 3voor12 api to get an RSS feed of your favorite playlist

June 02nd, 2009 | Category: 3voor12, api, ruby

A couple of months ago we developed 3voor12TV. During festivals like Noorderslag, Pinkpop and (upcoming) Lowlands the 3voor12 crew tries to get as much high quality (h264) material online in the shortest time possible.

The application was developed in actionscript 3.0 on top of a public API. Well, it is public... but no public documentation yet. Sorry ;-) The 3voor12 Pinkpop mashup was the first real utilization of the API.

Whilst waiting for the new video's I found myself refreshing the player over and over again; checking for new content.

I decided to automate this using the API. The Ruby script below uses the API to create a simple RSS feed with images and a direct link to each concert:

PLAIN TEXT
  1. require 'rubygems'
  2. require 'json'
  3. require 'open-uri'
  4. require 'rss'
  5.  
  6. API_URL_BASE = "http://3voor12.vpro.nl/api/media/1/rest/"
  7. PLAYLIST_ID = "41129661"
  8.  
  9. # function to convert urn to urls
  10. def urn_to_api_url(urn)
  11.   urn_parts = urn.split(":") # urn contains source, entity type and unique number
  12.   "#{API_URL_BASE}#{urn_parts[2]}/#{urn_parts[3]}.json" #  bypass content negotiation, force json formatted responses
  13. end
  14.  
  15. def urn_to_url(urn)
  16.   urn_parts = urn.split(":")
  17.   "http://3voor12.vpro.nl/tv/\#/#{PLAYLIST_ID}/#{urn_parts[3]}"
  18. end
  19.  
  20. # retrieve and parse a playlist (group)
  21. playlist = JSON.load(open("#{API_URL_BASE}group/#{PLAYLIST_ID}.json"))
  22. # extract the playlist items
  23. programUrns = playlist['group']['members']['member'].map{|m| m['media']['@urn']}
  24.  
  25. # create playlist
  26. rss = RSS::Maker.make("2.0") do |maker|
  27.   maker.channel.title = "3voor12tv :: #{playlist['group']['title']}"
  28.   maker.channel.description = playlist['group']['shortTitle']
  29.   maker.channel.about = "http://3voor12.vpro.nl/tv/"
  30.   maker.channel.link = "http://3voor12.vpro.nl/tv/"
  31.   # retrieve the playlist items
  32.   programUrns[0..15].each do |urn|
  33.     maker.items.new_item do |item|
  34.       program = JSON.load(open(urn_to_api_url(urn)))['program']
  35.       img_url = "http://images.vpro.nl/images/#{program['relatedImages']['relatedImage']['image']['@id']}+s(320)"
  36.      
  37.       item.title = program['title']
  38.       item.description = (program['synopsis'] || program['title']) + "<br/> <img src=\"#{img_url}\"/>"
  39.       item.link = urn_to_url(urn)
  40.      
  41.       enclosure = maker.items.last.enclosure
  42.       enclosure.url = img_url
  43.       enclosure.length = -1
  44.       enclosure.type = "image/jpeg"
  45.     end
  46.   end
  47. end
  48.  
  49. # write to disk
  50. File.open("3v12feed.xml","w") do |f|
  51.  f.write(rss)
  52. end

I have a cronjob executing the script once in a while:

http://feeds2.feedburner.com/3voor12tvPinkpop

3 comments

Next Page »