Unconventional architectures that might scale (I)

I’m experimenting with some unconventional architectural ideas (at least from a Java developers view) while designing a new architecture for Dutch broadcaster VPRO. In this series of blogposts I’ll try to explain some of the concepts; please feel free to point out issues/caveats I failed to notice!
Continue reading

Posted in architecture | 8 Comments

QCon 2009 – 3

QCon 2009 is no more. To bad, it was an really great conference. Today I mainly attended tracks aimed at how to be / become a good architect/teamlead and a really nice talk about the technical issue the twitter team faces (and solved).

Things I’ve learned (or heard again but think are important to keep mentioning) today:

  • Stories are important, teams need a ‘Shaman’ who is capable of explaining why specific choices where made in the past
  • Wise architects only need to have one answer to be capable of anwering any question: it depends…
  • Architects and developers tend to try to solve problems in far to generic ways. Generic systems don’t tend to deliver the best end-user experience.
  • 300 messages a second on twitter doesn’t sound like a lot. They are however send to a lot of followers!
  • On the question of why twitter decided to built their own messaging queue instead of using an existing one (ie. JMS) they answered: well, our current solution is about 1200 lines of scala and perfeclty fits are needs. No other product could offer us that.
  • Apache Camel and ActiveMQ are hot (a Dutch article I wrote on this should be published shortly)
  • Simple solutions are often the best solution. HTTP is the most common example.
  • Be very aware of the ‘here is a solution, what was the problem again’ approach.
  • Having the get in your flight home while the conference is not over yet sorta sucks; well actually the stress on getting on the airplane in time isn’t really nice

All in all, QCon 2009 was probably the best conference I’ve ever visited. I’ve learned tons of really useful stuff. Not only from a technological point of view, but from a social point of view is well. It is delightful to hear how other people are solving or have solved the same problems as I am. I really hope to visit QCon 2009 next year as well!

Posted in conference | 2 Comments

QCon 2009 – 2

Big Ben just outside the conference rooms I just returned from another great day of QCon. Today I focused on software architecture tracks. They where delightful.

Things I’ve learned (or heard again but think are important to keep mentioning) today:

  • BBC has plans very similar to what I’m working on for Dutch broadcaster VPRO. A REST oriented service layer with lightweight widget/mashup rendering above and tools suited for the problem domain below. They are just a tiny bit further then we are at the moment.
  • Guardian.co.uk built their application quite similar to the way I architected/developed cinema.nl apart from some details in caches. They are happy, but moving to REST oriented architecture to reduce coupling as well.
  • Architects are falling from their towers and starting to use common-sense technology (like HTTP, RSS, ATOM, REST) more and more and are abandoning ‘enterprise’ patterns and tools (think JEE, Portals, SOAP etc).
  • New architectures are often more about the social aspect then the technical aspect
  • Clojure is definitely going on my short-list of ‘languages to look at’

Now, to see what the third day will bring!

Posted in conference | Leave a comment

QCon 2009 – 1

Keynote with fowler and exley #qcon Two days ago I traveled to London to visit this years QCon London conference. It has been great so far!

I’ve listened (and asked questions) to various interesting members of the community like C.A.R. Hoare (Computer Science, inventor of qsort), Martin Fowler (Ruby, Keynote), Ola Bini (Emerging languages), Kirk Pepperdine (Scalability), Cameron Purdy (Scalability), Jonas Bonér (Scala), Rich Hickey (Clojure) and more!

The smaller scale of QCon (as oposed to say JavaOne or Devoxx) and talks in an almost classroom like settings really work for me; you can actually get into a discussion with the speaker/audience instead of just being bombarded with tons of slides.

Things I’ve learned (or heard again but think are important) sofar:

  • Scalability needs to be part of you architecture
  • Think about graceful degrading when the unexpected happens
  • You can actually do += on a val in scala: code snippet
  • I really need to look at Clojure, although I think the code looks hideous
  • Thoughtworks doesn’t seem to have strict guidelines in Ruby development but seems to do just fine without that
  • Start a project with an experienced team, bring in the less skilled developers when the foundation is ready
  • Some people have a really skewed view on using REST
  • It’s really nice to have a twitter ‘backchannel’
  • Guinness beer rocks

Now, to see what the second day will bring!

Posted in conference | Leave a comment

Ioke @ Amsterdam.rb

Tonight I visited Amsterdam.rb. Mainly because I wanted to see Ola Bini talk about his pet project Ioke. Although Ioke is far from finished, or even usable in real life it was really nice to see someone try out new language concepts.

Ioke is a strongly typed, extremely dynamic, prototype based object oriented language. It’s homoiconic and got built in support for several kinds of macros. The languages that most closely influence Ioke is Io, Smalltalk, Self, Ruby and Lisp (Specifically Common Lisp).

Ioke seems to be mainly focussed at writing nice-to-talk-to code, it gives developers much more freedom to shape the language into DSL-like syntax.

In contrast to what you would expect from a ‘pet language’ Ola demonstrated nice generated documentation and some cool debugger features. And even a RSpec-like testing framework: ISpec. Really nice. Here is an example of a parser definition (from paser_comb.ik example) written in Ioke:

[code]
IParse Parser(
digit = 0..9
letter = ("a".."z") | ("A".."Z")
id = letter (letter | digit)*
id2 = letter* (letter | digit)
number = 1..9 digit*
primary = "(" expr ")" | number | id
term = primary ("*" | "/") term | primary
expr = term ("+" | "-") expr | term
and = expr "and" expr | expr
)
[/code]

Looks kinda neat, doesn’t it? I don’t see Ioke as the next big thing, but I think it is a really nice testbed for new languages and language concepts.

Posted in java | Tagged | Leave a comment

Posting to twitter from Android (commons httpclient 4+)

While developing a helper for the Twitter API to be used in the Android application which I’m writing I ran into a nasty problem. While trying to post to Twitter Twitter kept sending “417 – Expectation Failed” responses. I solved this by specifically setting the UseExpectContinue flag to false.

The following snippets are part of my helper class to talk to twitter, it runs fine on my ADP1 using Android 1+ :

[java]
private static final DefaultHttpClient httpclient = new DefaultHttpClient();
private static final AuthScope AUTH_SCOPE = new AuthScope(“twitter.com”, 80, AuthScope.ANY_REALM);

private static JSONObject post(final String url, final Map fields, final UsernamePasswordCredentials credentials) throws Exception {
final HttpParams params = createParamsForPosting();

// Tell twitter who we are
httpclient.getCredentialsProvider().setCredentials(AUTH_SCOPE, credentials);
httpclient.setParams(params);

final HttpPost post = new HttpPost(url);

// Setup the form data to post
final UrlEncodedFormEntity requestEntity = mapToFormEntity(fields);
post.setEntity(requestEntity);

final HttpResponse response = httpclient.execute(post);
final HttpEntity responseEntity = response.getEntity();
final String data = IOUtils.convertStreamToString(responseEntity.getContent());

return new JSONObject(data);
}

private static HttpParams createParamsForPosting() {
final HttpParams params = new BasicHttpParams();
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(params, UTF_8);
HttpProtocolParams.setUseExpectContinue(params, false); // solves the ’417′ issue
return params;
}

private static UrlEncodedFormEntity mapToFormEntity(final Map fields) throws UnsupportedEncodingException {
final ArrayList values = new ArrayList(fields.size());
for (final Entry entry : fields.entrySet()) {
values.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));
}

final UrlEncodedFormEntity entity = new UrlEncodedFormEntity(values, UTF_8);
return entity;
}
[/java]

When complete I’ll probably opensource the twitter API wrapper in full, at the moment it is not complete enough to do so.

Posted in android | 6 Comments

ADP1

Today UPS brought me the Android Generation One Dev phone which I ordered only 2 days ago. Although I’m stuck with a bad case of the flu I had to unwrap and operate it.

Nice present you bought me dad!

Initial impressions of the device are quite good. But yes it is not and iPhone. Installing applications from the marketplace is a breeze.

I started hacking on an Android app myself, but felt to ill to figure out how to get it on the ADP1 today. As soon as I get better I’ll do a write-up!

Posted in android, g1, gadgets | 1 Comment

Changing the default image of the audio player of the Popcorn Hour A-110 / NMT

I recently replaced my broke Philips Streamium SL300 with a brand new Popcorn Hour A-110 networked media tank. I’m really happy with the fact that it plays anything I throw at it. The internet based features (think Flickr, Youtube, Last.fm etc) are cool as well. On think that is seriously lacking is features for audio playback and searching. The default software is a really basic file system browser with, which can read playlist files.

I started working on an application to create a more useful browser based on ID3 tags. Still in the early stages though.

One thing I quickly ran into was the fact that the default player doesn’t allow you the set the image (you can set a background image, but you loose the overlay features if you do). The default image (if no image specified in the ID3 tag) looks like this:

logo2

Which is not really what I’d like to see when playing music.

It took me a while to figure out how to do it though, and it is not for the fainthearted:

  • install telnet deamon
  • telnet to popcorn hour
  • cd to /bin/
  • replace logo2.jpg or logo2hd.jpg (depending on you screen) with your image
  • of coarse you can do this from a php or cgi script as well…
  • have fun
Posted in gadgets | 1 Comment

mx:VideoDisplay and rtmpe streaming

I’ve spend some time trying to figure out how the get the Flex VideoDisplay object to play RTMPE streams. It turns out it’s just not possible using the default component.

The underlying NCManager class has the following hardcoded list of supported protocols, which doesn’t include RTMPE:

[code]
private static var RTMP_CONN:Array = [
{ protocol: "rtmp:/", port: "1935" },
{ protocol: "rtmp:/", port:"443" },
{ protocol: "rtmpt:/", port:"80" },
{ protocol: "rtmps:/", port:"443" }
];
[/code]

The connectRTMP function just iterates over it without any support to force a specific connection protocol:

[code]
private function connectRTMP():Boolean
{
// setup timeout
connectionTimer.reset();
connectionTimer.start();

tryNC = [];

tryNCTimer = new Timer(DEFAULT_NC_TIMEOUT);
tryNCTimer.addEventListener(TimerEvent.TIMER, nextConnect);

for (var i:uint = 0; i < RTMP_CONN.length; i++)
{
tryNC[i] = new NetConnection();
tryNC[i].objectEncoding = ObjectEncoding.AMF0;
tryNC[i].client = new NCManagerConnectClient(tryNC[i], this, i);
tryNC[i].addEventListener(NetStatusEvent.NET_STATUS, connectOnStatus);
}

nextConnect(null);
return false;
}
[/code]

I've filed this as a bug in the adobe issuetracker.

Posted in flex | Leave a comment

Using Google maps for displaying large images in a page

Via the Trendmatcher weblog I came across a nifty tool to use the Google Maps viewer for arbitrary images.

The Google Maps Image Cutter, as the application is called. Is a small Java application which cuts a selected image to thousands of small images compatible with the viewer and a working HTML file.

Really useful for embedding your favorite diagrams (this is diagram I generated for a legacy MMBase application) in a blog post:

Now… the only wish I still have is the option to just use .dot (graphviz) files or a vector graphics format directly ;)

Posted in diagram, google | 1 Comment