some random thoughts I’ve been putting down on a writeboard

Recently, I discovered a cool feature on my Macbook. If I hold the control key and scroll with my mousewheel/pad – I can zoom in and zoom out very smoothly. I’ve started using this a lot and it’s become part of my digital habits. I do it that often that colleagues almost think of me as the crazy zooming guy. This full screen zooming is good for many things during my normal productivity hours. I often use the zooming to show people certain parts of a User Interface and sometimes I use it to zoom in on a piece of text to make a statement :]. Also, it’s useful for pair programming. Now, partly because of my deteriorating eyesight, I can zoom in on something and sit back. This could be a word or a picture and while I sit back, I can really think about it. Zooming-in gets rid of all the other chaos on the screen and helps you to focus.

I guess the zooming metaphor applies to many things in life. As work – trained as an engineer – I deal a lot with the nitty gritty of things. I used to program a lot of low-level stuff like Assembly and C code. However I’ve done some more abstract technical things over the past years like: programming in a variety of languages, design patterns and thinking about user interaction. So far, I’ve been apt at being able to zoom in and out in a certain scope of my skill-sets. Most of it has been technical or related to building software. Right now I’m trying to extend my professional zoom-scope to extend areas that require right-brain thinking like design and business/strategy. Also in personal life – which is heavily intertwined with that old term ‘professional life’ – you can apply zooming very well. Zooming-in means getting short term things done, like changing the light bulb in the toilet. Zoomed out means planning ahead and equally important: changing one’s habits.

Another brother of zooming is filtering. Filtering out the irrelevance in today’s information overload is already visible in multi billion dollar industries and in a notable company called Google. The coming semantic wave – or web 3.0 if you will – stresses the importance of user interfaces that deal with the abundance of information. In recent web design trends it’s easy to spot the shift towards information abundance. Websites now have big fonts for important summarizing one-liners whereas old websites i.e. portals tried to cram as much information on the screen as possible. Zooming and filtering will be important metaphors in future User Interfaces. Google Maps and the iPhone browser are notable first examples of these so-called Zooming User Interfaces (ZUI’s).

Last week I mashed together a slideshow showing the pros and cons of using AJAX. Basically it provides an overview of what to do and more importantly what not to do with Ajax, Prototype and RJS. Also it gives a brief hinter of related techniques.

| View | Upload your own

I’ve presented this at a Developer Lunch at iKnow and decided to take out the iKnow specific slides and put it online on slideshare.net.

This morning I received an email from slideshare that they chose this slide to be featured on the front page :]

Recently, there has been a logo design contest for the holy open standard DataPortability.org. Unfortunately, my submission was too late – so I’ll just show my submissions here. I spend about 1 hour on this to practice some basic Illustrator skills.

Thanks goes out to Kirk for inspiration on this one:

Check out the real winners over at dataportability.techcrunch.com

This is an idea I’ve been toying with the last couple of months after working a lot on integrating Twitter, gathering RSS feeds and implementing an activity stream for iknow.co.jp.

When talking about Twitter over a couple of drinks, my friends often roll up their eyeballs. Again, this idiot evangelizing this ‘look-at-me’ service. In a way they are right, Twitter has limited value for it’s users. But the thing that I found most interesting about Twitter is the potential of activity streams and the machine-to-human aspects of it. In my previous article on how to build a Twitter service, I concluded that Twitter has severe limits when it comes to general activity streaming.

But guess what, activity is still HOT and serious innovation here is the new gold.

Activity Streams?

After sites like Plaxo, more and more websites are restructuring certain areas of their sites to facilitate ‘acitivty streams’. LinkedIn just introduced one:

Only a few weeks before 37Signals made this update to my Backpack account:

These activity streams are still kind of vaguely defined some call it Newsroom, Latest Activity or Updates. Here in Tokyo we have many different names for it: ‘notifications’ in programming code, ‘Activity’ on one page and ‘My News/マイニュース’ on the other.

• Events are plotted along a timeline and serve as news flashes.
• They include updates about friends or followee’s.
• They notify you about system events (new message, you have listened to 1000 songs, congratulations!)
• They serve as call to actions: read this message, checkout this blog entry etc. This serves as a social and content creation lubricant.
  

Lifestreaming?

This word is flooding today’s buzztalk. Applications like friendfeed.com and tumblr.com basically allow you to aggregate stuff from other services and republish that in one big ass activity stream.

• They are not updated real-time, they poll external services and passively retrieve new updates.
• If they have filter/categorization capabilities at all, they provide that functionality through their site and not the actual API.
• They focus on social events and omit the upcoming wave of recommendation engine notifications.
• Actively syndicating events through eg XMPP is still hard to find.
• Their paradigm is Software as a Service rather than Platform as a Service
  

an Open Activity Platform

• Providing categorization and call-to-action hooks for notifications.
• Providing authentication for external services and easy access ways.
• Sophisticated filtering mechanisms for users.
• Allowing real-time pushing of these incoming notifications to third party services.

I’m not really aware of current standardization drafts that accommodate these – perhaps DataPortability.org will do the job. But standards like oAuth, JSON and XMPP play in very well with the implementation of such a platform.

Interestingly – while searching for a suitable home for such a platform – I discovered the domain openactivity.org has already been taken by a certain company from Redmond:

1
2
3
4
5

Domain Name:OPENACTIVITY.ORG
Registrant Name:Sean Lyndersay
Registrant Street1:One Microsoft Way
Registrant City:Redmond
Registrant Email:seanlynd@microsoft.com

Skinning the Platform

The other side of the coin is the tools that plug into this Open Activity Broker.User added value will lay in tools that provide display and control very well. Lifestreamers and the like should be mere frontends. Secondly, tremendous value can be added by plugging in recommendation engines and integrating with existing services.

I haven’t done a thorough comparison of current lifestream-like applications out there, but I do have some ideas for such an Activity Frontend. The core value will be sophisticated filtering and categorization on incoming notifications. A filter like that should be organic eg. voting down certain sources or notification types. Right now I’m following about 50 people on Twitter and I don’t have time to give any attention to this public timeline – I need to easily and seriously filter the noise!

During one of my takout-sushi lunches I made the above mockup. The important filtering/categorization controls are missing, because building those will require serious thinking about user interaction. Techniques like Comet) can make this web application real-time (pushing notifications on the page as they happen).

Also, integration with the Desktop world is an interesting prospective. As the platform should play well with current open standards like XMPP, so should the frontend play well with open UI libraries like Growl (or Snarl for windows and Ghosd for Linux):

a Project has been Born?

Not yet, although I can probably not resist writing some prototypes. However, for something like this to really work well – there needs to be some kind of community actively backing the non-profit part, the Open Activity Broker. This will require a lot of commitment making the whole package a fulltime gig.

Any takers? Or any tips that this is just reinventing the wheel? Please share your thoughts.

Last year I discovered the website Wakoopa.com. At first I didn’t quite get the usefulness of using Wakoopa but after using it I really came to love it. Wakoopa let’s you broadcast your software activity usage by installing a little tray icon. On your Wakoopa profile you can track your software usage trends. Since I’m a proud Mac user and believe in radical transparency I am broadcasting the usage of my software to the world:

Right now on this blog you can see my Latest Tweets and Interesting Links. That data get’s updated everytime I bookmark something (using Delicious) or tweet something (using Twitter). I really like the real-time and heartbeat aspect of it.

While taking a shower a couple of weeks ago I was thinking of using the Wakoopa data to make a little widget. Most of my time is spend in front of my Macbook, so it would be nice to display what I’ve been doing the last hours on my Mac. I want to display it in a simple single sentence like “Recently I’ve been programming”. I don’t want to overload my visitors with too much data. Of course when they click on the sentence they are taken to my Wakoopa profile

I contacted the Wakoopa team a couple of weeks ago to ask whether they could provide this activity data in form of a JSON feed (essential for making a badge/widget). They replied to me within a couple of hours and implemented it! That’s agile baby! Unfortunately I wasn’t so agile on my private projects and only build my widget yesterday.

Using the just released Wakoopa JSON api I’ve written a big ugly piece of Javascript that serves as the widget:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

<script type="text/javascript">
  var wakoopa_activity = {'Design': 'designing',
                          'Communication': 'mailing and chitchating',
                          'Browsers': 'surfing the web',
                          'Programming': 'coding like hell',
                          'Audio': 'slacking off',
                          'Video': 'slacking off', // jacking off? 
                          'Tools': 'doing some funky stuff'}
  var wakoopa_username = null;
  
  function wakoopa_doing_now(data) {
    var app = null;
    var activity = '';
    for(var i = 0; data.length > i; i++) {
      if (app == null || app.active_seconds <= data[i].active_seconds) {
        app = data[i];
      }
    }
    if(app.category) {
      activity = wakoopa_activity[app.category.name];
      if (!activity) {
        activity = app.category.name;
      }
      if (app.category.name == 'Tools') {
        if (app.name == 'iTerm' || app.name == 'Terminal') {
          activity = 'unix fu';
        }
      }
    }
    title = 'my software usage on wakoopa.com';
    document.getElementById('wakoopa_activity').innerHTML = '<a href="http://wakoopa.com/' + wakoopa_username + '" alt="' + title + '" title="' + title + '">Recently I\'ve been ' + activity + '.</a>';
  }
  
  function wakoopa_recent_activity_for(username) {
    var script = document.createElement( 'script' );
    wakoopa_username = username;
    script.type = 'text/javascript';
    script.src = 'http://wakoopa.com/' + username + '/recently_used.json?callback=wakoopa_doing_now';
    document.getElementsByTagName('head')[0].appendChild( script );
  }
</script>
<img style="float: left; margin-right: 6px" src="http://wakoopa.com/favicon.ico"/>
<div id="wakoopa_activity"></div>
<script type="text/javascript"> wakoopa_recent_activity_for('dominiek'); </script>

You can copy and paste this in your blog to display what you’ve been doing recently on your machine. Make sure to replace my username with yours.

Note, while working on this project this ReadWriteWeb article was released, illustrating the future potential of the Jabber/XMPP protocol.

In this article we will build an actual useful Twitter Service that will allow us to track the Blogosphere. In the process we will get hands on programming experience with Ruby, DRb, Twitter and Jabber. This will sharpen our developer skill-set to get ready for the upcoming (Folk)Semantic Web. Also we evaluate the problems seen and opportunities ahead.

Background

Whether you want to call it Web 2.0, Web 3.0, the Semantic Web or the Web of Data – change is happening. The past years we’ve seen the tremendous power of Folkosonomies and now this Social Web is colliding with the emergence of the Semantic Web, resulting in the first semantic services. For us developers and creative entrepreneurs it’s important to get ready for this new wave of business opportunities. I find the whole notion of Intelligent Agents very interesting. For our little project however, we will create a Stupid Agent :]

Technologies like Jabber/XMPP and DRb will enable us to move from a reactive web to a proactive web. Right now this proactive realtime push of data is important for the more liquid content creation services. Micro/nano blogging platforms as Twitter and Tumblr are good examples of this. This is one of the reasons that they already have a Jabber service set up.

I’ve had used Jabber before to communicate with my geek friends. For this project, I had to set up a Jabber client and Jabber account. Call me stupid, but I actually had to spend 30 minutes figuring out how the hell I had to create an account and choose which server to use (turns out you can do that in the client). Now of course XMPP/Jabber is just a standard for enabling IM communication, but apart from Google’s GTalk there hasn’t been much widespread use by ordinary users. In my view, these uses of XMPP for machine to machine (to human) are much more interesting.

Case: The Observatory Bot

I’ve been programming for quite some time now. When learning a new language I really hate doing little examples that produce zero user/business value. That’s why I think the best way to learn new technologies is to solve real world problems right away. Of course tutorials are valuable to just get a general idea of what’s going on, but don’t waste too much time on them – implement straight away.

Our Twitter Service will also need to create some value for the user and must be production ready. However, don’t get your hopes up too much since these are experimental technologies with dependence on external services like Jabber and Twitter.

Remember my last little project? Wigitize.com is actually generating a lot of data. It’s tracking about 5000 6000 feeds every hour! Let’s do something with that data :]

• is a Twitter-only service
• will allow you to ‘track’ the Blogosphere
• will send you a direct message when something happens in the Blogosphere

Basically this is like Twitter’s IM functionality to track the Twittersphere. So in a sense The Observatory will be a proof of concept portal between the Twittersphere and the Blogosphere.

The Architecture

Right now Wigitize.com uses BackgrounDRb to perform background tasks and also to update all RSS feeds periodically. Everytime the feed aggregation process finds a new feed entry it will create a FeedEntry instance. The creation of these objects serve as events for the Observatory Bot. These events have to be pushed to the Observatory Bot in some way.

What can Twitter’s IM service do for us?

To play around with Twitter’s agent you need to set up a Jabber account and a Twitter account. For debugging I’ve found the MacOS tool JabberFox very helpful.

Basically, these commands are available:

However, I think there are a lot more hidden commands which can be used. After emailing with the Twitter developers they told me there is a command called “d”. This can be used to send direct messages (d username message). Very useful!

Coding the Bot

Our implementation choice for today will be Ruby. If you’ve programmed intensively in other languages before you’ve probably come to the conclusion that Ruby is quite different from most other. Ruby’s flexible object models allow for great extension of the language itself (eg 3.minutes.ago). It is therefore no surprise that interesting Semantic Web projects like ActiveRDF are choosing Ruby as their language.

To communicate with Twitter we can use this cool Ruby Twitter API. Unfortunately all of these API’s are HTTP/REST driven and really limit what we can do in terms of realtime response. Also, if you want to build a serious production ready service, constantly polling Twitter will kill both parties.

So we need to interface with their Jabber Service. Jabber is a friendly name for Instant Messaging (IM) using the open XMPP protocol. Luckily, there is a Ruby library called XMPP4R which does most of the XMPP work for us. This blog post provides some simple examples and this German wiki entry provides sample code how to use callbacks (very important for a bot).

I’ve wrapped all of this in a simple JabberBot class jabber_bot.rb that can be used like this:

1
2
3
4
5
6
7
8

  class MyJabberBot < JabberBot
    def on_message(from, body)
      say(from, "You said: #{body}")
    end
  end
  my_jabber_bot = MyJabberBot.new('observatory@jabber.org', 'password')
  my_jabber_bot.connect_and_authenticate
  my_jabber_bot.run

As you can see in the diagram, I’ve build a TwitterBot on top of this JabberBot. Unfortunatly it’s not possible to do all communication with Twitter through Jabber yet. For example: there are no events for when users start following other users or ways to retrieve information. This is why twitter_bot.rb is essentially a hybrid using both the Twitter API and Twitter’s Jabber service. Feel free to use all sourcecode provided here, I know it will be useful to some of you out there. This is how to use this TwitterBot:

1
2
3
4
5
6
7
8
9
10
11
12

  twitter_bot = TwitterBot.new('observatory', 'password', 'observatory@jabber.org', 'password')
  twitter_bot.track_phrases = ['observatory.topoints.com']
  twitter_bot.on_directed_tweet do |username, message|
    puts("directed tweet: #{username} says #{message}")
  end
  twitter_bot.on_tweet do |username, message|
    puts("something from #{username}: #{message}")
  end
  twitter_bot.on_track do |username, message, phrase|
    puts("track: #{username} says #{message} (keyword: #{phrase}")
  end
  twitter_bot.runn(:follow_all_followers => true)

Now that we have the basic building blocks to build our service, let’s build our core business logic (observatory_twitter_bot.rb):

This means that we will send a greeting when people start following us:

1
2
3
4
5
6
7
8

  on_follow do |username|
    logger.info("#{username} is following us, will follow #{username} too and send welcome message")
    follow(username)
    direct_message(username, "the Observatory is now ready to serve you, use '@observatory track [keyword]' to get blogosphere updates.")
  end
  on_unfollow do |username|
    logger.info("#{username} stopped following us")
  end

Note: in order to get the on_follow event, we have to poll the Twitter HTTP API . Since Twitter limits the rate to 70 requests per hour, I poll every two minutes to be on the safe side.

And that we will start tracking the Blogosphere for them when they say the magic word:

1
2
3
4
5
6
7
8
9
10
11
12

  on_directed_tweet do |username, message|
    logger.info("directed tweet: #{username} says #{message}")
    if (phrase = track_phrase(message))
      logger.info("tracking '#{phrase}' for user #{username}")
      begin
        direct_message(username, "Will send a direct message anytime something happens in the Blogosphere regarding '#{phrase}'")
        Tracker.for(username, phrase)
      rescue => e
        logger.error("tracking failure: #{e.to_s}")
      end
    end
  end

Now when a new FeedEntry is created, we need to make sure that these Twitter users get notified when their tracked phrase matches the FeedEntry. Since this might take up some time, I’ve created a background worker task for it:

As you might see, Distributed Ruby (DRb) makes it extremely easy to control our bot remotely. In the ObservatoryBot we say:

1

  DRb.start_service("druby://:8997", self)

And all bot functionality can be accessed by calling: observatory_bot = DRbObject.new(nil, ‘druby://:8997’)

Now that we have our autonomous agent it would be nice if we could easily start and stop it in a production environment. I found the Ruby Gem called Daemons extremely useful to wrap these things up.

First, set up a file that runs the never ending process (eg script/observatory_twitter_bot.rb):

1
2
3
4
5
6
7
8

require 'logger'
require File.dirname(__FILE__) + '/../config/boot'
require File.dirname(__FILE__) + '/../config/environment'
require File.dirname(__FILE__) + '/../lib/observatory_twitter_bot'

logger = Logger.new(File.join(RAILS_ROOT, 'log/observatory_twitter_bot.log'))
observatory_twitter_bot = ObservatoryTwitterBot.new(logger)
observatory_twitter_bot.runn

Next, wrap this up in a daemon script (eg script/observatory_twitter_bot):

1
2
3
4
5
6
7

#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'rubygems'
require 'daemons'

Daemons.run('script/observatory_twitter_bot.rb')

The Demo

Right now if all communication lines with Twitter are working fine, the service is up and running. I’ve made a little bot homepage at observatory.topoints.com

@observatory track ‘Twitter’:

Problems and Opportunities

In the development of the Observatory I had one big obstacle: Twitter is often down and it can cripple your service and development time. I understand that Twitter is a small team under enormous pressure but there have been a lot of complaints about this.

Nevertheless Twitter and it’s developers really kick ass. I emailed Alex Payne and he was excited about what I’m doing (and also the Twitter things happening the iKnow! project in Japan). He responded fairly quickly and immediately whitelisted my Twitter account to up the rate limits.

While working on this project I realized that Twitter in it’s current state isn’t really suitable for system-to-human notifications. Twitter could expand their system to be a true notification framework, but I’m not sure if they will. If they don’t, there is a tremendous business opportunity here. Imagine an open API that mashes up with technologies like XMPP and Growl. A service like that could become THE notification-bus of the web! (Already Growl is pretty big in Mac land). A full blog post about this startup idea coming up!

Geek Food for Thought

What about RubyOnRails, Jabber, DRb, Daemons and BackgrounDRb? I think we are seeing a new framework here! In this interesting article by Danny Ayers he talks about a toolset for agents. On Java there is already “an Agent Framework” that I haven’t checked out yet. I can imagine that these frameworks ease doing development like this and facilitate better system autonomy. Of course it’s desirable to give such a framework a pragmatic paintjob by using Rails paradigms.

Above here I’ve illustrated Rails’ missing brother. I think it’s also good to take into account interesting technologies like Juggernaut and Comet. These are basically Javascript Push techniques to make a synchronous interaction on the asynchronous web possible.

When you combine all these micro- asynchronous communication lines you get one big synchronous connection line between machine agents and user agents.

Note: hashtags.org has banned me for spam and I understand. Sorry guys! The experiment was stopped as of 11:00 JST

Last night I did a little art experiment: What happens when you use a service like TwitterFeed.com to feed your Twitter status updates back into itself.

• created a user called loopyloop
• used twitterfeed to update both the RSS and Atom feed
• used the great service hashtags.org to keep track of the activity

I was hoping to see some interesting emergent behavior, but the end result was a simple update every half an hour with 5 updates.

I do think it is interesting and important to think about full-artificial organisms living and looping through our cyberspace.

This is a more extensive writeup of a problem I have talked about before. It’s about something many of us web innovators face today. Like many problems, they can be turned into opportunities if implemented well.

The new Web – let’s stop using 2.0 – is much more than social networking sites and round cornered layouts. As Tim O’Reilly said, people who think Web 2.0 is about AJAX are completely missing the boat. This is true, but it is important to keep in mind that the shift from desktop products to web services is a big deal. Online services are already gaining a lot of productivity for people and thanks to the characteristics of the web the companies developing it can distribute and evolve much better. So this means that certain value adders have now migrated to the web.

What is happening now is that many of these online services want to be able to harness the power of the new social characteristics of the web. People have their friends online now and there is a lot of activity between those people and services (UGC). Everybody wants to tap into this attention stream between people. It helps contextualize your offering and helps spread the goodness between like minded.

Unfortunately, getting this social activity going around your service is not easy. And frankly, most of the time it results in creating another lousy SNS with worse social features. Fortunately, most online services open up their API’s allowing you to integrate with their services to create a mashup.

• rate, review and discuss cheeses
• have support groups like: ‘Why is cheese so expensive in Japan?’
• add cheeses they’ve eaten, for example through IM or mail
• upload/import pictures of their cheese related experience
• provide little cheese diary entries for the real fanatics

• using Twitter to add and rate cheeses
• using Flickr to upload/import pictures about cheeses
• using Tumblr to provide journal functionality

We could use one of those create-your-own-community sites like Ning, but than we can’t build our core business around it: selling and recommending cheese based on their attention profiles. Fulfilling all cheese needs to our customers is why we exist.

Mashing up a service like this using Twitter, Flickr and Tumblr is easy. Turning that mashup into a success is nearly impossible. The problem lies in the shallowness of the way we do mashups now. People need a Twitter, Flickr and Tumblr account to use all functionality in our system. Explain that to the 60 year old French cheese farmer who just plugged in his DSL modem!

Now, I don’t want to start debating about web savvyness and targeting certain markets. My main point is that there is a certain flaw in all API’s: You need to register with the API provider in order to make use of the API anywhere.

• our cheese shop is using OpenID
• our cheese shop can create a new OpenID easily
• the services we integrate with are using OpenID
• the services allow first-time use of their service without ANY burden to our customers (truly seamless)

I think OpenID is the path we are moving towards, although I’m sure that at first most services will not be prepared for the seamlessness we want. Also, I have an interesting scenario for the case that OpenID doesn’t take off (will save that one for later).

Services will be able to focus on their core added value by specializing. Building things that have been build before is an extra baggage that seamless integration can fix. Also, when a service continues to specialize it needs to be thinking from the very start about it’s own integration in the collective. In our Cheese example, our service should have an API from day 1.

But right now, most services are not ready for this seamless integration. If FlickR would allow use of their API without showing any FlickR logos or Flickr ads, how would they make money? What if we want to display our own CheesR logo everywhere and completely exclude the user from knowing FlickR is used?

This is where Freemium Integration is born.

Right now, most popular services offer API’s. If you look carefully most of them don’t guarantee any service with the exception of Google’s. API’s are in their very early stage.

Soon, API’s will become the core of any successful online service. They will be the door to most of the revenue. Services will continue to specialize in the things they do well, recommending wine, recommending books or providing entertainment you appreciate. This specialization might go on to a point where there will no longer be an actual site, but just a widget, API, Facebook Application or some other integratable entity.

These integratable entities – that I will just call API’s for now – will have several ways of generating revenue for their creators.

• Free Integration. This could be either seamless or semi-seamless integration together with in-content contextual advertisement. In order to make these advertisements contextual there will need to be some kind of Attention Profile exchange between the two parties.
• Premium Integration. Seamless integration without brands or attention debt. The client-party will have to pay the provider an Amazon AWS like compensation for usage.

I think over the next short-term march towards the first intelligent agents, it is important to keep the API in mind. Make sure that your online service can be used with or without OpenID and that third parties can integrate seamlessly without any hassle. This also means changing your mindset towards branding since this is a concept likely to change very rapidly soon.

The purpose of this article is to spark discussion from the community. Have any of you encountered a similar discussion or desire for seamlessness? Any opinions welcome!

Yesterday I hit the Digg front page with my Building a .com in 24 hours article. Apart from that I was also number one and number nine in the delicious rankings. Very exciting!

What is also interesting is that my rapidly build website – wigitize.com – got a lot of hits. My blog got about 25000 uniques, Wigitize about 10000. All these hits caused an incredible load on my small 256MB memory server so I quickly bought a second one and scaled up. Yesterday, I could finally stabilize the backgrounding on the website and everything is running fine ever since. Someone offered me a free slicehost slice too last night!

It’s also funny that people commented on how I must have too much time on my hands. I even got offered participation in startups and companies haha. In fact, the opposite is true, but I do love hacking away in my free time. Besides, it was xmas, that means coding, drinking and eating right?!

While combating the performance issues I had yesterday, I learned these technical lessons:

Make sure you’re running the right version of tools. Apparently I was running a very old backgroundrb version. So I now upgraded to Ezra’s 0.2.1 version

Debug things in production-mode on your local machine! Some code – especially funky code like backgroundrb – will behave differently on production as opposed to dev. Often people like me are strictly developing in ‘development’ mode and than publishing to staging/production environments. I found a mayor issue that was screwing up everything in backgroundrb by debugging things in production mode on my MacBook.

1

/<link.*href=['"]*([^\s'"]+)['"]*.*application\/rss\+xml.*>/.match(html)

1

/<link.*href=['"]+([^'"]+)['"]+.*application\/rss\+xml.*>/.match(html) 

Update: Last 18 hours I have been combating performance issues because I got digged. Things seem to work more stable now, but still the background processing fails on me sometimes. Good lessons are being learned here so I will wrap that in a nice article soon!

Update: I just added another 1 GB slice, that should get us going for a while. Now just have to wait for the damn DNS entries to update.

I am preparing a new machine as we speak. Wow, did not to run into these luxury problems already! Thanks for the support!

If anyone thinks of buying a slicehost.com slice, please use this referral URL to donate:

https://manage.slicehost.com/customers/new?referrer=129028580

Machine 1 (256mb memory, how I started out with 2 days ago):

Machine 2 (1024mb memory) new

This is about how I spend 24 concentrated hours spread out over 4 days during Holidays to build the online service Wigitize.com. It is part of my ongoing learning process on how to run a successful web startup.

Even though I’m a super pimple-faced code-geek, I strive to be a creative entrepreneur who can utilize modern day tools and navigate the chaos to build cool stuff. What I tried to do for this project is use some new methods/tools out there to solve practical problems in my weakness area: design, frontend coding, system administration and SEO.

Purpose of this article is to show my thought process on the multidisciplinary aspects of this project. Also to invoke the discussion on how these things could be done much better (correct me!) and hopefully also to educate other entrepreneurial minded hackers.

Wigitize.com, A geeky Idea

While working for my current client here in Tokyo, I’m often running into problems where I wish there was a third party web service that could solve them. For example: Uploading/Managing pictures for user generated content. We want to build cool shit, not reinvent the wheel.

Another one of these ‘problems’ – by saying problems I mean ‘important new features’ – was allowing users to import their blog’s RSS feed. When users input the URL of their blog, a listing of their most recent blog entries will be displayed, sounds simple enough right?

Several weeks before, I already wrote a feature that allowed users to display their latest Twitter ‘tweets’. Building this ‘passive twitter integration’ seriously did not take any longer than one hour! This is because Twitter provides a blog badge or widget. More and more sites are starting to provide these widgets making it easier for people to take their data and display it on their blog. And that’s great!

However, the most popular data feed format – RSS – does not have these widget benefits. Widgets rely on a smart technique called JSON which allows your browser to fetch the actual data. With RSS feeds you need relatively complicated server-side processing to display the data.

So wouldn’t it be cool if there was a web service that allowed you to convert any RSS/Atom feed into a embeddable widget? I’m sure some of you will suggest similar services now :]

A service like that should be:

• simple, with one big ass URL input
• smart, it should detect these data feeds
• integratable (another adjective!), by providing an API for other web services

Aahhh a new project has been born. Let’s open up MS Word and fill in our Prince 2 Project template. Herein we can properly plan for all things we need to be doing and all risks and…

Neh, just kidding! Let’s open up that bottle of wine and… just fucking build it!™

Design Mockup, 3 hours

For me, this is one of the more tricky parts. But fortunately, the project is simple and Web 2.0 designs simple too! Since I don’t aspire to be the next whoever cool designer, I will just give you my Idiot’s view of doing design:

• being able to draw is not a necessity
• always start by putting down your content first
• learn a bit about colors: Color in Motion, Vleere and Kuler (not cool but useful)
• don’t force making a design, do it in the evening when you find the creative flow – or in an onsen if you have a waterproof mac

The weapon of choice: Adobe Photoshop (CS3, the latest shizne!). (Any good web alternatives yet? :) They say Photoshop has a steep learning curve and I agree. Still there are some things you can do to speed things up:

Learn the most important shortcut keys, you will use them a hell lot (ordered by usage):

• V, get the arrow key to start dragging shit around
• M, marker to select things
• T, text tool
• I, eyedropper to pick a color
• CTRL+T, transform current selection
• G, paint bucket to fill in stuff
• W, magic wand to select stuff of the same shape/color

There are some things you will be seeing a lot in particular when doing Web20-like designs:

When performing blending options on a layer, you should be playing a lot with the following marked options:

You can learn here and here how to make these babies:

Woops, I shouldn’t get to deep into the details. Anyway, based solely on the content a design like this rolled out:

The colors can be justified as soil and nature, yin yang, peace man!

Storing your Files in an online Project Manager, 30 minutes

After having made those designs, I’d like to throw them online somewhere right away. Preferably in a SCM repository like Subversion. Luckily, I remembered an article on ReadWriteWeb about such Developer Tools

Basically there are two startups that provide freemium SVN and project management hosting. I chose Assembla for disk space, but Unfuddle was a close second.

That said, 10 minutes later I could start throwing things into my very own http://svn2.assembla.com/svn/wigitize/trunk

Hell, it even puts changeset notifications in my Junk box!

Playing with Rails 2.0, 4 hours

Weapons of choice: RubyOnRails 2.0 and Textmate.

Note: If one piece of software is worth paying for, it’s TextMate – am doing so now

RubyOnRails is very suitable for me, but I’m not sure if it is that suitable for you. I myself have a clear CS-engineering background and am very comfortable with digging in deep. On the other hand, depending slightly on luck, you can do a lot with this framework by just modestly hacking away and watching a screencast every now and then.

Rails is quite controversial in terms of scaling and production-ready. However, I think things are changing fast and as I will show you later. Serious innovation is done on how this framework plays with it’s systems layer and technologies like Amazon’s Elastic Computer Cloud

Like making the design, the coding also starts with content. First you jack in the important texts/inputs you have in your design and than you enclose them in the necessary divs so they can be CSS-styled later. In Rails this starts with making a home/welcome controller and writing the default layout: app/views/layouts/application.html.erb. Yes we are using the new Rails 2.0 way of html.erb.

A very important part of Rails I think is the config/routes.rb file. This file holds all your pretty URLs like: http://wigitize.com/json/for/http://dominiek.com/ . In a way routes.rb lays out all the abstract functionality of your web application. Rumors are that the routes part of the framework is internally the most complex one.

Some quick hinters on Rails practices that will speed up things:

• Never do things like: link_to(’’, :controller => ’’), use Named Routes
• If something seems complicated, break it down into simple steps and write a unit test first
• Breathe consistentness, everything you code – even whitespace – has perfect reason. This also means applying Don’t Repeat Yourself all the time as a habit.
• Naming can be a bitch and you can take up a lot of time. Do your best and pick a name consistently, you can always change it later.
• In Rails, take Skinny Controller, Fat Model to extreme heights, it will make your life more easy
• Do it in AJAX right away, it’s often much more simple than supporting plain HTML CRUD in the first place

There is a lot of material out there available on coding Rails in general, so I will let go here.

These first two hours are basically setting up all the routes, filling in the HTML and making things work in a very basic sense. For wigitize that meant:

• hooking up the URL input to a feed detector and aggregator (most of that code is from here)
• making sure that the aggregated data is stored in JSON format (In rails that means calling .to_json on any Object, easy as pie!)
• adding a Widget model that can actually hold the URL, detected feed URL and JSON data:

I spend the next 2 hours seriously code-monkeying on the feed detecting and parsing part of the system. I will soon open source it under the name feedeater.

Style that HTML! 1 hour

Styling, already? Yes, I think it’s good to style quite early in the process. For me, there is one single argument: Flow.

When you are making something work it is nice when things already look quite tangible and usable. When you apply styling in an early stage you can see direct usable results of the things you are building, increasing the psychological state of the Flow.

Ok, so I also kinda suck doing CSS, but I learned enough to turn an image into a web page. These are my bullet point lessons:

• Develop for Firefox first, using the tool Firebug. If you aren’t using Firebug and doing web development you’re either slow or an imbecile ;]
• Always put in some basic CSS, I got some here and made it into my own html.css which I can include as a base all the time.
• Padding and margin: These things are great and you need a lot of them. However, padding often gives you shit so try to choose margin over padding. Also, choose margin-bottom over margin-top since everything tends to float upwards (Thanks Simon)
• Current fashion: Get some nice fonts going on, I’m using Trebuchet MS a lot for Wigitize and I mix it in with plain Arial.
• Current fashion dictates that you use a less fierce black for your text, make it #111 or #222.
• For mozilla/safari I’m using -moz-border-radius/border-radius, will never work in IE6 – fuck them, being a plain user is fine, but you’re not getting any round corners! Besides, isn’t Microsoft planning a forced upgrade soon?
• You only need tables for tables. They will make your life a pain and you will not be cool if you use them for layout. Other than that it’s fine.
• Little side note about AJAX: don’t use AJAX for navigation EVER! There are strong SEO and usability arguments against it (learned that the hard way, like most things in my life).

Spin it Baby! 2 hours

OK, I must admit that I spend way WAY too much time on this! However, when you’re doing things with AJAX, you need to put a spinner somewhere to indicate loading.

This so called spinner is pretty kickass and based on a piece of Javascript I wrote earlier:

Yeah Baby! It’s all hypnotic and stuff.

Note: As someone pointed out in the comments, on ajaxload.info you can get a lot of spinner images, like the one I’m using here.

Designing and Coding the Footer, 2 hours

I really like footers and I think they are becoming more and more important. Nowadays footers are used as sitemaps and often they are contextualized as well. These are nice examples of footers: last.fm and snooth.com

For Wigitize the footer is rather small since it’s a small site:

Making the JSON Embeddable, 3 hours

<div class="feed_widget">
  <ul id="feed_widget_34"></ul>
  <script type="text/javascript" src="http://wigitize.com/javascripts/wigitize.js"></script>
  <script type="text/javascript" src="http://wigitize.com/feeds/34.json"></script>
</div>

This works in three steps:

• define a containing list (ul)
• include a JS library that has a special callback function, in our case wigitize_feed()
• include the JSON file that will call wigitize_feed with the appropriate data

Providing styles obviously complicates things a bit. When choosing a style, it will include a generic wigitize.css and assign a class to the containing div.

Near future improvements:

• Provide a ‘grab the grabber’ so that people can provide widgets of their own feeds (eg ReadWriteWeb providing a last-10 articles widget). This could have a lot of potential if provided in a simple feed-burner like button.
• Put in better default styles than the lame ass ones I have now.
• Option to include data (useful for photo feeds).
• Option to display other kinds of aggregated data, eg microformats.

Making things run in the Background, 3 hours

Running things in the background – dubbed backgrounding – is an important part of production ready web applications. It’s a relatively new concept, since web applications used to be less complex. Now however, we are marching towards the Semantic Web where web apps are expected to become intelligent (the Intelligent Agents are coming, just like in the Langoliers!). I think being able to make your applications autonomous now will already reap you benefits (to be continued as an article).

There are several ways to achieve backgrounding in Rails, but the far away most easy one is using BackgrounDRb. BackgrounDRb is a plugin for Rails that allows you to easily kick off background processes and schedule regular tasks. Good for our purposes: detecting and fetching feed data.

I don’t agree with all of Zed Shaw’s big rant about the Rails community being a ghetto, but I sure do agree that there are a lot of idiots out there that produce things that can screw over your production apps. BackgrounDRb has become one of these projects and I strongly recommend that you do not use the latest code. If you start comparing code and read the mailing list you will see that a new guy has taken over Ezra fine project and I suspect that he has lowered the project’s level to pre-stable. I don’t hear any signals from the community and that worries me. Either I’m seeing ghosts or people are blindly accepting anything that’s marked as stable. In any case, I’m using Ezra’s version which works fine.

Finishing up the API, 4 hours

The Wigitize API for now is quite quick and dirty. There are two simple ways of using the API and examples are provided. In this area, a lot of improvements will be made down the line since it’s a key point in making any future freemium revenue.

Domain and Domain Email, 30 minutes

When I buy a domain I always buy a DNS managing package with it as well. This means that I can login somewhere and setup subdomains and set mail records. So the total price of Wigitize.com was 20$ per year.

Providing info@domain.com email is easy, just get yourself a free Gmail for organizations account. In your Google Apps domain manager you can simply add your domain and in your DNS tool you set the MX record to ASPMX.L.GOOGLE.COM. Now you can use gmail and IMAP to read mails sent to your domain.

Setting up a Production Server, 1 hour

I was really eager to put this project on Amazon’s Elastic Computer Cloud until I calculated my monthly costs. Running costs alone for simple projects like these will cost you 60$ a month. Still I think it’s worthy to look into this once you scale beyond simple project.

After digging around I figured out that slicehost.com would be a good cheap second. For 20$ a month I have a 256 memory slice with 100GB in data transfer – awesome.

Setting up your slice takes 5 minutes with a credit card. This slice is essentially a virtual machine with an IP address, completely yours. And the best thing: you get an awesome web console to control everything! Adding a new machine is a no brainer.

Now, I’ve used linux/unix for a long time as a working station. Eventually I got lazy and switched to my current MacBook. Fortunately, you can be lazy for the systems side of things too. All thanks to a lovely tool called Deprec.

Deprec allows you to install the complete Rails stack with a small set of commands. Shortly thereafter you can deploy your application to your production server by typing cap deploy_with_migrations. Please note that for Deprec you need to install Ubuntu Linux on your machine which you can do as follows:

Deprec installs the Rails stack as: Apache, Mongrel cluster (default of 2 instances) and MySQL. As I’ve written earlier, NginX is a nice nano-alternative for Apache. I would like to see that in my Rails stack someday, but I’m not going to worry about that now. Clock is ticking!

Little Pimps and Tweaks, 3 hours

I think it’s good to prepare a little bit for the storm (and I felt like doing something else for a bit), so I’ve created a nice maintenance message for in case there are system/scaling problems. In here I think it’s important to give people an extra reminder to bookmark and come back.

Which brings us to another great service, addthis.com. AddThis provides you with a button that makes it easy to bookmark on multiple social bookmarking services.

Another one of those little tweaks was proper error checking and displaying it to the user. I am using a pink error message to make it look more friendly (maybe I should go even further and make it yellow or something):

Statistics and Search Engine Optimization, 1 hour

Statistics is another 10 minute no-brainer by using Google Analytics. However I’m on the lookout to find something more real-time like Mint (but than free). Any suggestions?

Note: I just added getclicky.com to get more realtime stats than GA

I found a blog which is solely about Rails and SEO which I thought was very promising but in fact doesn’t have much content. I do found something on how to provide different meta tags in rails which I applied right away.

After looking around a bit I also saw some discussion about whether to use www. or not. The way to roll with this is: you permanently redirect your www. domain to say http://wigitize.com. It makes sense, www is old and architecture centric, http://wigitize.com is less typing and pretty.

On that note, make sure that you always write pretty URL’s by practice. This means thinking them through extra carefully, because changing URL’s is painful after going live.

Wigitize.com was also chosen with SEO in mind. The word is a mis-spelling of the term widgetize and yields 105 Google results (at this moment). Additionally, it is a verb, which lubricates the prettiness of the URLs :]

• Making a list of ‘last wigitized sites’ and ‘most wigitized sites’, those pages can be accessed by the search spiders and thus associating external content with Wigitize.com. I think this might improve search rankings.
• Providing a sitemap.xml, would that help?
• Focus on the viral aspects of these widgets. For example a FeedB urner style button for on popular blogs.

Let’s throw it out there!

While writing this article – which took quite some time. Wigitize.com is already running and doing it’s job. However, I’m sure there are still some kinks to work out which I will do over the past coming days (eg IE6 support, SEO/viral tasks).

Also, I didn’t discuss anything about an important aspect: How to make revenue with Wigitize.com? I’m not sure yet, but since I’m solving a problem for myself, I’m sure others out there had it. Besides, the costs are extremely low at this point so I will worry about monetizing later. Although I would like to hear YOUR thoughts about it!

I realize that this is a geeky project and I must say it’s quite different than the web apps I normally work on. It was fun for me however to write down my thought process, especially on the non-tech parts of building which I find increasingly interesting.

Hey, I think I’ve done this before :] A new blog design for the new season. Still not very colorful, but a lot more focussed on the actual content.

This is all in preparation of a series of articles I will write about:

Wigitize.com – Building a Production Ready .com in 5 days

BackgroundRB was a nice project that allowed you to easily do background jobs in Ruby and Rails.

Now, it is maintained by some PRICK who is rewriting everything making it IMPOSSIBLE to understand. (Maybe Zed Shaw has a Point?)

Today and yesterday I’ve spend another couple of hours coding and designing away on my (autistic) little project Wigitize.com. I’m keeping an exact log of the amount of hours I’m spending and what’s happening so I will publish some full-exposure articles about how to build a project like this.

Currently these are the things that still need to be done:

• enabling people to choose a style for their widget
• write a simple API and a page to show how it works
• optimize backend so that a background worker will be used
• allowing people to ‘grab the grabber’ so that they can offer widgets of their own content to their users
• renting a machine with bandwidth, I’m thinking of slicehost.com (much cheaper than EC2)
• rethinking the name WIGITIZE. It sounds good to my Dutch-English, but it might not for Natives.

Another little sneak preview:

click image to expand

I think it's getting more and more easy to start up your own knowledge-driven venture. For a couple of years now, I have been managing my own server (together with my friend). Managing your own server is a pain, but a must if you are a tech entrepreneur.

Luckily, lately there have been services popping up that alleviate a lot of IT-pains. This week, I have been looking for services that allow me to externalize the hosting of my Email service. A few services like SproutIt's Mailroom came close, but didn't quite cut it.

Then, I found out, that I can actually attach my domain names using Gmail! Also, I can use IMAP, POP etc. I now have most of my domain names attached to it, and I pay JACK SHIT!

All thanks to another great article from ReadWriteWeb.