some random thoughts I’ve been putting down on a writeboard

Recently, I discovered a cool feature on my Macbook. If I hold the control key and scroll with my mousewheel/pad – I can zoom in and zoom out very smoothly. I’ve started using this a lot and it’s become part of my digital habits. I do it that often that colleagues almost think of me as the crazy zooming guy. This full screen zooming is good for many things during my normal productivity hours. I often use the zooming to show people certain parts of a User Interface and sometimes I use it to zoom in on a piece of text to make a statement :]. Also, it’s useful for pair programming. Now, partly because of my deteriorating eyesight, I can zoom in on something and sit back. This could be a word or a picture and while I sit back, I can really think about it. Zooming-in gets rid of all the other chaos on the screen and helps you to focus.

I guess the zooming metaphor applies to many things in life. As work – trained as an engineer – I deal a lot with the nitty gritty of things. I used to program a lot of low-level stuff like Assembly and C code. However I’ve done some more abstract technical things over the past years like: programming in a variety of languages, design patterns and thinking about user interaction. So far, I’ve been apt at being able to zoom in and out in a certain scope of my skill-sets. Most of it has been technical or related to building software. Right now I’m trying to extend my professional zoom-scope to extend areas that require right-brain thinking like design and business/strategy. Also in personal life – which is heavily intertwined with that old term ‘professional life’ – you can apply zooming very well. Zooming-in means getting short term things done, like changing the light bulb in the toilet. Zoomed out means planning ahead and equally important: changing one’s habits.

Another brother of zooming is filtering. Filtering out the irrelevance in today’s information overload is already visible in multi billion dollar industries and in a notable company called Google. The coming semantic wave – or web 3.0 if you will – stresses the importance of user interfaces that deal with the abundance of information. In recent web design trends it’s easy to spot the shift towards information abundance. Websites now have big fonts for important summarizing one-liners whereas old websites i.e. portals tried to cram as much information on the screen as possible. Zooming and filtering will be important metaphors in future User Interfaces. Google Maps and the iPhone browser are notable first examples of these so-called Zooming User Interfaces (ZUI’s).

Note, while working on this project this ReadWriteWeb article was released, illustrating the future potential of the Jabber/XMPP protocol.

In this article we will build an actual useful Twitter Service that will allow us to track the Blogosphere. In the process we will get hands on programming experience with Ruby, DRb, Twitter and Jabber. This will sharpen our developer skill-set to get ready for the upcoming (Folk)Semantic Web. Also we evaluate the problems seen and opportunities ahead.

Background

Whether you want to call it Web 2.0, Web 3.0, the Semantic Web or the Web of Data – change is happening. The past years we’ve seen the tremendous power of Folkosonomies and now this Social Web is colliding with the emergence of the Semantic Web, resulting in the first semantic services. For us developers and creative entrepreneurs it’s important to get ready for this new wave of business opportunities. I find the whole notion of Intelligent Agents very interesting. For our little project however, we will create a Stupid Agent :]

Technologies like Jabber/XMPP and DRb will enable us to move from a reactive web to a proactive web. Right now this proactive realtime push of data is important for the more liquid content creation services. Micro/nano blogging platforms as Twitter and Tumblr are good examples of this. This is one of the reasons that they already have a Jabber service set up.

I’ve had used Jabber before to communicate with my geek friends. For this project, I had to set up a Jabber client and Jabber account. Call me stupid, but I actually had to spend 30 minutes figuring out how the hell I had to create an account and choose which server to use (turns out you can do that in the client). Now of course XMPP/Jabber is just a standard for enabling IM communication, but apart from Google’s GTalk there hasn’t been much widespread use by ordinary users. In my view, these uses of XMPP for machine to machine (to human) are much more interesting.

Case: The Observatory Bot

I’ve been programming for quite some time now. When learning a new language I really hate doing little examples that produce zero user/business value. That’s why I think the best way to learn new technologies is to solve real world problems right away. Of course tutorials are valuable to just get a general idea of what’s going on, but don’t waste too much time on them – implement straight away.

Our Twitter Service will also need to create some value for the user and must be production ready. However, don’t get your hopes up too much since these are experimental technologies with dependence on external services like Jabber and Twitter.

Remember my last little project? Wigitize.com is actually generating a lot of data. It’s tracking about 5000 6000 feeds every hour! Let’s do something with that data :]

• is a Twitter-only service
• will allow you to ‘track’ the Blogosphere
• will send you a direct message when something happens in the Blogosphere

Basically this is like Twitter’s IM functionality to track the Twittersphere. So in a sense The Observatory will be a proof of concept portal between the Twittersphere and the Blogosphere.

The Architecture

Right now Wigitize.com uses BackgrounDRb to perform background tasks and also to update all RSS feeds periodically. Everytime the feed aggregation process finds a new feed entry it will create a FeedEntry instance. The creation of these objects serve as events for the Observatory Bot. These events have to be pushed to the Observatory Bot in some way.

What can Twitter’s IM service do for us?

To play around with Twitter’s agent you need to set up a Jabber account and a Twitter account. For debugging I’ve found the MacOS tool JabberFox very helpful.

Basically, these commands are available:

However, I think there are a lot more hidden commands which can be used. After emailing with the Twitter developers they told me there is a command called “d”. This can be used to send direct messages (d username message). Very useful!

Coding the Bot

Our implementation choice for today will be Ruby. If you’ve programmed intensively in other languages before you’ve probably come to the conclusion that Ruby is quite different from most other. Ruby’s flexible object models allow for great extension of the language itself (eg 3.minutes.ago). It is therefore no surprise that interesting Semantic Web projects like ActiveRDF are choosing Ruby as their language.

To communicate with Twitter we can use this cool Ruby Twitter API. Unfortunately all of these API’s are HTTP/REST driven and really limit what we can do in terms of realtime response. Also, if you want to build a serious production ready service, constantly polling Twitter will kill both parties.

So we need to interface with their Jabber Service. Jabber is a friendly name for Instant Messaging (IM) using the open XMPP protocol. Luckily, there is a Ruby library called XMPP4R which does most of the XMPP work for us. This blog post provides some simple examples and this German wiki entry provides sample code how to use callbacks (very important for a bot).

I’ve wrapped all of this in a simple JabberBot class jabber_bot.rb that can be used like this:

1
2
3
4
5
6
7
8

  class MyJabberBot < JabberBot
    def on_message(from, body)
      say(from, "You said: #{body}")
    end
  end
  my_jabber_bot = MyJabberBot.new('observatory@jabber.org', 'password')
  my_jabber_bot.connect_and_authenticate
  my_jabber_bot.run

As you can see in the diagram, I’ve build a TwitterBot on top of this JabberBot. Unfortunatly it’s not possible to do all communication with Twitter through Jabber yet. For example: there are no events for when users start following other users or ways to retrieve information. This is why twitter_bot.rb is essentially a hybrid using both the Twitter API and Twitter’s Jabber service. Feel free to use all sourcecode provided here, I know it will be useful to some of you out there. This is how to use this TwitterBot:

1
2
3
4
5
6
7
8
9
10
11
12

  twitter_bot = TwitterBot.new('observatory', 'password', 'observatory@jabber.org', 'password')
  twitter_bot.track_phrases = ['observatory.topoints.com']
  twitter_bot.on_directed_tweet do |username, message|
    puts("directed tweet: #{username} says #{message}")
  end
  twitter_bot.on_tweet do |username, message|
    puts("something from #{username}: #{message}")
  end
  twitter_bot.on_track do |username, message, phrase|
    puts("track: #{username} says #{message} (keyword: #{phrase}")
  end
  twitter_bot.runn(:follow_all_followers => true)

Now that we have the basic building blocks to build our service, let’s build our core business logic (observatory_twitter_bot.rb):

This means that we will send a greeting when people start following us:

1
2
3
4
5
6
7
8

  on_follow do |username|
    logger.info("#{username} is following us, will follow #{username} too and send welcome message")
    follow(username)
    direct_message(username, "the Observatory is now ready to serve you, use '@observatory track [keyword]' to get blogosphere updates.")
  end
  on_unfollow do |username|
    logger.info("#{username} stopped following us")
  end

Note: in order to get the on_follow event, we have to poll the Twitter HTTP API . Since Twitter limits the rate to 70 requests per hour, I poll every two minutes to be on the safe side.

And that we will start tracking the Blogosphere for them when they say the magic word:

1
2
3
4
5
6
7
8
9
10
11
12

  on_directed_tweet do |username, message|
    logger.info("directed tweet: #{username} says #{message}")
    if (phrase = track_phrase(message))
      logger.info("tracking '#{phrase}' for user #{username}")
      begin
        direct_message(username, "Will send a direct message anytime something happens in the Blogosphere regarding '#{phrase}'")
        Tracker.for(username, phrase)
      rescue => e
        logger.error("tracking failure: #{e.to_s}")
      end
    end
  end

Now when a new FeedEntry is created, we need to make sure that these Twitter users get notified when their tracked phrase matches the FeedEntry. Since this might take up some time, I’ve created a background worker task for it:

As you might see, Distributed Ruby (DRb) makes it extremely easy to control our bot remotely. In the ObservatoryBot we say:

1

  DRb.start_service("druby://:8997", self)

And all bot functionality can be accessed by calling: observatory_bot = DRbObject.new(nil, ‘druby://:8997’)

Now that we have our autonomous agent it would be nice if we could easily start and stop it in a production environment. I found the Ruby Gem called Daemons extremely useful to wrap these things up.

First, set up a file that runs the never ending process (eg script/observatory_twitter_bot.rb):

1
2
3
4
5
6
7
8

require 'logger'
require File.dirname(__FILE__) + '/../config/boot'
require File.dirname(__FILE__) + '/../config/environment'
require File.dirname(__FILE__) + '/../lib/observatory_twitter_bot'

logger = Logger.new(File.join(RAILS_ROOT, 'log/observatory_twitter_bot.log'))
observatory_twitter_bot = ObservatoryTwitterBot.new(logger)
observatory_twitter_bot.runn

Next, wrap this up in a daemon script (eg script/observatory_twitter_bot):

1
2
3
4
5
6
7

#!/usr/bin/env ruby
require File.dirname(__FILE__) + '/../config/boot'
require 'rubygems'
require 'daemons'

Daemons.run('script/observatory_twitter_bot.rb')

The Demo

Right now if all communication lines with Twitter are working fine, the service is up and running. I’ve made a little bot homepage at observatory.topoints.com

@observatory track ‘Twitter’:

Problems and Opportunities

In the development of the Observatory I had one big obstacle: Twitter is often down and it can cripple your service and development time. I understand that Twitter is a small team under enormous pressure but there have been a lot of complaints about this.

Nevertheless Twitter and it’s developers really kick ass. I emailed Alex Payne and he was excited about what I’m doing (and also the Twitter things happening the iKnow! project in Japan). He responded fairly quickly and immediately whitelisted my Twitter account to up the rate limits.

While working on this project I realized that Twitter in it’s current state isn’t really suitable for system-to-human notifications. Twitter could expand their system to be a true notification framework, but I’m not sure if they will. If they don’t, there is a tremendous business opportunity here. Imagine an open API that mashes up with technologies like XMPP and Growl. A service like that could become THE notification-bus of the web! (Already Growl is pretty big in Mac land). A full blog post about this startup idea coming up!

Geek Food for Thought

What about RubyOnRails, Jabber, DRb, Daemons and BackgrounDRb? I think we are seeing a new framework here! In this interesting article by Danny Ayers he talks about a toolset for agents. On Java there is already “an Agent Framework” that I haven’t checked out yet. I can imagine that these frameworks ease doing development like this and facilitate better system autonomy. Of course it’s desirable to give such a framework a pragmatic paintjob by using Rails paradigms.

Above here I’ve illustrated Rails’ missing brother. I think it’s also good to take into account interesting technologies like Juggernaut and Comet. These are basically Javascript Push techniques to make a synchronous interaction on the asynchronous web possible.

When you combine all these micro- asynchronous communication lines you get one big synchronous connection line between machine agents and user agents.

Note: hashtags.org has banned me for spam and I understand. Sorry guys! The experiment was stopped as of 11:00 JST

Last night I did a little art experiment: What happens when you use a service like TwitterFeed.com to feed your Twitter status updates back into itself.

• created a user called loopyloop
• used twitterfeed to update both the RSS and Atom feed
• used the great service hashtags.org to keep track of the activity

I was hoping to see some interesting emergent behavior, but the end result was a simple update every half an hour with 5 updates.

I do think it is interesting and important to think about full-artificial organisms living and looping through our cyberspace.

This is a more extensive writeup of a problem I have talked about before. It’s about something many of us web innovators face today. Like many problems, they can be turned into opportunities if implemented well.

The new Web – let’s stop using 2.0 – is much more than social networking sites and round cornered layouts. As Tim O’Reilly said, people who think Web 2.0 is about AJAX are completely missing the boat. This is true, but it is important to keep in mind that the shift from desktop products to web services is a big deal. Online services are already gaining a lot of productivity for people and thanks to the characteristics of the web the companies developing it can distribute and evolve much better. So this means that certain value adders have now migrated to the web.

What is happening now is that many of these online services want to be able to harness the power of the new social characteristics of the web. People have their friends online now and there is a lot of activity between those people and services (UGC). Everybody wants to tap into this attention stream between people. It helps contextualize your offering and helps spread the goodness between like minded.

Unfortunately, getting this social activity going around your service is not easy. And frankly, most of the time it results in creating another lousy SNS with worse social features. Fortunately, most online services open up their API’s allowing you to integrate with their services to create a mashup.

• rate, review and discuss cheeses
• have support groups like: ‘Why is cheese so expensive in Japan?’
• add cheeses they’ve eaten, for example through IM or mail
• upload/import pictures of their cheese related experience
• provide little cheese diary entries for the real fanatics

• using Twitter to add and rate cheeses
• using Flickr to upload/import pictures about cheeses
• using Tumblr to provide journal functionality

We could use one of those create-your-own-community sites like Ning, but than we can’t build our core business around it: selling and recommending cheese based on their attention profiles. Fulfilling all cheese needs to our customers is why we exist.

Mashing up a service like this using Twitter, Flickr and Tumblr is easy. Turning that mashup into a success is nearly impossible. The problem lies in the shallowness of the way we do mashups now. People need a Twitter, Flickr and Tumblr account to use all functionality in our system. Explain that to the 60 year old French cheese farmer who just plugged in his DSL modem!

Now, I don’t want to start debating about web savvyness and targeting certain markets. My main point is that there is a certain flaw in all API’s: You need to register with the API provider in order to make use of the API anywhere.

• our cheese shop is using OpenID
• our cheese shop can create a new OpenID easily
• the services we integrate with are using OpenID
• the services allow first-time use of their service without ANY burden to our customers (truly seamless)

I think OpenID is the path we are moving towards, although I’m sure that at first most services will not be prepared for the seamlessness we want. Also, I have an interesting scenario for the case that OpenID doesn’t take off (will save that one for later).

Services will be able to focus on their core added value by specializing. Building things that have been build before is an extra baggage that seamless integration can fix. Also, when a service continues to specialize it needs to be thinking from the very start about it’s own integration in the collective. In our Cheese example, our service should have an API from day 1.

But right now, most services are not ready for this seamless integration. If FlickR would allow use of their API without showing any FlickR logos or Flickr ads, how would they make money? What if we want to display our own CheesR logo everywhere and completely exclude the user from knowing FlickR is used?

This is where Freemium Integration is born.

Right now, most popular services offer API’s. If you look carefully most of them don’t guarantee any service with the exception of Google’s. API’s are in their very early stage.

Soon, API’s will become the core of any successful online service. They will be the door to most of the revenue. Services will continue to specialize in the things they do well, recommending wine, recommending books or providing entertainment you appreciate. This specialization might go on to a point where there will no longer be an actual site, but just a widget, API, Facebook Application or some other integratable entity.

These integratable entities – that I will just call API’s for now – will have several ways of generating revenue for their creators.

• Free Integration. This could be either seamless or semi-seamless integration together with in-content contextual advertisement. In order to make these advertisements contextual there will need to be some kind of Attention Profile exchange between the two parties.
• Premium Integration. Seamless integration without brands or attention debt. The client-party will have to pay the provider an Amazon AWS like compensation for usage.

I think over the next short-term march towards the first intelligent agents, it is important to keep the API in mind. Make sure that your online service can be used with or without OpenID and that third parties can integrate seamlessly without any hassle. This also means changing your mindset towards branding since this is a concept likely to change very rapidly soon.

The purpose of this article is to spark discussion from the community. Have any of you encountered a similar discussion or desire for seamlessness? Any opinions welcome!

This is about how I spend 24 concentrated hours spread out over 4 days during Holidays to build the online service Wigitize.com. It is part of my ongoing learning process on how to run a successful web startup.

Even though I’m a super pimple-faced code-geek, I strive to be a creative entrepreneur who can utilize modern day tools and navigate the chaos to build cool stuff. What I tried to do for this project is use some new methods/tools out there to solve practical problems in my weakness area: design, frontend coding, system administration and SEO.

Purpose of this article is to show my thought process on the multidisciplinary aspects of this project. Also to invoke the discussion on how these things could be done much better (correct me!) and hopefully also to educate other entrepreneurial minded hackers.

Wigitize.com, A geeky Idea

While working for my current client here in Tokyo, I’m often running into problems where I wish there was a third party web service that could solve them. For example: Uploading/Managing pictures for user generated content. We want to build cool shit, not reinvent the wheel.

Another one of these ‘problems’ – by saying problems I mean ‘important new features’ – was allowing users to import their blog’s RSS feed. When users input the URL of their blog, a listing of their most recent blog entries will be displayed, sounds simple enough right?

Several weeks before, I already wrote a feature that allowed users to display their latest Twitter ‘tweets’. Building this ‘passive twitter integration’ seriously did not take any longer than one hour! This is because Twitter provides a blog badge or widget. More and more sites are starting to provide these widgets making it easier for people to take their data and display it on their blog. And that’s great!

However, the most popular data feed format – RSS – does not have these widget benefits. Widgets rely on a smart technique called JSON which allows your browser to fetch the actual data. With RSS feeds you need relatively complicated server-side processing to display the data.

So wouldn’t it be cool if there was a web service that allowed you to convert any RSS/Atom feed into a embeddable widget? I’m sure some of you will suggest similar services now :]

A service like that should be:

• simple, with one big ass URL input
• smart, it should detect these data feeds
• integratable (another adjective!), by providing an API for other web services

Aahhh a new project has been born. Let’s open up MS Word and fill in our Prince 2 Project template. Herein we can properly plan for all things we need to be doing and all risks and…

Neh, just kidding! Let’s open up that bottle of wine and… just fucking build it!™

Design Mockup, 3 hours

For me, this is one of the more tricky parts. But fortunately, the project is simple and Web 2.0 designs simple too! Since I don’t aspire to be the next whoever cool designer, I will just give you my Idiot’s view of doing design:

• being able to draw is not a necessity
• always start by putting down your content first
• learn a bit about colors: Color in Motion, Vleere and Kuler (not cool but useful)
• don’t force making a design, do it in the evening when you find the creative flow – or in an onsen if you have a waterproof mac

The weapon of choice: Adobe Photoshop (CS3, the latest shizne!). (Any good web alternatives yet? :) They say Photoshop has a steep learning curve and I agree. Still there are some things you can do to speed things up:

Learn the most important shortcut keys, you will use them a hell lot (ordered by usage):

• V, get the arrow key to start dragging shit around
• M, marker to select things
• T, text tool
• I, eyedropper to pick a color
• CTRL+T, transform current selection
• G, paint bucket to fill in stuff
• W, magic wand to select stuff of the same shape/color

There are some things you will be seeing a lot in particular when doing Web20-like designs:

When performing blending options on a layer, you should be playing a lot with the following marked options:

You can learn here and here how to make these babies:

Woops, I shouldn’t get to deep into the details. Anyway, based solely on the content a design like this rolled out:

The colors can be justified as soil and nature, yin yang, peace man!

Storing your Files in an online Project Manager, 30 minutes

After having made those designs, I’d like to throw them online somewhere right away. Preferably in a SCM repository like Subversion. Luckily, I remembered an article on ReadWriteWeb about such Developer Tools

Basically there are two startups that provide freemium SVN and project management hosting. I chose Assembla for disk space, but Unfuddle was a close second.

That said, 10 minutes later I could start throwing things into my very own http://svn2.assembla.com/svn/wigitize/trunk

Hell, it even puts changeset notifications in my Junk box!

Playing with Rails 2.0, 4 hours

Weapons of choice: RubyOnRails 2.0 and Textmate.

Note: If one piece of software is worth paying for, it’s TextMate – am doing so now

RubyOnRails is very suitable for me, but I’m not sure if it is that suitable for you. I myself have a clear CS-engineering background and am very comfortable with digging in deep. On the other hand, depending slightly on luck, you can do a lot with this framework by just modestly hacking away and watching a screencast every now and then.

Rails is quite controversial in terms of scaling and production-ready. However, I think things are changing fast and as I will show you later. Serious innovation is done on how this framework plays with it’s systems layer and technologies like Amazon’s Elastic Computer Cloud

Like making the design, the coding also starts with content. First you jack in the important texts/inputs you have in your design and than you enclose them in the necessary divs so they can be CSS-styled later. In Rails this starts with making a home/welcome controller and writing the default layout: app/views/layouts/application.html.erb. Yes we are using the new Rails 2.0 way of html.erb.

A very important part of Rails I think is the config/routes.rb file. This file holds all your pretty URLs like: http://wigitize.com/json/for/http://dominiek.com/ . In a way routes.rb lays out all the abstract functionality of your web application. Rumors are that the routes part of the framework is internally the most complex one.

Some quick hinters on Rails practices that will speed up things:

• Never do things like: link_to(’’, :controller => ’’), use Named Routes
• If something seems complicated, break it down into simple steps and write a unit test first
• Breathe consistentness, everything you code – even whitespace – has perfect reason. This also means applying Don’t Repeat Yourself all the time as a habit.
• Naming can be a bitch and you can take up a lot of time. Do your best and pick a name consistently, you can always change it later.
• In Rails, take Skinny Controller, Fat Model to extreme heights, it will make your life more easy
• Do it in AJAX right away, it’s often much more simple than supporting plain HTML CRUD in the first place

There is a lot of material out there available on coding Rails in general, so I will let go here.

These first two hours are basically setting up all the routes, filling in the HTML and making things work in a very basic sense. For wigitize that meant:

• hooking up the URL input to a feed detector and aggregator (most of that code is from here)
• making sure that the aggregated data is stored in JSON format (In rails that means calling .to_json on any Object, easy as pie!)
• adding a Widget model that can actually hold the URL, detected feed URL and JSON data:

I spend the next 2 hours seriously code-monkeying on the feed detecting and parsing part of the system. I will soon open source it under the name feedeater.

Style that HTML! 1 hour

Styling, already? Yes, I think it’s good to style quite early in the process. For me, there is one single argument: Flow.

When you are making something work it is nice when things already look quite tangible and usable. When you apply styling in an early stage you can see direct usable results of the things you are building, increasing the psychological state of the Flow.

Ok, so I also kinda suck doing CSS, but I learned enough to turn an image into a web page. These are my bullet point lessons:

• Develop for Firefox first, using the tool Firebug. If you aren’t using Firebug and doing web development you’re either slow or an imbecile ;]
• Always put in some basic CSS, I got some here and made it into my own html.css which I can include as a base all the time.
• Padding and margin: These things are great and you need a lot of them. However, padding often gives you shit so try to choose margin over padding. Also, choose margin-bottom over margin-top since everything tends to float upwards (Thanks Simon)
• Current fashion: Get some nice fonts going on, I’m using Trebuchet MS a lot for Wigitize and I mix it in with plain Arial.
• Current fashion dictates that you use a less fierce black for your text, make it #111 or #222.
• For mozilla/safari I’m using -moz-border-radius/border-radius, will never work in IE6 – fuck them, being a plain user is fine, but you’re not getting any round corners! Besides, isn’t Microsoft planning a forced upgrade soon?
• You only need tables for tables. They will make your life a pain and you will not be cool if you use them for layout. Other than that it’s fine.
• Little side note about AJAX: don’t use AJAX for navigation EVER! There are strong SEO and usability arguments against it (learned that the hard way, like most things in my life).

Spin it Baby! 2 hours

OK, I must admit that I spend way WAY too much time on this! However, when you’re doing things with AJAX, you need to put a spinner somewhere to indicate loading.

This so called spinner is pretty kickass and based on a piece of Javascript I wrote earlier:

Yeah Baby! It’s all hypnotic and stuff.

Note: As someone pointed out in the comments, on ajaxload.info you can get a lot of spinner images, like the one I’m using here.

Designing and Coding the Footer, 2 hours

I really like footers and I think they are becoming more and more important. Nowadays footers are used as sitemaps and often they are contextualized as well. These are nice examples of footers: last.fm and snooth.com

For Wigitize the footer is rather small since it’s a small site:

Making the JSON Embeddable, 3 hours

<div class="feed_widget">
  <ul id="feed_widget_34"></ul>
  <script type="text/javascript" src="http://wigitize.com/javascripts/wigitize.js"></script>
  <script type="text/javascript" src="http://wigitize.com/feeds/34.json"></script>
</div>

This works in three steps:

• define a containing list (ul)
• include a JS library that has a special callback function, in our case wigitize_feed()
• include the JSON file that will call wigitize_feed with the appropriate data

Providing styles obviously complicates things a bit. When choosing a style, it will include a generic wigitize.css and assign a class to the containing div.

Near future improvements:

• Provide a ‘grab the grabber’ so that people can provide widgets of their own feeds (eg ReadWriteWeb providing a last-10 articles widget). This could have a lot of potential if provided in a simple feed-burner like button.
• Put in better default styles than the lame ass ones I have now.
• Option to include data (useful for photo feeds).
• Option to display other kinds of aggregated data, eg microformats.

Making things run in the Background, 3 hours

Running things in the background – dubbed backgrounding – is an important part of production ready web applications. It’s a relatively new concept, since web applications used to be less complex. Now however, we are marching towards the Semantic Web where web apps are expected to become intelligent (the Intelligent Agents are coming, just like in the Langoliers!). I think being able to make your applications autonomous now will already reap you benefits (to be continued as an article).

There are several ways to achieve backgrounding in Rails, but the far away most easy one is using BackgrounDRb. BackgrounDRb is a plugin for Rails that allows you to easily kick off background processes and schedule regular tasks. Good for our purposes: detecting and fetching feed data.

I don’t agree with all of Zed Shaw’s big rant about the Rails community being a ghetto, but I sure do agree that there are a lot of idiots out there that produce things that can screw over your production apps. BackgrounDRb has become one of these projects and I strongly recommend that you do not use the latest code. If you start comparing code and read the mailing list you will see that a new guy has taken over Ezra fine project and I suspect that he has lowered the project’s level to pre-stable. I don’t hear any signals from the community and that worries me. Either I’m seeing ghosts or people are blindly accepting anything that’s marked as stable. In any case, I’m using Ezra’s version which works fine.

Finishing up the API, 4 hours

The Wigitize API for now is quite quick and dirty. There are two simple ways of using the API and examples are provided. In this area, a lot of improvements will be made down the line since it’s a key point in making any future freemium revenue.

Domain and Domain Email, 30 minutes

When I buy a domain I always buy a DNS managing package with it as well. This means that I can login somewhere and setup subdomains and set mail records. So the total price of Wigitize.com was 20$ per year.

Providing info@domain.com email is easy, just get yourself a free Gmail for organizations account. In your Google Apps domain manager you can simply add your domain and in your DNS tool you set the MX record to ASPMX.L.GOOGLE.COM. Now you can use gmail and IMAP to read mails sent to your domain.

Setting up a Production Server, 1 hour

I was really eager to put this project on Amazon’s Elastic Computer Cloud until I calculated my monthly costs. Running costs alone for simple projects like these will cost you 60$ a month. Still I think it’s worthy to look into this once you scale beyond simple project.

After digging around I figured out that slicehost.com would be a good cheap second. For 20$ a month I have a 256 memory slice with 100GB in data transfer – awesome.

Setting up your slice takes 5 minutes with a credit card. This slice is essentially a virtual machine with an IP address, completely yours. And the best thing: you get an awesome web console to control everything! Adding a new machine is a no brainer.

Now, I’ve used linux/unix for a long time as a working station. Eventually I got lazy and switched to my current MacBook. Fortunately, you can be lazy for the systems side of things too. All thanks to a lovely tool called Deprec.

Deprec allows you to install the complete Rails stack with a small set of commands. Shortly thereafter you can deploy your application to your production server by typing cap deploy_with_migrations. Please note that for Deprec you need to install Ubuntu Linux on your machine which you can do as follows:

Deprec installs the Rails stack as: Apache, Mongrel cluster (default of 2 instances) and MySQL. As I’ve written earlier, NginX is a nice nano-alternative for Apache. I would like to see that in my Rails stack someday, but I’m not going to worry about that now. Clock is ticking!

Little Pimps and Tweaks, 3 hours

I think it’s good to prepare a little bit for the storm (and I felt like doing something else for a bit), so I’ve created a nice maintenance message for in case there are system/scaling problems. In here I think it’s important to give people an extra reminder to bookmark and come back.

Which brings us to another great service, addthis.com. AddThis provides you with a button that makes it easy to bookmark on multiple social bookmarking services.

Another one of those little tweaks was proper error checking and displaying it to the user. I am using a pink error message to make it look more friendly (maybe I should go even further and make it yellow or something):

Statistics and Search Engine Optimization, 1 hour

Statistics is another 10 minute no-brainer by using Google Analytics. However I’m on the lookout to find something more real-time like Mint (but than free). Any suggestions?

Note: I just added getclicky.com to get more realtime stats than GA

I found a blog which is solely about Rails and SEO which I thought was very promising but in fact doesn’t have much content. I do found something on how to provide different meta tags in rails which I applied right away.

After looking around a bit I also saw some discussion about whether to use www. or not. The way to roll with this is: you permanently redirect your www. domain to say http://wigitize.com. It makes sense, www is old and architecture centric, http://wigitize.com is less typing and pretty.

On that note, make sure that you always write pretty URL’s by practice. This means thinking them through extra carefully, because changing URL’s is painful after going live.

Wigitize.com was also chosen with SEO in mind. The word is a mis-spelling of the term widgetize and yields 105 Google results (at this moment). Additionally, it is a verb, which lubricates the prettiness of the URLs :]

• Making a list of ‘last wigitized sites’ and ‘most wigitized sites’, those pages can be accessed by the search spiders and thus associating external content with Wigitize.com. I think this might improve search rankings.
• Providing a sitemap.xml, would that help?
• Focus on the viral aspects of these widgets. For example a FeedB urner style button for on popular blogs.

Let’s throw it out there!

While writing this article – which took quite some time. Wigitize.com is already running and doing it’s job. However, I’m sure there are still some kinks to work out which I will do over the past coming days (eg IE6 support, SEO/viral tasks).

Also, I didn’t discuss anything about an important aspect: How to make revenue with Wigitize.com? I’m not sure yet, but since I’m solving a problem for myself, I’m sure others out there had it. Besides, the costs are extremely low at this point so I will worry about monetizing later. Although I would like to hear YOUR thoughts about it!

I realize that this is a geeky project and I must say it’s quite different than the web apps I normally work on. It was fun for me however to write down my thought process, especially on the non-tech parts of building which I find increasingly interesting.

I think it's getting more and more easy to start up your own knowledge-driven venture. For a couple of years now, I have been managing my own server (together with my friend). Managing your own server is a pain, but a must if you are a tech entrepreneur.

Luckily, lately there have been services popping up that alleviate a lot of IT-pains. This week, I have been looking for services that allow me to externalize the hosting of my Email service. A few services like SproutIt's Mailroom came close, but didn't quite cut it.

Then, I found out, that I can actually attach my domain names using Gmail! Also, I can use IMAP, POP etc. I now have most of my domain names attached to it, and I pay JACK SHIT!

All thanks to another great article from ReadWriteWeb.

For me, an important part of enjoying these holidays is having a nice drink. And what better than to go drinking 2.0!

I'm trying to input the wines I'm drinking into snooth.com, a wine recommendation website. Based on how I rank certain wines, it will recommend me others.

Also, if you're new to wine (like me), checkout this guy:

360+ web episodes about drinking wine, their slogan: 'changing the wine world'. Very interesting indeed!

Twitter is great, and this small 5 million $ company is growing like crazy. The core of their service is inherently simple: blogging/chatting with no longer than 150 characters. They opened up all of their API’s and Twitter is now flurishing with activity. Hell, we even integrated it in our language learning platform: http://iknow.co.jp/ (more to come) It’s only the first step however, these services could open up even more!

iKnow is a service that specializes in online learning and therefore the SNS part of the site is nevertheless important but still secondary. Right now in our service iKnow you can upload a picture together with your journal entry. It would be nice if we could provide more picture uploading and managing functionality, but we don’t want to build that. Flickr specializes in these things and would make a nice addition. Unfortunately, a service like Flickr doesn’t provide a real transparent API yet – people still need to register for a Flickr account.

Also, it would be nice if could provide status updates for all people on the website using Twitter – but people still need to go through the registration procedure at Twitter.com to be able to use it. I think the next success in web integration lies in opening up your API’s to an extend where it is completely transparent to use them. You don’t need to worry about registering at the third party service.

But what about the revenue? If people don’t come to your website anymore, you cannot get any advertising money! That maybe so since most sites still rely on people being exposed to banners/adwords on their website. These things will change however:

• there will be in-content advertising, an example is the already emerging in-video ads
• the freemium model will also be applied to API’s. A free API is for personal use, a premium API is for integration use in other web services. This premium API will not require you to let people pre-register at the very least.

The big picture really makes sense: online services will be more specialized. Right now you could imagine that photo management is done by something like flickr/picasaweb, status updates by twitter, music integration by last.fm etc. But things could fragment even more when there is an open integration market: facebook-style wall service, embeddable message/mail service, tagging service, rating service or image cropping service. An example of the latter is PicNik a service recently integrated into Flickr.

sidethought: will this kill branding?

Music - the symphonic phenomenon that binds all us Humans and maybe even separates us from the lower lifeforms - is something I've been forgetting over the past two years. The last two years I've been very focussed on my career and I haven't spend much time downloading or listening to music. Since my rock teenager days my passion for music has been decreasing. The only moments of music for me was around the time I purchased an iPod or went to [the Royal Dutch Concert Building].

But now, all has changed.

I found a way to be lazy, not having to purchase or download any songs. This new way, this revolution, is called Last.fm. Some of my friends already had an account for a long time. Typically the real audiophiles were the early users of this system. Ok so what's that last.fm and why is it so great?

Last.fm User Experience

Last.fm is a little web startup from London. A few months ago they probably had a double dozen of employees (now 40). Last.fm is an online audio player that gives you these experiences:

• generate your own 'radio' by entering artists/categories
• unlike normal radio you can skip tracks, love tracks or ban tracks
• while playing, real-time user generated information is displayed in a wiki fashion
• the company's obsession with statistics allows you to see your musical compatibility with other users
• when listening to iTunes, last.fm will try to spy on what you listen to mostly
• The Long Tail, one of last.fm's goals is making all the music in the world available

Basically last.fm and the whole world knows what kind of music you like. For most people this is not a problem, since they really really like showing off what music they like (especially teenagers).

The effects of the Long Tail allows you to explore niches you didn't think existed:

Look I'm the top listener of that dude from Okinawa!

Last.fm Freemium

Last.fm get's the essence of music, they just 'get it'. To quote the first sentence from this research paper on 'Why do Humans Value Music?':

Whenever and wherever humans have existed music has existed also. Since music occurs only when people choose to create and share it, and since they always have done so and no doubt always will, music clearly must have important value for people.

You hear that? Sharing the music!

Like Google, last.fm connects supply with demand. Thanks to their thorough statistical systems, they really know what people like. Last.fm has a Freemium business model that allows you to upgrade your account for 30$/year to get these benefits:

• listening to the tracks you specified with the 'love' button
• listening to the tracks of your musical compatible friends
• listen to your own composed playlists
• social status (like Flickr-pro)

For now, last.fm doesn't offer real 'select and play'. All tracks played are randomly contained in a radio stream of at least 10 tracks. They do have a little link that allows you to 'buy the cd' (linking you to amazon.com). I assume that in the future we can expected a button that says 'buy and put in my songs'. Of course they depend on the slowly evolving record industry for that.

Maybe these advertisements on their website will speed things up:

Yesterday Last.fm was sold to US' CBS for 280 mln $

The project is well underway to developing a nice product. Also, as you might see in the below picture, it's kind of fun to use!

Click here to read more and vote for the name

This article is located here, at my joint entrepreneurial blog.

During my scarce free time in the weekends and nights, I will work on some of my web 2.0 projects. A good friend of mine, Aram Versteegen, who’s doing the same thing will join me in blogging the hardships of starting cool web projects.

digigen.nl will be a full-disclosure of our idea’s, technologies and experiences.

Newest articles:

Digigen, Short History and Naked Future

Web Video Conference for the Masses

I ve got a double coupon for that

Digigen blog goes live!

This article is better explained on my new joined entrepreneur blog: digigen.nl

Ok I think I will blog my idea’s more openly. Being afraid that someone might steal it is just stupid, for several reasons. As Clive Thompson correctly points out in the latest Wired businesses are starting to realize the importance of getting naked. Sharing the secrets, mistakes and hardships. Also in the great manifest Getting Real there is this simple equation:

idea can be:

• -10 really dumb
• 1 stupid
• 5 ok
• 10 smart
• 20 excellent

execution can be:

• 1 almost nothing
• 10 bad
• 1000 ok
• 10000 good
• 10000000 excellent

If you multiply these two variables, you will get the amount of cash you earn (in euro’s I recommend).

Confabio.com

This idea is not a 20, but it could be at least a 5. It happened while I was doing two things:

• communicating a lot overseas, for work and private matters
• playing around with my hot sexy black macbook (named burakubuuku) and its cool built-in webcam
• playing around with Adobe flex

This let me to make a little website that does the following:

• Show your head on the screen
• Recording your nasty head and broadcasting it to a Red5 server
• Showing your head again by streaming that same broadcast

Confabio will be as simple as possible. First it was intended to be an arty farty project, but I think it can actually be useful. When a person goes to confabio.com he will see his own video. When another internet user opens confabio.com a second video-stream will be displayed, a third, fourth and so on. Of course, the more people join, the smaller the screens become.

I haven’t really looked at so-called competitors yet and I don’t intend to. The only features I’m willing to add is:

• displaying your location discretely under your video-stream
• having tags/channels/rooms: confabio.com/dominiek_room, confabio.com/cooking

Techniques

Mochiron I will use RubyOnRails, but most of the stuff I will need to use flash for.

• RubyOnRails will facilitate metadata to swf: geo-location information, IP
• RubyOnRails will use the weborb plugin to facilitate serverside data facilities for the swf app
• Adobe Flex will be used to compile .swf files (using .as, .mxml and .css files)
• Red5 is a Java server that facilitates video play/publish streams (Note: I really hate this piece of software because it solves a solution and not a problem, checkout these 5 pages hello world)
• Adobe Flash Player > 8 is needed in order to access microphone and webcam

Why will Confabio.com work?

I don’t know and I shouldn’t care. Primary focus should be: getting this working and actually start using it. The factors that could make the execution a success:

• Simplicity, this is relatively not-complex as long as we don’t add too much features
• the need to use it, I really need a tool like this so I will start using my own product

Why will Confabio.com not work?

There is no business-model yet, and I think a nice extra in this project will be a bandwidth monitor to project expenses. Also, we need a good design and perhaps a better name.

note: I wrote this article in January and I just found it hidden deep down my backpack

I think that one of the mayor success of the new Web is thanks to open communication, open standards and open information. Opening up information has given rise to the Community websites. The meaning of Community is ‘People that are grouped together and share’. Opening up information and communication is the lubricant of this sharing process. But I think there is more to this then just social websites.

About two years ago I started reading a self-help book called ‘The Seven Habits of Highly Effective People’. This book contains some nice principles/habits of achieving success in life. Success, in this case, means both material, mental and spiritual success. One of the core principles of the book is achieving ‘Interdependence’. In order to become interdependent, one must first become independent (rise from co-dependency). Only after becoming independent we can start reaping the benefits of synergy: The whole will become more then the sum of it’s parts.

Websites are starting to offer information loosely, for example trough RSS, Webservices and other data offerings. They not only offer data, but they also create new data from multiple offerings. I think this loosely coupled way of exchanging information can be seen as the first steps of the internet becoming interdependent. The early web was static and the only form of interdependency was exchanging links (which nonetheless made it a tremendous success!).

I think it’s time we start examining the impacts of interdependency on humans and their systems. We can then use this to anticipate the Web’s needs and innovate accordingly.

Before Oreilly dropped the term ‘Web 2.0’ there was no Web 1.0. At this point, people are discussing the definition of Web 3.0 – while we don’t even have a definition of Web 2.0.

In his article on KurzweilAI, Nova Spivack outlines the characteristics of all three versions. All of them are valid observations I think. However, Web3.0 observations like mobile internet access, broadband adoption, SaaS, open communications etc. can also be done in the Web2.0 and Web1.0 eras.

I don’t think we’re entering the ‘third generation’ because there can be no clear definition of ‘the new version’. I think the term Web2.0 was just a joke (and intended so by Oreilly), funny, ha ha, over. Let’s just drop the dot and the whole versioning system altogether. The evolution of the Web is real however, and it’s dead serious.

The post 2006 Web – I like to call it ‘the Webemoth’ – is growing and shaping itself in numerous trends. In the Herald Tribune, John Markoff observes the following ‘next steps’:

• natural language understanding
• machine learning
• the semantic web
• data mining

I think these trends – when added to our current trends – will indeed move us towards a better web.

Natural Language Parsing is something which is popular among some of the latest ‘top secret’ start-ups like powerset.com. I also think it’s something very interesting and I bet Google has been working on it for quite some time. NLP is already being done in high-end systems like NSA’s echelon to keep track of our emails here in Europe. Just like powerful encryption, it’s the next thing to come in hands of ordinary cattle like us.

Machine Learning is something being done all over the world – like for example at your credit card company – and will advance the web enormously. Even though, some of the top ranking websites are already harvesting the power of pattern recognition for music and book recommendations (pandora.com, amazon.com). In my country there are a lot of artificial intelligence studies available and I know a lot of people attending them.

The Semantic Web is something we are missing now. Many websites are already making advances to this semantic web below the radar. Every day there are numerous new so called mash-ups that stitch together data from various sources. In order to do something useful with this data they need to know the semantics. Sites providing API interfaces like Wikipedia and Amazon.com offer only limited semantics. Sometimes they are missing and then developers switch to data mining techniques like scraping (parsing the user interfaces to ‘pick out’ the data).

I must say I’m a bit skeptical about the move towards the Semantic Web in the short run. Semantic – semantikos, giving signs – refers to the meaning of information. Current .com behemoths like youtube.com profit from the user contributed information. More specifically, they profit from displaying this information but not from the information itself (at least, not yet). Sometimes I watch some TV at youtube like sites and I guess I’m a very bad customer: I block the advertisements (so I guess they should move the adds to the stream itself). This is my skepticism:

• Are there business models that can profit from just providing semantics?
• If there are, will they affect the semantics/content itself?