Self-organization

2009-12-03 00:27

The death of a star

It’s not often you get to experience a star dying. Up until yesterday, I was unaware that I had accidentally collected evidence of this happening. How and what evidence you might ask. Well, this was how it happened.

Primo November I did a minor study where I wanted to correlate the structural elements of web sites with the actual content it provided. I knew this would definitely be possible on the broader level, but I was unsure to what degree this differentiation was possible and what the similarities would look like.

Therefore I began to partition a lot of different web sites into categories. Examples of such categories are portals, blogs, and media. I also choose specialized search engines. Further to this, I knew beforehand that there existed a type of search engine that specialized in finding one thing, and one thing only, namely the infamous torrent files. Torrent files are small files that contain information pointing to the whereabouts of other files. In other words, a torrent file is quite simliar to a telephone book that contains information about the whereabouts of people and businesses. In turn, these search engines knows the whereabouts of lots and lots of phone books, each having information about the whereabouts of a certain number of people and businesses.

To put it simple, choosing this specialization would provide me with a near perfect setup for a comparative analysis.

The dying star I am referring to is the web site Mininova (link). Mininova, whos name literally means little star, is one of these highly specialized search engines I mentioned above. Some weeks prior to November 26th it showed evidence of being a highly used and visited web site. Amongst other things, it showed the top ten most popular torrent searches for each major category. Example of such categories are movies, music, games, and other.

This is the structure of the web site primo November (see picture below). Notice the big flower-like structures emanating outwards from the center. These are the categories I mentioned earlier and testify to its heavy usage.

Figure 1: Structural evidence of a widely used and visited torrent search engine.

So what happened November 26th? Well, according to Mininova, they decided to change their business model on that specific date. And they did.

Actually, it began August 2008 when the Dutch court of Utrecht ruled that Mininova’s business model was illegal according to Dutch law. The changing business model was therefore a response to this court decision and its purpose is to comply with Dutch law while hopefully retaining a revivable business opportunity.

This is what happened to Mininova’s web site within 24 hours after launching their new business model (see picture below). Pretty remarkable isn’t it?

Figure 2: A drastically deteriorated structure that bear witness to a failing business model. (zoomed in)

The structure is heavily deteriorated because the consumers have disappeared to other sustainable search engines. Even Google’s hit counter show clear evidence that Mininova is becoming a virtual black hole. Its Swedish counterpart, The Pirate Bay, who might also face changes to its business model in the near future, is doing more than twice as good. The real winner is surely the other competing search engines, mostly specialized but also generalized, and of course the plaintiff’s new and unique legal position within the Dutch free market.

Ending on this business opportunity note, I strongly recommend that all other business areas take a very hard look at how they can achieve similar unique legal positions before they are overrun by the plaintiff’s unique area of expertise.

Notes

If you're unsure of what the pictures above show, please read my previous blog "Visualizing the web", where I explain the details. (link)

Postad av Martin Kaarup

Kommentarer (0)   Kategorier:  World Wide Web    Self-organization

2009-10-16 09:14

My scale-free contacts

One drawback of returning from Avega’s 2009 conference in Karpathos, Greece, is having a head full of ideas and recharged batteries. It manifested itself a couple of days ago. Around bed time, I felt a sudden urge to map all my contacts, and their connections as well, on the professional network site LinkedIn. Supposedly, I should be able to see the scale-free invariance property of a social network.

That night I went to bed at around 4 a.m. and I was still miles from completing my data gathering. I didn’t finish the next night either – or the next. The fourth night it dawned on me. The very raison d’être for my investigation, the scale-free invariance, was the problem.

Since LinkedIn is a typical social network site, it follows the power law function which, simply put, states that the vast majority of people would have few contacts, but also that there exist a small handful of people that would have lots and lots of contacts.

And so, on the fifth evening, I haven’t finished gathering the data yet. I expect to be done in a week or so, and then I can finally begin automate a solution to connect the dots, so to speak. On the positive side, I have finished connecting my 1st degree contacts, and if they are ordered by number of connections, it’s possible to observe the scale-free invariance pattern – albeit with some minor variance, mainly because the sample set is small. It’s truly remarkable.

The center of the universe is… me

References

Postad av Martin Kaarup

Kommentarer (3)   Kategorier:  Graph Theory    Scale-free Networks    Self-organization

2009-09-07 00:55

I like ants

Since this is my first blog post I thought it should be memorable and so I will share a secret passion of mine.

I like ants, I really do. Compared to Avega I might even like ants a little bit more. Compared to my wife, well, she’ll always win big, but only because I know my priorities in life – not because of the ants.

Fastest path

Even though I’ve first read about ants five, ten, fifteen years ago, I still enthusiastically teach others about the beautiful fastest path algorithm that ants use to optimize their harvest – and sometimes I also do this against people's will (like when they get stuck with me in front of a white board). I think it’s the emergent behavior that arises from doing simple things tens of thousand times or more that still amazes me.

And so, true to my heart, I will tell you about this beautiful simple algorithm; suppose you have to travel back and forth from A to B, then when you come to a cross road, just be slightly more likely to pick the road that is traveled by most of the people that came before you. That’s it – that’s the essence of the algorithm.

Well, I’ve hidden some minor details, such as how to find out about the people before you. In my human analogy above, we can just add to the requirements that everyone wears the same perfume, which means you can use your nose to decide where the strongest odor is coming from and then go that way – much like how discotheques or school yards function (which in turn could explain why children intuitively understand the phenomena and drunk adults don’t).

Anyway – given enough people, the crowd will eventually diverge onto the fastest path (because people that choose correct will travel the path much faster and in turn leave more odors for the next person to make a slightly easier decision and so on…). In other words, it’s basically a positive feed-back mechanism (and if you're control theory inclined, you can extrapolate its benefits and liabilities).

If you aren’t convinced I’ll bet you the next drink at the discotheque that if you were to spray a woman’s perfume onto one side of the entrance and observed the men’s choice entering (deciding whether to go right or left), you’ll find that more men would autonomously choose to follow the perfume – and vice versa by the way; it’s not the case that men are the only lemmings in this nocturnal game. (And if you’re a child reading this, don’t expect me to let you into any discotheque before you turn 30 or so. And since you’re reading about ants, you’re obviously clever already, so stay home, read some more, get rich, and buy your own discotheque!).

In any case, my only regret is that I haven’t found any justified reasons for using the fastest path algorithm during my programming career and therefore should probably have forgotten about it years ago, but apparently it’s a permanent resident in my long term memory.

Sorting

Well, I wasn’t quite honest. There are several reasons why I don’t forget about ants, namely because they do a whole lot more than just finding the fastest way home from a discotheque. For instance, they can also pile stuff of equal sized objects, which is what they do when moving eggs of different maturity around the nest (and perhaps also when they do midden work outside, I’m not sure…)

I won’t bother you with the details surrounding this algorithm, but only say that it involves moving smaller piles of eggs to the approximate vicinity of nearby equal sized but bigger piles of eggs. The emergent behavior of being sorted arises from doing the task iteratively, starting with a pile of one egg, then a pile of two eggs, then three eggs and so on (much like divide and conquer algorithms such as Quick sort).

In computer science such algorithmic qualities (i.e. being distributed, iterative, map-reducible, and easy to implement) is highly sought after today – for instance, when sorting huge amount of data in a cloud-based mesh of computers (and each computer having multiple cores, just to add some locality to the example).

Recruitment

As I mentioned, there are several reasons why I don’t forget ants and here comes another one; seemingly, a clever recruitment scheme is partly the reason that ants are grand masters at exploiting new unforeseen business opportunities without managerial control. Such agile recruitment techniques should rightly be a study of its own right in every business school in Scandinavia (An MBA in Ants so to speak).

For instance, several R&D intensive trans-Atlantic companies have adopted ant-based recruiting mechanisms to harvest the ideas that flow around the corridors and hallways of their domiciles. These ideas are often cross-departmental and therefore almost always span several communities of practice. Such extreme plasticity surely means that ideas thrown in the air have a greater tendency not to be caught by anyone. And lost ideas might be lost revenue – a quality that investors dislike.

Since the mechanism involves recruitment (which is needed to evolve the sustainable ideas from infancy and into a business case), it requires some notion of agile workforce deployment and might at first scare of smaller companies from adopting such a mechanism. This is a fallacy of thought, in that the natural selection quality of the mechanism also takes care of size differences, release cycles, etc. (much the same way that smaller ant-hills can harvest smaller resource, while big ant-hills can harvest bigger resource – both using the same recruitment scheme).

Resource allocation

Recently, I’ve just finished a book by the biologist Deborah Gordon in which she presents her collected research of the red harvester ant in Arizona, USA. She argues, among other thing, about how ant hills resource allocates their workforce in a distributed manner. Apparently, she has found a single algorithm for all sizes of ant-hills (one size fits all) that can explain how ants allocate their workforce – again without managerial control. One of the algorithm’s most interesting attribute is its use of temporal effects.

Admittedly, I haven’t digested it all yet and can’t really say anything else meaningful about it, other than just point out, that such algorithm would prove interestingly for any industry that are resource planning or logistics intensive (like the shipping industry).

Besides delving into the resource allocation algorithm and hopefully be able to comprehend it in greater depths, I’m already looking forward to next marvelous reason why I like ants.

Footnotes

Firstly, I purposefully used the term fastest path algorithm instead of shortest path algorithm, because while it’s often the case that the shortest path is also the fastest, it’s not necessarily always the case.

Secondly, I don’t know where the pictures originated from, so if they infringe on anybody’s claim of ownership, please let me know and I’ll remove them right away.

Thirdly, my oldest child doens't like ants very much. This summer, at Trollhättan in Sweden, she ignorantly climbed the biggest ant hill she could find and proclaimed to be it's queen. Being pissed on by a million angry ants, literally, cured her dreams of feudal overlordship.

References (in no particular order)

• Deborah M. Gordon, Ants at Work: How An Insect Society Is Organized, W. W. Norton & Company Inc., October 2000.
• Deborah Gordon, Deborah Gordon digs ants, TED Talks 2003, http://www.ted.com/talks/deborah_gordon_digs_ants.html.
• Eric Bonabeau & Christopher Meyer, Swarm Intelligence: A whole New Way to Think About Business, Harvard Business Review, May 2001.
• Eric Bonabeau, Marco Dorigo & Guy Theraulaz, Swarm Intelligence: From Natural to Artificial Systems (Santa Fe Institute Studies on the Sciences of Complexity), OUP USA, October 21 th October 1999.
• Marco Dorigo, Ant Colony Optimization, MIT Press, July 6th 2004.
• David Gordon, Collective Intelligence in Social Insects, http://ai-depot.com/Essay/SocialInsects.html.
• Susan Leigh Star & Geoffrey C. Bowker, Sorting Things Out - Classifications and its consequences, MIT Press, 1999.
• Susan Leigh Star, Got Infrastructure? How Standards, Categories and Other Aspects of Infrastructure Influence Communication, University of California, April 2002.
• Kurt Mehlhorn & Peter Sanders, Algorithms and Data Structures: The Basic Toolbox, Springer-verlag, August 6th 2008.

Postad av Martin Kaarup

Kommentarer (0)   Kategorier:  Inspired by Nature    Complex Systems Theory    Self-organization