This is a free-form brain dump of some thoughts I have after reading Wisdom of Crowds and having been an active contributor to Cambrian House for about a month now. I just now caught up on most of the Primordial Soup podcast episodes (which is excellent btw) and reviewed the current top ten ideas in Idea Warz on the CH site. The idea of CH itself is very similar to what we were after with Grid7 Labs and having first-hand experience trying to do what they’re doing, I feel I have qualifications and context to make these assertions.

Wisdom of Crowds

First off, if you haven’t read James Suroweicki’s book yet, you should. It’s an extremely interesting proposition: the idea that you can collect many independent judgements from a group of diverse members, tally it all up and get a highly-accurate prediction of what the truth really is. What’s neat about WOC is that it can apply to anything- from jelly bean counting to sports games to complex business decisions. We had proposed the development of an app that would enable corporations to conduct decision markets internally and with their customers as a great candidate for a Grid7 Labs project. Inkling Markets beat us to the punch though and it seems they’ve done a partnership with Cambrian House to handle the decision markets under the hood for CH that are used in determining which ideas end up getting built. This all makes perfect sense given the similarity of the Y Combinator concept to CH and knowing Paul Graham’s affinity for incubating early-stage tech businesses. We wish both companies the best in this effort as the developer co-op concept is something we tried ourselves with difficulty.

Getting back to WOC though- the example used at the beginning of the book is a carnival environment where a group of 500 or so attendees is asked to submit guesses as to the weight of an ox. The mean average of all guesses ends up being 1lb. off from the true weight of the ox supporting the theory that many independent judgements from diverse people yield a very accurate prediction. James goes on to cite numerous other examples of situations where this phenomenon has been tested and proven. For instance, the disappearance of a navy sub where the potential resting location of the sub could have been anywhere in a thousand-mile radius in the Atlantic. They conduct a focus group with a bunch of different navy people and submarine-related people in various roles and have them posit guesses as to where the sub went down. The aggregation of these guesses ends up being within a hundred yards of the actual location of the sub yet again supporting the premise of WOC. There are a ton of other examples in the book but here’s the core principle:

In order for the WOC phenomenon to work, there must be four pre-conditions fulfilled for the environment in which the guessing occurs:

1. Independence – that the judgements be made in a way that doesn’t influence each other
2. Diversity – that the participants making the judgements be composed of the most diverse possible group
3. Decentralization – that the participants be able to draw upon knowledge that is obtained locally and not funneled through a central source
4. Aggregation – that there be a mechanism to capture and gauge all the judgements made

The Flaws with CH

Herein lies the problem with the existing implementation of Cambrian House:

The current system breaks three of the four necessary pre-conditions.

Let me preface the following by saying, I love CH. I love the spirit of this concept of crowdsourced software (deep down i’m really a modern-day hippy- i’ve had a peace sign around my neck for the past 15yrs). And I really like the execution so far on how they’ve told the story and managed to ignite the community behind their drive. The whole Viking / Napoleon Dynamite theme is genius and keeps things fun and engaging when they could easily drift towards sterility and blahness. I want to see them succeed because this is a difficult problem they’re tackling and we struggled with making it work ourselves in the Grid7 Labs cooperative. So this is not a slam at all but rather a collection of constructive suggestions that will hopefully make their system tighter.

Problem #1: The current Idea Warz implementation allows the rater to see the score once he/she has rated an idea. Bad! There needs to be a “quiet period” where new ideas are rated without divulging their status. It’s too easy to give a thumbs up/down and then immediately change your vote once you see the score. This breaks the first pre-condition of independence because it allows the hivemind / groupthink / herd mentality to skew the results.

Solution: They need to conceal the rating and maybe even the comments for at least the first week that an idea is submitted.

Problem #2: The current participants are homogeneous and consist of the slashdot / digg crowd. It’s not surprising that the early adopters of this thing are highly-technical people. But in order for it to truly work it needs to attract a more diverse group that bring other skills to the fray. I’ve submitted eleven ideas so far, all of which came from the Grid7 Labs project and were the distillation of many hours of brainstorming and working through business models and technical ideas within the Labs team. In the first three days of submitting these ideas, eight of them kited up to the top-twenty out of a total of 2,500 ideas. I was excited about that but then a strange thing started to occur- all eleven ideas were getting great comments but their ratings took a dump. They are all now hovering around the -10 rating (+ scores are better). This is a bit disturbing and after reading the top five ideas just now, here is my hypothesis of what’s occurring here:

In order for an idea to gain traction in CH right now, the pitch needs to be tailored to a slashdotty audience.

Anyone who has made a successful pitch will tell you that the content of the pitch itself is not nearly as important as the level of connection you establish with the pitchee. We learned this through an experiment at our second FastTrac class when we went around the room and had to recite the elevator pitch of another entrepreneur from our class. It was a powerful lesson that your pitch is only as effective as the impression it makes on the listener. The contrapositive to this negative example was the experience that happened only a few days after we launched the JumpBox entity in which we were contacted by someone who had caught wind of what we were building through a friend of a friend and was interested enough to track us down and ask us when he could purchase it (and all this only one week after deciding that we were going to build it).

I imagine the typical CH participant right now as someone who sleeps with his gameboy controller and checks Digg twelve times every hour for the latest AJAX tutorial or YouTube video. There are a ton of intelligent, insightful people in there but unfortunately the masses right now seem to be composed of mostly the slashdot crowd and are basing judgements on ideas solely on the technical “wow factor” rather than the business-viability factor. Having your ideas (essentially your babies) torn apart by this crowd feels a bit like I imagine it would feel to come home and find your pet slaughtered by a roving pack of jackals that then wanders to the next neighborhood in search of fresh meat.

Solution: CH needs to spend some of that VC money to promote this concept amongst the business schools and MBA programs. It’s neat that their community is a global one (good news on the geographic and ethnic diversity axes) the industry axis is hosed and way too skewed towards tech nerds. They need more people thinking about and contributing to the business model aspects and worrying less about the technology. This will undoubtedly be a concern of theirs and surely be addressed some time soon. It would be fairly trivial to run a facebook ad campaign and reach some of the talented students of the various MBA programs to get this diversity in motion sooner rather than later.

Problem #3: And I don’t know how you solve this one but the nature of participants’ interaction with the system is such that it will always be filtered through the CH lens. It can’t be any other way or else the idea becomes loose in the wild and it’s impossible to apply useful metrics to the feedback that gets generated. It’s conceivable that they could use the structured blogging plugins and something like MC Ping to create a specific microcontent definition for a CH idea and then allow people to post them via their own blogs and elicit feedback there and aggregate it on CH via the MCPing notification service, but it seems like they have bigger fish to fry right now with the need for getting projects in the queueu and people contributing to those. They do need to stress the business model aspects of the ideas more. It was suggested in the podcast that they create a separate field for “how this idea makes money” in the idea submission process – this could also be incorporated in the MCD.

Solution: There is no silver bullet for this one unfortunately. They could do the above suggestion with the SB plugin but at the cost of losing the reins on the data and not being able to have the tight metrics on how the ideas are being interacted with. My recommendation on this is to run with the current setup but stay cognizant of the fact that the CH interface itself (as unbiased as it may be) is still a lens that adds distortion to the true picture. They need to try and minimize this distortion as much as possible at all times.

What I like about their current setup

For the handful of things that go wrong and that I’m choosing to pick on, I know from experience that this is difficult stuff and that there is a million other things that are going right under the hood in order for them to establish the momentum they’ve built so far. So once again, I applaud CH for telling the story well and creating a surprisingly-good mechanism for aggregating the feedback and using it to determine the winning project ideas.

They’re on the ball with the various collaboration mechanisms in place- Norseforge appears to be built with the same GForge technology that is used to create SourceForge. We explored using this ourselves for G7 but ended up going with a simple Trac/SVN setup using a private Google Groups and traditional IM for real-time exchange. They have an IRC channel setup which is also something we toyed with but never ended up doing. We’re now planning to run an internal Jabber server rather than a private IRC channel but both achieve the same function of having chat that accrues in a central archive. The style of their marketing, blog, podcast, public site and backoffice is pure genius: it’s just the right tone- a ongoing fun yet legit cooperative game showing consistent progress each week and celebrating the small wins. They use fresh and concise writing style and seem to value the same transparency that we did in how things are conducted- they have to in order for it to succeed. They’re setting up an independent “tribal counsel” composed of both members and CH people to solve the inevitable IP disputes that will arise once a project hits and becomes valuable given that it emerged from a stew of ideas submitted and touched by many people along the way. My overall impression is that they’re hearts are in the right place and they’re using agile processes to make things “light yet sufficient” and roll with the punches as they come.

Anyways, this post ended up being longer than I had hoped so I’ll discuss the Inkling Markets stuff at another time. The bottomline here: CH has a good thing going. I have a bunch more ideas to submit and I plan to apply what I’ve learned in the next round of submissions and tailor the pitch more to the slashdot crowd. They have interesting and challenging problems to solve but so far they’re doing the right things, they have money and there is no fundamental, show-stopper flaw that I can see in their model. They will undoubtedly meet the same challenges we encoutered in relying upon labor that is not on the hook to actually deliver but only motivated by promise. My involvement in CH is hobbyist status at this point given the fact that JumpBox dominates 110% my schedule right now but I could see after the JumpBox liquidity event becoming more involved in CH. It’s even possible that the Grid7 Labs team itself becomes dedicated to executing CH projects as a long-term play for recurring revenue and recruiting talent. It’s very consistent with the goals of Grid7 and would essentially mean that our people get to contribute without actually having to host the infrastructure ourselves. Apparently they’re announcing a new Idea Warz voting system this week. I would love to see them work with Luke to incorporate some of the theory from Innovation Games into the next iteration of Idea Warz. At any rate, it will be interesting to see how the CH system evolves to address some of its current flaws.