Karim Lakhani of Harvard Business School (and a Berkman associate, and a member of the Harvard Institute for Quantititative Social Science) is giving a talk called “How disclosure policies impact search in open innovation, atopic he has researched with Kevin Boudreau of the London Business School.
NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.
Karim has been thinking about how crowds can contribute to innovation for 17 years, since he was at GE. There are two ways this happens:
1. Competitions and contests at which lots of people work on the same problem. Karim has asked who wins and why, motives, how they behave, etc.
2. Communities/Collaboration. E.g., open source software. Here the questions are: Motives? Costs and benefits? Self-selection and joining scripts? Partner selection?
More fundamentally, he wants to know why both of these approaches work so well.
He works with NASA, using topcoder.com: 600K users world wide [pdf]. He also works with Harvard Medical School [more] to see how collaboration works there where (as with Open Source) people choose their collaborators rather than having them chosen top-down.
Karim shows a video about a contest to solve an issue with the International Space Station, having to do with the bending of bars (longerons) in the solar collectors when they are in the shadows. NASA wanted a sophisticated algorithm. (See www.topcoder.com/iss) . It was a two week contest, $30K price. Two thousand signed up for it; 459 submitted solutions. The winners came from around the globe. Many of the solutions replicated or slightly exceeded what NASA had developed with its contractors, but this was done in just two weeks simply for the price of the contest prize.
Karim says he’ll begin by giving us the nutshell version of the paper he will discuss with us today. Innovation systems create incentives to exert innovative effort and encourage the disclosure of knowledge. The timing and the form of the disclosures differentiates systems. E.g., Open Science tends to publish when near done, while Open Source tends to be more iterative. The paper argues that intermediate disclosures (as in open source) dampen incentives and participation, yet lead to higher perrformance. There’s more exploration and experimentation when there’s disclosure only at the end.
Karim’s TL;DR: Disclosure isn’t always helpful for innovation, depending on the conditions.
There is a false debate between closed and open innovation. Rather, what differentiates regimes is when the disclosure occurs, and who has the right to use those disclosures. Intermediate disclosure [i.e., disclosure along the way] can involve a range of outputs. E.g., the Human Genome Project enshrined intermediate disclosure as part of an academic science project; you had to disclose discoveries within 24 hours.
Q: What constitutes disclosure? Would talking with another mathematician at a conference count as disclosure?
A: Yes. It would be intermediate disclosure. But there are many nuances.
Karim says that Allen, Meyer and Nuvolari have shown that historically, intermediate disclosure has been an important source of technological progress. E.g., the Wright brothers were able to invent the airplane because of a vibrant community. [I'm using the term "invent" loosely here.]
How do you encourage continued innovation while enabling early re-use of it? “Greater disclosure requirements will degrade incentives for upstream innovators to undertake risky investment.” (Green & Scotchmer; Bessen & Maskin.) We see compensating mechanisms under regimes of greater disclosure: E.g., priority and citations in academia; signing and authorship in Open Source. You may also attract people who have a sharing ethos; e.g., Linus Torvalds.
Research confirms that the more access your provide, the more reuse and sharing there will be. (Cf. Eric von Hippel.) Platforms encourage reuse of core components. (cf. Boudreau 2010; Rysman and Simcoe 2008) [I am not getting all of Karim's citations. Not even close.]
Another approach looks at innovation as a problem-solving process. And that entails search. You need to search to find the best solutions in an uncertain space. Sometimes innovators use “novel combinations of existing knowledge” to find the best solutions. So let’s look at the paths by which innovators come up with ideas. There’s a line of research that assumes that the paths are the essential element to understand the innovation process.
Mathematical formulations of this show you want lots of people searching independently. The broader the better for innovation outcomes. But there is a tendency of the researchers to converge on the initially successful paths. These are affected by decisions about when to disclose.
So, Karim and Kevin Boudreau implemented a field experiment. They used TopCoder, offering $6K, to set up a Med School project involving computational biology. The project let them get fine-grained info about what was going on over the two weeks of the contest.
700 people signed up. They matched them on skills and randomized them into three different disclosure treatments. 1. Standard contest format, with a prize at the end of each week. (Submissions were automatically scored, and the first week prizes went to the highest at that time.) 2. Submitted code was instantly posted to a wiki where anyone could use it. 3. In the first week you work without disclosure, but in the second week submissions were posted to the wiki.
For those whose work is disclosed: You can find and see the most successful. You can get money if your code is reused. In the non-disclosure regime you cannot observe solutions and all communications are bared. In both cases, you can see market signals and who the top coders are.
Of the 733 signups from 69 different countries, 122 coders submitted 654 submissions, with 89 different approaches. 44% were professionals; 56% were students. The skewed very young. 98% men. They spent about 10 hours a week, which is typical of Open Source. (There’s evidence that women choose not to participate in contests like this.) The results beat the NIH’s approach to the problem which was developed at great cost over years. “This tells me that across our economy there are lots of low-performing” processes in many institutions. “This works.”
What motivated the participants? Extrinsic motives matter (cash, job market signals) and intrinsic motives do too (fun, etc.). But so do prosocial motives (community belonging, identity). Other research Karim has done shows that there’s no relation between skills and motives. “Remember that in contests most people are losing, so there have to be things other than money driving them.”
Results from the experiment: More disclosure meant lower participation. Also, more disclosure correlated with the hours worked going down. The incentives and efforts are lower when there’s intermediate disclosure. “This is contrary to my expectations,”Karim says.
Q: In the intermediate disclosure regime is there an incentive to hold your stuff back until the end when no one else can benefit from it?
A: One guy admitted to this, and said he felt bad about it. He won top prize in the second week, but was shamed in the forums.
In the intermediate disclosure regime, you get better performance (i.e., better submission score). In the mixed experiment, performance shot up in the second week once the work of others was available.
They analyzed the ten canonical approaches and had three Ph.D.s tag the submissions with those approaches. The solutions were combinations of those ten techniques.
With no intermediate disclosures, the search patterns are chaotic. With intermedia disclosures, there is more convergence and learning. Intermediate disclosure resulted in 30% fewer different approaches. The no-disclsoure folks were searching in the lower-performance end of the pool. There was more exploration and experimentation in their searches when there was no intermediate disclosure, and more convergence and collaboration when there is.
Increased reuse comes at the cost of incentives. The overall stock of knowledge created is low, although the quality is higher. More convergent behavior comes with intermediate disclosures, which relies on the stock of knowledge available. The fear is that with intermediate disclosure , people will get stuck on local optima — path dependnce is a real risk in intermediate disclosure.
There are comparative advantages of the two systems. Where there is a broad stock of knowledge, intermediate disclosure works best. Plus the diversity of participants may overcome local optima lock-in. Final disclosure [i.e., disclosure only at the end] is useful where there’s broad-based experimentation. “Firms have figured out how to play both sides.” E.g., Apple is closed but also a heavy participant in Open Source.
Q: Where did the best solutions come from?
A: From intermediate disclosure. The winner came from there, and then the next five were derivative.
Q: How about with the mixed?
A: The two weeks tracked the results of the final and intermediate disclosure regimes.
Q: [me] How confident are you that this applies outside of this lab?
A: I think it does, but even this platform is selecting on a very elite set of people who are used to competing. One criticism is that we’re using a platform that attracts competitors who are not used to sharing. But rank-order based platforms are endemic throughout society. SATs, law school tests: rank order is endemic in our society. In that sense we can argue that there’s a generalizability here. Even in Wikipedia and Open Source there is status-based ranking.
Q: Can we generalize this to systems where the outputs of innovation aren’t units of code, but, e.g., educational systems or municipal govts?
Q: We study coders because we can evaluate their work. But I think there are generalizations about how to organize a system for innovation, even if the outcome isn’t code. What inputs go into your search processes? How broad do you do?
Q: Does it matter that you have groups that are more or less skilled?
A: We used the Topcoder skill ratings as a control.
Q: The guy who held back results from the Intermediate regime would have won in real life without remorse.
A: Von Hippel’s research says that there are informal norms-based rules that prevent copying. E.g., chefs frown on copying recipes.
Q: How would you reform copyright/patent?
A: I don’t have a good answer. My law professor friends say the law has gone too far to protect incentives. There’s room to pull that back in order to encourage reuse. You can ask why the Genome Project’s Bermuda Rules (pro disclosure) weren’t widely adopted among academics. Academics’ incentives are not set up to encourage automatic posting and sharing.
Q: The Human Genome Project resulted in a splintering that set up a for-profit org that does not disclose. How do you prevent that?
A: You need the right contracts.
This was a very stimulating talk. I am a big fan of Karim and his work.
Afterwards Karim and I chatted briefly about whether the fact that 98% of Topcoder competitors are men raises issues about generalizing the results. Karim pointed to the general pervasiveness of rank-ordered systems like the one at TopCoder. That does suggest that the results are generalizable across many systems in our culture. Of course, there’s a risk that optimizing such systems might result in less innovation (using the same measures) than trying to open those systems up to people averse to them. That is, optimizing for TopCoder-style systems for innovation might create a local optima lock-in. For example, if the site were about preparing fish instead of code, and Japanese chefs somehow didn’t feel comfortable there because of its norms and values, how much could you conclude about optimizing conditions for fish innovation? Whereas, if you changed the conditions, you’d likely get sushi-based innovation that the system otherwise inadvertently optimized against.
[Note: 1. Karim's point in our after-discussion was purely about the generalizability of the results, not about their desirability. 2. I'm trying to make a narrow point about the value of diversity of ideas for innovation processes, and not otherwise comparing women and Japanese chefs.]
Tagged with: innovation
• open source
Date: March 5th, 2014 dw
For $3 at a library book sale I picked up a copy of Releasing an Independent Record, revised 4th edition, by Gary Hustwit, published in 1994 by Rockpress Publishing Co. The short review is: Times have changed.
Gary’s advice is that if you want to get your music out, don’t go to one of the existing labels. Start your own. In 1993, that was pretty radical even though it required you to emulate the major labels’ processes, albeit starting from scratch and with no budget. So,the bulk of Gary’s manual is a directory of the services you’ll need to hire. He assumes you’ve already got a tape of your music. So, now you need to find a tape duplication house. You also need to get the paperwork done to set up your label’s bank account, and don’t forget the rubber stamp: “Depending on what formats you release, you’ll need a ton of different sized envelopes, and stamping the return address is easier than having them printed or writing it by hand.”
There are also handy, multi-page lists of the press to contact and the local radio stations (remember them?) to flog your songs to. And booking agents and promoters. And record labels so you can “See if your label name is already taken.” Oh, and you might want to check “if they’re interested in licensing your record.”
A quick google reveals that Gary is now a director of documentaries. I saw and liked Helvetica, and Objectified is on my Netflix list.
On the last page, there’s an ad for Rockpress’ other four books. My favorite is Hell on Wheels, by Greg Jacobs:
A compilation of tour stories from 40 bands, including ALL, aMINIATURE, Babes in Toyland, Big Drill Car, Buck Pets, Buffalo Tom, Butthole Surfers, Cadillac Tramps, Chune, Circle Jerks, Coffin Break, The Cult, Descendents, Doughboys, The Dwarves, Ethyl Meatplow, fIREHOSE, The Germs, God Machine, Kill Sybil, King Missile, L7, Luscious Jackson, Mary’s Danish, Melvins, Minutemen, Naked Raygun, Overwhelming Colorfast, Popdefect, Rockets from the Crypt, Screaming Sirens, Skin Yard, Superchunk, Supersuckers, Surgery, UK Subs, and X.
I recognize a couple —it’s not my demographic, people — but that list’s got a bit of Key and Peele about it, don’t you think?
Tagged with: music
Date: February 22nd, 2014 dw
Peter Cappelli has written an excellent post at HBR that falls simultaneously into the “Well, Duh” and “Needs to Be Said” bins: “It’s Not OK That Your Employees Can’t Afford to Eat.” Well, duh! It’s amazing that it even needs to be said. (Note that the Duh belongs not to Peter but to whomever needed to hear that.)
Let me put it differently. From my point of view, here are the two fundamental objectives for any business:
1. Enable your customers to lead lives that are a little bit better.
2. Enable your employees to lead good lives.
Profit is what you use to do both of those things.
Tagged with: business
Date: December 17th, 2013 dw
HBR.com has just put up a post of mine about some new guidelines for “paid content.” The guidelines come from the PR and marketing communications company Edelman, which creates and places paid content for its clients. (Please read the disclosure that takes up all of paragraph 4 of my post. Short version: Edelman paid for a day of consulting on the guidelines. And, no, that didn’t include me agreeing to write about the guidelines)
I just read the current issue of Wired (Aug.) and was hit by a particularly good example. This issue has a two-page spread on pp. 34-35 that features an info graphic that is stylistically indistinguishable from another info graphic on p. 55. The fact that the two pager is paid content is flagged only by a small Shell logo in the upper left and the words “Wired promotion” in gray text half the height of the “article’s” subhead. It’s just not enough.
Worse, once you figure out that it’s an ad, you start to react to legitimate articles with suspicion. Is the article on the very next page (p. 36) titled “Nerf aims for girls but hits boys too” also paid content? How about the interview with the stars of the new comedy “The World’s End”? And then there’s the article on p. 46 that seems to be nothing but a plug for coins from Kitco. The only reason to think it’s not an ad in disguise is that it mentions a second coin company, Metallium. That’s pretty subtle metadata. Even so, it crossed my mind that maybe the two companies pitched in to pay for the article.
That’s exactly the sort of thought a journal doesn’t want crossing its readers’ minds. The failure to plainly distinguish paid content from unpaid content can subvert the reader’s trust. While I understand the perilous straits of many publications, if they’re going to accept paid content (and that seems like a done deal), then this month’s Wired gives a good illustration of why it’s in their own interest to mark their paid content clearly, using a standardized set of terms, just as the Edelman guidelines suggest.
(And, yes, I am aware of the irony – at best – that my taking money from Edelman raises just the sort of trust issues that I’m decrying in poorly-marked paid content.)
Tagged with: ethics
• paid content
Date: July 22nd, 2013 dw
Greg Silverman [twitter:concentricabm], the CEO of Concentric, has a good post at CMS Wire about the democratization of market analysis. He makes what seems to me to be a true and important point: market researchers now have the tools to enable them to slice, dice, deconstruct, and otherly-construct data without having to rely upon centralized (and expensive) analytics firms. This, says Greg, changes not only the economics of research, but also the nature of the results:
The marketers’ relationships with their analytics providers are currently strained as a service-based, methodologically undisclosed and one-off delivery of insights. These providers and methods are pitted against a new generation of managers and executives who are “data natives” —professionals who rose to the top by having full control of their answering techniques, who like to be empowered and in charge of their own destinies, and who understand the world as a continuous, adaptive place that may have constantly changing answers. This new generation of leaders likes to identify tradeoffs and understand the “grayness” of insight rather than the clarity being marketed by the service providers.
He goes on to make an important point about the perils of optimization, which is what attracted the attention of Eric Bonabeau [twitter:bonabeau], whose tweet pointed me at the post.
The article’s first point, though, is interesting from the point of view of the networking of knowledge, because it’s not an example of the networking of knowledge. This new generation of market researchers are not relying on experts from the Central Authority, they are not looking for simple answers, and they’re comfortable with ambiguity, all of which are characteristics of networked knowledge. But, at least according to Greg’s post, they are not engaging with one another across company boundaries, sharing data, models, and insights. I’m going to guess that Greg would agree that there’s more of that going on than before. But not enough.
If the competitive interests of businesses are going to keep their researchers from sharing ideas and information in vigorous conversations with their peers and others, then businesses simply won’t be as smart as they could be. Openness optimizes knowledge system-wide, but by definition it doesn’t concentrate knowledge in the hands of a few. And this may form an inherent limit on how smart businesses can become.
Tagged with: 2b2k
Date: May 30th, 2013 dw
CNN.com has posted my op-ed about why where you work is not about the quality of your life so much as about the substance of it.
Judging from some of the reaction, I should emphasize that if the only way to save Yahoo were to require everyone to come to work every day, that would certainly be the right decision. But it seems clear to me that Marissa Mayer was sending a signal with this policy, for surely there are some people who were working productively from home. So, if the new policy is a signal and is not actually required to save Yahoo, then I think she has underestimated how disruptive a signal it is. [To late to stick in a spoiler notice? That was the essence of my op-ed.]
Also, CNN.com has stripped out the links, I’m pretty sure unintentionally. Here they are:
Tagged with: business
Date: March 2nd, 2013 dw
Dylan Tweney notes that Lotus Notes, which invented a bunch of the enterprise collaboration stuff we now take for granted, has become a drag on IBM’s revenues. Dylan writes:
I used it extensively at several companies I worked with. Initially, it was mysterious and powerful. Like most end-users of Lotus Notes, I used it primarily as an email program. It had its quirks, but it worked. But there was another dimension to Notes, a powerful, programmable backend that let you create databases and workspaces for collaborative work, contact management, information sharing, and communication.
Today, we’d call it a collaboration tool or a corporate social-media tool, and it would be web-based and standards-compliant, like Yammer, Jive, and Huddle. In the absence of standards, Notes’ engineers had to invent everything themselves, making it a clever but proprietary solution.
But long before those web-based startups came along, Notes was already losing its cool. The client software became huge and bloated. It was expensive to implement and difficult to customize.
I think I’m legally not supposed to remember that in about 1995, a company I worked for — Open Text — ran a full-page ad in the Wall Street Journal proclaiming “Notes is dead.” ‘Twas the Web that killed it, the ad claimed. I was Marketing VP at Open Text at the time, but the ad was conceived and placed by a different VP. I didn’t hear about it until the morning it appeared; Open Text was that sort of place. And, yes, a lawyer did call us rather promptly.
Anyway, 18 years later, it seems like that bold headline might be coming true.
To be fair, it was true enough at the time. Notes has hung on primarily as an email tool, not living up to its promise as an enterprise collaboration system. And that was indeed because the Web came along with more open solutions that ran in browsers. Eventually. It took a visionary to think that the crappy browsers of that era would someday host fullscale apps — I floated the phrase “client/surfer architecture” but it never took off — but Netscape had such visionaries, and so did Open Text in the form of its CEO, Tom Jenkins.
Lotus Notes was noble software. Brilliant idea. Immensely powerful. But once the Web happened, the jig was up. It took about a decade for enterprises to be willing to trust their mainstream collaborative processes to the Web and its browsers. But eventually Web clients scaled up in power, functionality, and robustness…enabling systems far beyond what the old proprietary backend systems could manage.
Conclusion: There is no escaping Ozymandias.
Tagged with: collaboration
Date: January 22nd, 2013 dw
Last week I paid a visit to my old PC store, ICG Computers, in Brookline. I hadn’t been there in maybe 5 years because I switched to Macs and thus spend time at the Computer Loft in Brighton. Also, when I was a PC, I was building my own computers out of components, so lots more went wrong (= I did lots more wrong). And, yes, I wish I could compile my own hardware and install the Mac OS on it. (Hackintosh scares me. Someday.) But, my remaining Windows machine crapped out last week, so I carried it to ICG’s small storefront.
Ray greeted me by name. Because no one else was there, we took the opportunity to catch up.
Ray comes from China and runs a quintessential American small business. He’s honest as the day is long, and could teach any bigger company about customer service. But it’s been a lean few years for ICG. Ray says that the recession hurt his primary customer base, small businesses. There haven’t been a lot of new businesses formed, so they’re not coming in to equip their offices. And, of course, the PC business has gotten commoditized. So, ICG relies on repairs and aggressively trying to beat the Internet on prices.
The walls of the store are lined with components. Then there are a few tables of new and used machines. He prices his used machines against eBay, and his new machines against Net low-ballers. As a result, you can get a power-packed laptop for $250 or $300. And you can do so knowing that Ray knows the tech and stands behind what he sells.
ICG is a great place to buy a computer. It’s also a great place to hang out and talk about tech. Ray knows my own level of expertise and talks at that level. No condescension, no salesmanship, no BS. I always learn something talking with Ray. In this case it turns out that my PC needed a new power supply, and the one I’d put in was under-powered. So, yeah, Ray upsold me, but I have complete confidence that he also right-sold me, so to speak.
Bunches of small, locally-owned computer stores have gone out of business here over the past few years. So have most of the larger stores. Remember EggHead? CompUSA? Me neither. And much as I love the Internet, I hate what it’s doing to the Rays of our town, who epitomize the best of small business. ICG is surviving and will continue to serve our community. But I want Ray’s business to do more than that. It seems unfair that honesty, expertise, friendliness, and low, fair prices aren’t enough for a business to go gangbusters.
Am I plugging ICG? Damn straight.
Tagged with: brighton
Date: October 20th, 2012 dw
My latest column in KMWorld is about why your business needs scholars. In fact, though, it’s about why the idea of scholarship is more helpful than focusing your thinking on knowledge.
, too big to know
Tagged with: 2b2k
Date: October 1st, 2012 dw
Mitt Romney is taking some flack for using some notoriously flaky science as his example of good science. But in the same passage he betrays a Big Corporate view of how innovation works that should cost him the support of every entrepeneurial startup in the country.
Here’s the passage from his Washington Examiner interview (with a hat tip to BoingBoing):
CARNEY: What role should government have in promoting certain industr
And keep in mind that Romney here is not talking about the auto industry specifically; rather, he is explaining why governments ought not to back entrepreneurial companies. It’s not just that governments are bad at picking winners, it’s that when the winners are startups — even when they’re way out of the prototypical garage — they’re unlikely to get past “delight.” So, wies or economic activities such as homeownership, or manufacturing, renewable energy or fossil fuel energy, eBig Corp xports, or just advanced technology? What sort of subsidies and incentives do you favor? You had some of these in Massachusetts, I know.
ROMNEY: Very limited — my answer Big Corp to your first question. I’m not an advocate of industrial policy being formed by a government. I do believe in the power of free markets, and when the government removes the extraordinary burdens that it puts on markets, why I think markets are more effective at guiding a prosperous economy than is the government.
So for instance, I would not be investing massive dollars in electric car companies in California. I think Tesla and Fisker are delightful-looking ve
And keep in mind that Romney here is nBig Corp ot talking about the auto industry specifically; rather, he is explaining why governments ought not to back entrepreneurial companies. It’s not just that governments are bad at picking winners, it’s that when the winners are startups — even when they’re way out of the prototypical garage — they’re unlikely to get past “delight.” So, whicles, but I somehow imagine that Toyota, Nissan, and even General Motors will produce a more cost-effective electric car than either Tesla or Fisker. I think it is bad policy for us to be investing hundreds of millions of dollars in specific companies and specific technologies, and developing those technologies.
I do believe in basic science. I believe in participating in space. I believe in analysis of new sources of energy. I believe in laboratories, looking at ways to conduct electricity with — with cold fusion, if we can come up with it. It was the University of Utah that solved that. We somehow can’t figure out how to duplicate it.
So, first the problem with his science remark. I understand that he’s boosting Utah. But the 1989 experiment by Stanley Pons and Martin Fleischmann was famous not only because it could not be replicated, but because it was prematurely hyped by Pons and Fleischmann before it had gone through peer review or had been replicated. (As BoingBoing points out, the Wikipedia article is worth reading.) No matter what you think of the experiment, it is a terrible example to use as proof that one appreciates basic science…unless you’re citing the rejection of the Pons-Fleischmann results, which Romney explicitly was not. The issue is not merely that Romney continues to believe in a discredited claim. The real issue is that this suggests that Romney doesn’t understand that science is a methodology, not merely the results of that methodology. That’s scary both for a CEO and for a possible president.
I’m at least as bothered, however, by Romney’s casual dismissal of entrepreneurial startups as a source of innovation: “I think Tesla and Fisker are delightful-looking vehicles, but I somehow imagine that Toyota, Nissan, and even General Motors will produce a more cost-effective electric car than either Tesla or Fisker.” “Delightful” is a dismisive word in this context, as evidenced by the inevitability of the “but” that follows it. Romney, it seems, doesn’t believe that startups can get beyond delight all the way to the manly heavy lifting that makes innovation real. For that you need the established, massive corporations.
Wow. Could there be a more 20th century vision of how a 21st century entrepreneurial economy should work?
Tagged with: economy
Date: September 7th, 2012 dw
Next Page »