When your average tech journalist wants to talk about the success of a new product or service, he has a lot to draw on. Netflix subscriptions can be compared to cable subscriptions, iPhone sales can be compared to Android sales. Even the number of reported returns or one star Amazon reviews can tell you a lot about the success of a company’s new product.

Unfortunately, in the world of crowd funding, there isn’t a whole lot to go on when trying to gauge success. Many crowd funded products are too unique to compare to products already on the market. Others attempt to butt heads with established brands, but journalists can’t verify their claims without review units. Readers want to hear who won and who lost, but the world of product development makes that distinction cloudy.

Fortunately, Kickstarter, Indiegogo, and just about every other crowd funding site offer an arbitrary and random way to gauge the success of a campaign without drawing on information from any of the established and reliable sources mentioned above: the funding goal.

What should it cost?

When trying to gauge the likelihood of a particular crowd funding campaign making it to market, we at DropKicker often look at how much money they’re asking for and draw on our experience in product development to determine if it’s enough. Most people have no idea what it costs to develop a product, so it’s worth talking about it a little bit here before continuing on.

Any mass-produced product is going to require some level of tooling. Tooling can be as simple as cutting a stencil out of a plastic sheet or as complicated as inventing an entirely new kind of machine to perform a task. The idea is that you pay an up-front tooling cost a single time and use the resulting tool to create a large quantity of product very quickly and cheaply. As an example, consider injection molding. Many electronic devices with plastic enclosures will require a custom injection-molding tool to be fabricated. This tool usually consists of two metal parts (aluminum or steel) with a cavity carved out between them. When pressed together and injected with plastic under pressure, they can pop out a plastic shape in a matter of seconds.

These tools are not cheap.

Every product is different, so there isn’t a fixed price for an custom mold, but I have found a few resources that should give an idea. This page lists a few examples of extremely low-precision molds (bird feeder part, cardboard sign holder, etc) that cost $3-5000 to tool. This cost estimator provides some examples for higher precision molds that can easily cost over $15,000. This paper attempts to create a formula for predicting tool cost using a table of previous job prices some of which are in excess of $140,000. That’s just talking about price. Many tools take a number of months to design and manufacture.

Whether it’s injection molding, PCB manufacturing, product assembly, or raw material procurement, these are the kinds of up-front costs and delays that make product development so difficult and expensive. Many of the simple products that you interact with on a daily basis cost over a million dollars to develop. Fortunately, once developed, they only cost a few dollars to manufacture, so the hope is to make back the money in volume. Crowdfunding makes it easy to get this volume money up-front.

Poor estimates

The amount of experience required to estimate the above costs accurately is substantial, but pretending you can doesn’t take nearly as much. Let’s take a look at Amiigo on Indiegogo:

This video and the associated website have a high level of polish that matches the style of many successful crowd funded campaigns. Unlike many of those campaigns however, they are asking for a pitifully small $90,000 and six months to make a sophisticated waterproof wireless electronic device using never-before-seen signal processing techniques. Just the tools for the rubber wrist band could easily eat up a third of their money and most of their time.

Regardless of whether or not this is enough money on its own, it should stand to reason that by blowing past their goal by almost a half million dollars, they should have had more than enough resources available to deliver on time. Sadly, despite their promised June 2013 ship date, they still haven’t managed to ship a product. Their most recent update is dated September 2013 where they state that the new mid-October delivery date outlined in the previous update will have to be pushed back a month due to unexpected lead times. The comments section is rife with people requesting refunds.

So why didn’t they succeed? A cursory glance at their team might provide some answers. David Scott and Abe Carter have degrees in economics and finance respectively (class of 2012); from what I could find online Nahom Workie and Max Mann are apparently still undergrad students at MIT; Daniel Klöhn is a visual effects guru; and Zac Park likes to refer to raising money on Indiegogo as “selling product“. None of these people have any experience in product development despite their excellent marketing skills.

Without access to their project plan, I have to assume based on their level experience that they didn’t have a detailed plan of how they were going to make this product happen for such a small amount of money and in such a short time. Tooling alone could take four to five months, and even with the half million they raised, they still might not have enough cash to deliver. Of course, this is easy to say in hindsight; back in early 2013, the tech media was happy to report on Amiigo’s “success”:

The ability of a team to surpass their fundraising goal is only a measure of their marketing skills and not their product development skills. Considering most backers are more interested in receiving a product than giving away money, “success” should be measured only after the former takes place. Using a fundraising goal to gauge performance is paramount to asking a team “you guys got this or no? Be honest”.

That isn’t to say that you can’t try to judge success ahead of time; you would just need some sort of outside reference to judge how much cash a project will require. The average journalist or backer could learn to identify unrealistically low funding goals, but unfortunately it’s even more complicated than that. Even when derived accurately by competent people, not all funding goals mean the same thing.

Two kinds of goals

Kickstarter’s blog states:

Kickstarter’s mission is to help bring creative projects to life. That’s our mission. We’re a tool available to anyone (in the US and UK, currently) to fund and build a community around their creative project.

Summarizing, Kickstarter exists for two reasons.

To fund a creative project.

To build a community around a creative project.

At first glance, it may not make sense to draw this statement out into two separate bullet points, but when looking at Kickstarter through the eyes of a product development engineer, it makes it clear that it is used in precisely these two distinct ways. Let’s look at an example for the first bullet point. A quick search through Kickstarter’s front page found LUXPONG:

This simple electronic toy utilizes open-source Arduino-compatible hardware to drive a strand of programmable LEDs and create a simple game where two players can face off to see who has the best reaction time. The device is sold as a completed product or as a kit that can be assembled by hobbyists. The kit includes a small controller board, a strand of LEDs, two buttons, a battery holder and a power switch. These are all off-the-shelf parts that won’t require tooling, and the optional enclosure looks like it’s just a milled out plank of wood.

According to the team behind the project, they need $25,000. They detail why they need the money:

This is a reasonable set of requirements. Depending on product complexity, UL compliance testing can run several thousand dollars, and getting a manufacturer to even talk to you is going to cost at least that much. The team’s video has a modest level of production value, and it really looks like they’re counting on Kickstarter funding to make their product a reality. If they fail to fund, it doesn’t look like they’ve put too much money on the line, so they can afford to walk away.

This project clearly falls into the first group. They’ve got an idea for a product and want to make it happen, but they can’t afford to risk the up-front development costs.

Now let’s look at the second category with Moment lens:

This project plans to deliver a precision made collection of glass lenses that will mount magnetically to a plate adhered to the back of a smartphone. The lenses are precisely manufactured with advanced techniques to reduce chromatic aberration and image distortion. They already have several fully functioning prototypes which they have used to generate a gallery of images displaying what their lenses can do, and the level of polish on the rest of their campaign is impressive (just look at their website). Despite the level of money invested, and the quality and complexity of the final product, they’re only asking for $50,000 which is only twice what the substantially less sophisticated LUXPONG needs.

Given the quality of its presentation, it comes as no surprise that the list of people behind the project includes Richard Tait, the creator of the board game Cranium who sold it to Hasbro for $77 Million dollars and Marc Barros, founder of Contour Cameras, who raked in $27.3 million dollars in revenue in 2011.

The project video claims that they “need your help” to make the product happen, but it’s pretty clear that these people could drum up $50,000 if they really wanted. They either have enough money on hand, or know enough people in the industry to get that money through some private source. It’s also pretty clear that $50,000 isn’t nearly enough to start manufacturing their product as they’ve probably spent about half that much just on the marketing materials alone. Lens manufacturing isn’t cheap, and even if all of the money raised goes towards tooling costs as they’ve claimed, they’re going to need more than $50,000 to make something this sophisticated.

This project clearly falls into the second category. They already have the funding, but they’re looking to build up a community of people interested in their product. In other words, they’re using Kickstarter as a means to advertise cheaply. Some people might complain that independently wealthy people or people with good professional connections (such as Zach Braff) shouldn’t be using the same service as Joe Entrepreneur, but Kickstarter has already made it clear that they have no problem with this.

While some people will use Kickstarter as the only lifeline for their burgeoning idea, others will use it for cheap advertising on a project that is well on its way to eventual productization success. The real question is: how do you gauge the success of each project campaign? Which is more impressive? Two dudes with an interest in electrical engineering drumming up $1,596 on their first product or a team of independently wealthy and well connected product development experts surpassing an arbitrarily low and ultimately unnecessary funding goal?

Big surprise!

When a team asks for an abysmally small amount of money, it doesn’t necessarily mean that they will fail. With no context, surpassing a goal by 622% sounds impressive and builds a lot of excitement. Given the level of buzz generated by projects that reach their goal unusually fast, I would argue that many project creators set their funding goals intentionally low just so they can “Kaboom” past them. In fact, Isabel Hoffman, co-founder of Tellspec offered that technique as a “tip“:

In the end, it doesn’t really matter if a project needs the money from its funding goal or not; all that matters is that they deliver, and in the case of the two examples above, it’s pretty clear that they will both have the resources required to ship a product after reaching the funding goals they provided.

So what then?

Gauging success is a tough problem to solve. Project creators are not required to specify what their backer money will be used for, and even if they were, it’s impossible for Kickstarter or Indiegogo to really verify that their estimates are accurate. A funding goal can mean a number of things, and surpassing it isn’t always good news.

Every reporter wants to be able to say with conviction how successful a crowd funded campaign will be while the buzz is still building, but without a taxing and thorough investigation into the complexity of the project and the people behind it, it’s impossible to come up with anything substantive. Simply comparing the amount of money raised to the amount requested is much easier and will continue to be a crutch in the typical tech media despite how meaningless these numbers are.

And these numbers are meaningless. Smarty Ring is a ridiculously complicated wireless waterproof stainless steel fitted ring, and they’re only asking for $40k. Airfy wants to make a global network of wireless hotspots for $20k. To my knowledge, nothing is stopping a new campaign from setting a goal of $1 just so they can say they reached theirs faster than anyone else.

Remember that the only people presenting the valuation of a project have a vested interest in its success, and they may have chosen their funding goal for a plethora of reasons that they don’t make clear. When backing a project, try not to get drawn into the buzz and excitement of “blasting” past a funding goal. Try to learn as much as you can about the company and the people behind it. Look at their prototypes. Think about their timeline. Get excited about their level of experience and level of enthusiasm. Just remember that surpassing a goal grants you a promise to try and not a promise to deliver.

Reaching the goal quickly promises nothing at all.

Amiigo, LUXPONG, Moment Lens