I have discussed the topic of quality and testing with a few robotics startups and the conversation tends to reach this consensus: formal quality assurance processes have no place in a startup. While I totally appreciate this view, this blogpost provides and alternative approach to quality for robotics startups.

The main priority of many startups is to produce something that will attract investment – it has to basically work well enough to get funding. Investors, customers and users can be very forgiving of quality issues, especially where emerging tech is involved. Startups should deliver the right level of quality for now and prepare for the next step.

In a startup, there is not likely to be any dedicated tester or quality strategy. Developers are the first lines of defence for quality – they must bake it in to the proof of concept code – they might do this with unit tests. The developers and founders probably do some functional validation. They might experience more extreme use cases when demo’ing the functionality. They might do limited testing with real life users.

What are the main priorities of the company at this phase and the matching levels of quality? The product’s main goal, initially, is to fulfil requirements of application development, demo’ing, and to be effective and usable to its early adopters. Based on these priorities, I’ve come up with some quality aspects that could be useful for robotics startups.

A Good quality demo

Here are some aspects of quality which could be relevant for demoing:

Portable setup Can be transported without damaging the robot and supporting equipment Is possible to explain at airport security if needed Works under variable conditions in customer meeting room Poor wifi connections Power outlets not available Outside of company network Uneven floors Stairs Noise Different lighting Reflective surfaces Will work for the duration of the demo Demo will be suitable for audience Demo’ed behaviour will be visible and audible from a distance, e.g. in a boardroom Mode can be changed to a scripted mode for demos Functionality actually works and can be shown – a checklist of basic functionality can take away the guesswork, without having to come up with heavy weight testcases

Quality for the users and buyers

The robot needs to prove itself fit for operation:

Functionality works What you offer can be suitably adapted for the customer’s actual scenario Every business has its own processes and probably the bot will have to adapt to match terminologies workflows and scenarios that fit the users processes Languages can be changed Bot is capable of conversing at the level of the target audience (e.g. children, elderly) Bot is suitable for the context where its intended to work like a hospital or school, will not make sudden movements or catch on cables Reliability Users might be tolerant to failures up to a certain extent, until it gets too annoying or repetitive, or if they cannot be recovered from Failures might be jarring for vulnerable users like the mentally or physically ill Is the robot physically robust enough to interact with in unplanned ways? Security Will port scanning or some other exploitative attacks easily reveal vulnerabilities which can result in unpredictable or harmful behaviour Can personal data be hijacked through the robot Ethical and moral concerns Users might not understand that there is no consciousness interacting with them, thinking the robot is autonomous There might be users who think their interactions will be private while they might be reviewed for analysis purposes Users might not realise their data will be sent to the cloud and used for analysis Legal and support issues What kind of support agreement does the service provider have with the robot manufacturer and how does it translate to the purchaser of the service?

Quality to maintain, pivot and grow

During these cycles of demoing to prospects, defects will be identified and need to be fixed. Customers will give advice or provide input on what they were hoping to see and features will have to be tweaked or added. The same will happen during research and test rounds at customers, and user feedback sessions.

The startup will want to add features and fix bugs quickly. For this to occur, it will help them to have good discipline with clean code which is maintainable, and at least unit tests to give quick feedback on change quality. They will but hopefully also have some functional (and a few non functional) acceptance tests.

When adoption increases, the startup might have to do a quick pivot to a new application, or to be able to scale to more than one customer or usecase. At this phase, probably a lot of refactoring will happen to make the existing codebase scalable. In this case, good unit tests and component tests will be your best friend, and ensure you are able to maintain the stability of the functionality you already have (as mentioned in this techcrunch article on startup quality).

Social robot companies are integrators – ensure quality of integrated components

As a social robotics startup, if you are not creating your own hardware, OS, or interaction and processing components, you might want to consider becoming familiar with the quality of any hardware or software components you are integrating with. Some basic integration tests will help you keep confident that the basics work when an external API is updated, for instance. It’s also worth consider your liability when something goes wrong somewhere in the chain.

Early days for robot quality

To round up, it seems to indeed be early days to be talking about social robot quality. But it’s good for startups to be aware of what they are getting into because this topic will no doubt become more relevant as their company grows. I hope the above post can help robotics startups now and in the future to ensure they stay in control of their quality as they grow.

Feel free to contact me if you have any ideas or questions about this topic!

Thanks to Koen Hendriks of Interactive Robotics, Roeland van Oers at Ready for Robotics and Tiago Santos at Decos, as well as all the startups and enthusiasts I have spoken to over the past year for input into this article.