Powered by Google Cloud Speech-to-Text

welcome to the voice Tech podcast join me Carl Robinson in conversation with the world’s leading voice technology expert discovered the latest product tools and techniques and learn to build the voice app to the Future

incorporating usability testing early and often saves you time and makes your application a lot better

hello and welcome back to another episode of The Voice Tech podcast my name is Carl Robinson and today’s episode is entitled usability testing for voice you help me talk with Dylan zwick the chief product officer at Pulse Labs the officer remote testing on a platform The Voice app Developers

the post office team leaders in the field of voice application testing and the official partners with both Google and Amazon helping the tech Giant’s to test the money app submitted to both papon and in the process to find the best practices for voice application design it was a real pleasure to talk with Dylan his knowledge and experience really a second to none and as a result the number of inside per minute in this conversation is off the charts you learned so much about voice application design and testing will definitely want to keep a pen and paper handy do not conversation you will learn about the different types of testing including unit testing functional testing Certification testing and usability test we talked about how about you suppose he’s asking what and practice how to build you a panel of tests to fit your app how and when to Bass conduct usability test I don’t know how to interpret the results

we cover how to become a remote usability testing of South to join the pulse Labs panel in order to get access to the latest voice app

then that Dylan has some of the design lessons he’s learned from texting hundreds of voice apps for both Google and Amazon including the difference between improving an existing service with voice that says inventing Italian you use case invoice the importance of always moving forward in the flow wide variation of word and sentence structure can maintain attention and also how to add variety without increasing complexity

I recently spoke at the voice connected home conference in Cologne Germany it was fantastic event so many great people great presentations I will bring you my Tok as soon as I can and I possibly some of the other speakers talks about Ally and if you trap a site to look out for that I’m really excited about the Vita Tech conference coming up in Paris technology conference so do let me know if you going to be that is great fun loved it last year I’ll be great to meet up in person if you are in attendance

this podcast is sponsored by Manning publishing and independent publisher of computer books who really are the perfect fit for the knowledge testing voice Tech audience at money.com there’s a huge range of software development including voice applications for Alexa and Google assistant voice develop adjusting coats as well as many books on python JavaScript database is a much more with the money Early Access program you can read a book chapter by chapter while it’s being written this unique feature I give you access to up-to-the-minute knowledge on the hottest Technologies all the books are available at Manning. Calm and right now you can get 40% off old books with the promo code pod voice Tech 19

okay so now it’s my pleasure to bring you today’s guests Dylan zwick

I’m very excited to have on the line Dylan zwick from pulse Labs Dylan welcome to the show thank you so much for having me excited to be here at the subject of this episode will the chief product officer ants at Pulse Labs a seattle-based startup working to broil testing for developers working in the voice app space you wanted company station for the first Alex or accelerates a which is why I’m so stressed out if it’s alright to pack my text Dawn’s and your family director of data science I just thought that the design usability shops I’m sorry about pulse labs and what was it do and what clients you say I have a company that’s focused on providing basically testing and really user research solutions for developers and designers a voice application so

building a voice application weather be on you know Alexa or Google assistant and you want to have your application tested by real world users basically understand how real people are actually using or going to use your application we provide a platform and a panel for doing that so we have a large bedded panel of over ten thousand Saints of usability testers I in the United States Canada the UK France and Australia and we’re expanding that all the time and we also have a platform for essentially administering all these tests Gathering all the data we have to be able to get a level of data from the usability testing sessions that is has a depth of data that’s on unavailable anywhere else in the market today and then also kind of following up with post test questions and then aggregating goes all together for you and do a report

summarizes it but also let allows you to kind of dive as deep as you want to go see it at the centrally you know what pulse Labs does and then we are actually we have to handle all of the usability testing for Amazon’s manage partner program in North America so we have been testing with the marriage partner program for over a year now and through that and through our other customers as well we’ve actually done testing for applications from over a hundred Brands so we’ve seen a huge spectrum of voice applications and you’ll probably if there’s a skill out there that that you like we probably did we probably did testing for it and trust and yes I really got a huge huge wealth of experience right there on the on the testing side of things to just a recap some of that stuff things I was a lot by 27 Europe

your platform so for your voice app developer agency looking to do some YouTube is he testing this is remotes usability testing we should say and I’m able to plug in my my I get my beautiful me on the specifics but somehow I can give my app to selection of from the huge penile it sounds like you’ve got diarrhea of his ability test is all around the world that I can attest that they can see what they think they’re going to find bugs are going to come up with feedback I seem like going to end tonight into the platform somehow you would be like a gate that and then you send that back to me as a report which I can take action on exactly so that’s all aggravated but you can also you can also dig down and have to get the transcripts from all the usability testing sessions and even get the user Side Audio so if there’s some question about maybe there was a discrepancy between what the user heard what the user said and what the platform heard you can actually investigate that on our panel because I said that the level of data

it is unavailable anywhere else in the market today and dressing so you flag problems we’ve detected but you also give the world a tribute to the customer so they can they can spot things that maybe I don’t need that can spot cuz I know exactly at ten thousand on the penile that’s pretty interesting different testing channels are basically panelist recruitment channels that we use you know of anybody out there would like to be a tester for us to go to pulse Labs. Ai and sign up would love to have you and yeah over last year we actually build a panel as I said over 10,000 testers so you know what that means is that you can especially if you’re building an application for a Target demographic so if you got a particular type of end-user in mind you can specify that Target demographic when you’re sitting at the test and we can

make sure that the people who test your application are the six and keep play the amino profile people as the people that are going to be using your application when she wanted really important really important that they get it ready for I’d look like as well I guess when you really want to make sure that I speak that language who exactly exactly and it is also left to get so one of the more effective ways of doing testing with us also is to try to get a very broad span figure launching an application just within the United States well the English that is spoken languages are spoken throughout the United States varies enormously region-by-region and so I’ll be able to get a kind of a sampling of people from all over the country lets you kind of surface usability issues that might exist as a result of you know changes in

tax an idiom that you might see from one region to another Union necessarily get if you were only limited to testing within your kind of a certain local area so you guys really it is this the words that people are using even within one language that’s a huge variety messenger that wouldn’t have thought I’ve been thinking about lying in the UK as well go from north to south I know you’ve been on quite well and he talks about using slang of his voice assistant light Uniden father and that goes I never understands but yeah that’s exactly the kind of thing that you want to let do you want to catch for the for the panelists and I know you said you what you’re looking for panelist what what’s that what’s the process for getting involved if you coming out that was interesting being upon this cuz this is sounds like a really cool white so yeah so do you sign up for panels you basically just you can come to our website the set pulse Labs. Ai and click on the link that

what’s there and then you just basically say I want to sign up for the panelist and what we do then is you answer some basic demographic questions about yourself and you basically do a practice tests and so you know you had we have to, we have to make sure that the quality of your your feedback that you’re going to give is of a you know your meats are high bar for the type of quality of the type of feedback that is useful for a customer’s so you take the practice test and if the practice test goes well then you were formed your Dell member I testing pool and you will start getting invitations to two tests a very soon and in addition to being able to provide feedback on kind of what is coming out in the voice Marketplace and to help you know improve that make that better we also do compensator panelist

I’m alright said I said that the panel aside and it sounds like when we were saying that there’s a certain sense down to defeat about the US Patent I think I’ll be a good basis for us to ask the me twice later what does it take to do conduct a good usability test before we do that I just wanted to ask you just said something about high-level about what what the different types of testing our because obviously that’s not just usability testing I’m aware of unit testing integration testing all these times around testing device breakfast on usability testing so why why do you suppose he testing so important and what you guys are focused on that specifically so yeah absolutely so I just had their number different types of testing that are all you know very important for the development of the design of Any Given product and we’re actually expanding our testing offering to offering you an addition sort of functional testing also pre-certification testing particularly for those that are worth

with Amazon and they’re kind of certification management procedures but usability testing is you know important for is very important actually in the design and development of any product or any application but is particularly critical in the design of a voice application the reason for that is too full one is that human beings have a well-formed and 1/8 expectations of how conversations are supposed to go see how things are going to go and help people and how responses should work when were speaking we’ve been doing it all our lives and speaking with other humans all our lives and so I’ve got a wealth of experience and and expectations for when I’m talking to you and so when I start talking to a machine a lot of those expectations are going to be carried

and so if those expectations aren’t met then you just going to have a bad time the other aspect of the voice that makes usability testing important is that when you are when you’re building on voice you are much less constrained then if your building in a primary visual meeting you can’t really control or limit what your users are going to say in a primarily visual medium like this and so what that means is that in order to really make sure that your application is robust enough to handle the you know. Inputs you’re going to be getting from Real World users you need to actually get it out there in front of real-world users and have them test it and have them try it out before you launch to the public because

if your public users try your application and it give it fails to understand you know a way of phrasing one of their responses that you should be able to respond to but maybe it’s not prepared for that’s going to be a bad time and your you know your user is your your your initial users who you were so hard to try to get you get your tickets to try to wrap your application aren’t you have a good time we’re not going to be a Savoy Aces is much less structured it’s much more open so you can get an example a wide variety of responses as well as us being much more sensitive to any kind of IRAs with anything that sounding I’m not true so you absolutely have to take an endoparasite approach have to test that repeatedly tested with uses I guess in the design phase and I’m going as well as usual I do making update to the to the product as the usage of bulbs on your product is about all this is I seen or just a one-and-done kind of prosise it’s a it’s an ongoing thing

oh yeah absolutely and it’s not even a one-and-done process in the Prien before release the best engagements that we have by far are those that incorporate usability test think I’m very early on in the design development process and kind of didn’t really use it throughout the stages of the development process before launch and one of the big reasons for this is also if you if you wait until the end of the usability testing and you find out you have a lot of usability issues frequently addressing and fixing those issues require changes that affect a whole bunch of other aspects of your application and so changing and fixing those issues can be a lot more time consuming a lot a lot more difficult later on the process where if you just kind of incorporate usability testing early and often it actually saves you time and makes your application along set up multiple

go buses that the agile methodology and if you wait until right before the end of your testing the fit into that the whole lot La Jolla half frame White Band

the basic idea there is that you you put the Emmy we’d even on a lot of testing even when the application of prototype state so they might not even be in a development might not even started on it yet but you can still get feedback from users about okay here is kind of how how the experience is introduced to the to the to the end user

what are they understand when they experience that is what do they understand when they have expectations for what this application is going to do is going to be able to help them with based upon how that’s presented to them and then you can make sure that in your end and do they think that’s going to be useful you think you can make sure and you can tweak things so that you can get you can be certain that all right what we are trying to provide here and what we are providing here is actually understood and appreciated by the end users and then you can kind of see and then you can take the next step and say okay let’s build-out kind of the initial happy path here let’s see how users are interacting with that happy path how many people are going down that happy path and for those that get stuck there for those to go off go off the path why are they doing that and how can we guide that sounds very difficult because I was head that you can’t ask you to directly what do you want what do you like because of that that they’re not even show themselves in a lady would they make things up in order to provide it

I can understand the power of usability testing to watch and do things cuz then you get to sit and her today but how do you solve your problem asking them and I’m finding out why that that doing things when I was not element of consent to you, trust what I want you to the responses I give you

yeah absolutely so that was the best way that I’ve heard that encapsulated is believe everything that your users do half of what they say they do and very little about what they say they would do and so the best and most reliable day that you’re going to get is by actually observing them as they are interacting with your information and as I mentioned earlier that’s actually something that you can do with pulse labs and you can get a level of detail on those interactions that is that is more specific than the detail you can get pretty much anywhere else in the market what you can do effective leaders of the questions that you can ask them is that it’s not so much a question of you what would you like to see here it’s a question of when I presented this what was your answer

what we what were your expectations here when it when you when this initial user experience is presented to you what did you think this application was going to be able to do because frequently exactly there’s a misalignment between you know when you’re building an application you kind of know you know exactly what it’s supposed to do and if you work on it all the time everyday you think that everybody else is going to know exactly what I’m going to do exactly what it does so I would like but you can ask them what they were thinking out the time past tense I have a confession to make I was a project manager for a couple of years I’ve never conducted a usability test what what was guilty of really never doing I was just one of the reasons why I didn’t go so well so yeah

if I have I’m disgusted these kind of attendance with designers and I actually know these list of questions that you can ask you and he’s he’s busy testing and a lot of them I like could you show me how you would do something and then watch them do it and explain explain what you’re doing as you’re doing something as opposed to something hot to Pensacola something in the future you ask people get a gym membership will you go to the gym for five times a week in there like that yeah absolutely to know exactly what they would do but I do they do have a better understanding of what they what they do and some of the best way the best day to you’re going to get though is active serving them while they’re doing okay

The Voice Tech podcast is now on medium each week a medium publication brings you a hundred selection of the best voice technology articles written by a guest offers you can read about the wide variety of topics related to voice technology including design development machine learning athex opinion and mall to receive all the content for free just go to voicetechpodcast.com medium and follow the publication

that’s what I say I was going to ask so this is intensive voice applications are you mostly send them the the link to your development version of of The Voice app in the night they use it and I seem to do you train a video on Amazon with that something on the web cam that records them look at that face of the doing as well as it would be lonely

so I’ll be at the Tesseract I can take place directly upon the devices themselves so you can use basically any Alexa Google Assistant enabled device and we actually can get the the transcripts and the audio from just testing on your own device and so the transcripts are an indication active kind of flatworm heard and then how the platform responded and then you know the audio is just your straight up your what the what the device heard for dealing with multimodal situation building on the SDK that can capture all of that data as well and record all of that data during the testing sessions over tricks that you can bring her yeah basically Flex you know and what they saw so that you can you can actually your visualize that as well and then we have conducted tests where we’ve also Incorporated video recording

I’m so that you can go see what people are doing while they’re using the application and then also you know basically visual recordings of the multimodal situation what does it mean to reproduce interesting okay fantastic what about the ad that said they said depends on having a voice app at least partially developed I know nothing of quick voice app is is quite to do these days but you probably wouldn’t want to invest results and testing that so that it requires something that you know this is reasonably close to add to what what is it you want to release are they supposed but after you have rested respond to that. Sumsion Festival, so if you got some ideas that you’re playing around with we can actually go get even eat if your application isn’t even you know close to being ready but it has kind of the baby just the initial user experience or even though you’re the first kind of couple terms of interacting

we can test that and it’s actually I’ve strongly recommend the best you know kind of test engagements that we’ve had have been engagements where we’ve done like three or four rounds of user testing during the design and development process and is a mention we’ve even done a prototype testing and so those are scenarios where you know you might not even have anything developed or available on an application but we can put together like since your web-based experience where the testers can you hear your baby even just a number of different ideas or number different introduction or kind of the first few interactions and can respond to them say okay I know this one you know sounds more appealing than that one or I when I heard this my understanding of what this was doing was this you know so on and so forth

belonging company said it’s all about getting out for you back as soon as soon as possible early and often and I was going to ask you like you could really go lotad can even just give them you know why frames or the lawn pay for the SketchUp so something just take him through a flow play some more tea or trials that you’ve recorded in and get good feedback like that which can can really guide the entire direction to go projects and then the other these decisions that you were saying I’ve been less less impact less costly have on that the April project in the first round of testing it been you know very quick and dirty like that ok wonderful so what what are the main steps in for usability testing for voice reminder Genesis remote usability testing I’m so where we going to panel that set this out there in the in The Ether in multiple countries they can be used in the I’m thinking reporting that findings at what do I have to do as a as a Klein

someone with a knife I want to test in order to get my feedback

so what you need to do is just go to go to a website and create a customer account and then we will get back to you right away and we can discuss kind of exactly what kind of test do you want to set up and what sort what your what your target market is for your for your application and we can use to get a test setup your the actual test set up time the request the time that it takes for a customer to set up a test is less than 10 minutes usually if there if we want to have a conversation with them and adjusted just engaged and understand what they’re testing what does Target use is it going to be and also depend a help guide them in terms of your these are the sort of instructions that you might want to but you might want to provide an order to get the feedback and they can answer the questions that you’re interested in answering and then also can help with the Canada post test questions that we ask the end

it’s up to the two final questions sorry that the final questions are always the consequence abilities I was them have to find a sign up with what they can to ask but you provide the guidance provided example questions of white wine in the past and no Steve gold experience regarding these exactly so we have we have basically a standard set of questions that we ask for most of our tests they’re not going to require questions and they can certainly be modified or changed or not used by a by a customer but they are standard questions we ask for most of our test and then if there are no additional questions that the testers particular that the customer is particularly interested in asking then we can certainly corporate those welcome for the men on the test aside then right where we took him out for the panelists what classified or what what do you count as a as a great usability test what were the characteristics you look for in a great ability test a and m on the other side what what

would you be considered to be a rap artist so a great usability tester is is somebody that is a

the first off is Ace is somebody who’s enthusiastic New Uses voice applications on their own and second is somebody that will be your is observant while there while they’re using the application so somebody that says okay I’m paying attention here this is where I want this is what I’m trying to do here and then can provide actually be actionable feedback as compared to I liked it I didn’t like it and I liked it you know I like this particular aspect or this particular aspect was confusing to me and here’s why you know that I when I heard this I didn’t know if it meant ask her if it meant why so those the sort of you know those tend to be the the characteristics of a particularly good tester panelist granted there’s a lot of variability there we want to make sure that

glycerol so kind of did the feedback that they that they are providing itrax to sort of feedback you’re going to get in the real world and so far at Amazon even to the study on this be the predicted ratings coming from our tasks acumat very closely with the predicted to or with the actual ratings of the tests get once their lawns are running and then you can even be the case that it’s not necessary you don’t necessarily have to be

a perfect representation of a group yourself in order to provide feedback that is extremely relevant to that population growth what I mean by that Yeah Yeahs that you could be you know it might be the case that not everybody from a particular population is going to be as observant while they’re testing as you are but the feedback in the suggestions that you can provide are suggested it going to be really really useful to a particular Target population okay so you’re looking for the pilot or interested there were seven and then I’ll take you to it and but you don’t necessarily the exact type of a customer every time because of the you have to serve and you can have a sub sample of the profiles of personas interesting to voice I know you mentioned before that you need to know about misunderstandings I think that might have something different about you as much as both directions you can have the voice assistant Miss understand what that what they use us as

well yeah so that’s one what other other brothers

yeah absolutely so I mean one of the one of the more profound differences between voices Ian is compared to visual design is is that you on visual design data can be presented simultaneously and it’s persistent on the other hand Space is really up at a premium so you can resent kind of five or six different options to somebody at the same time and that’s not overwhelming but you cannot present like 50 or 60 different options to something same time down box or Nest it’s kind of drop-down box exactly and so something like a nested menu makes a ton of sense in a primarily visual interactive media like Weber mobile nested menus make

very little sense in voice you almost always want to keep as much as possible top-line do you want to make things Broad and flat no don’t require somebody to make three different request in order to get somewhere when they could just make that request directly so that’s you know something that we see frequently Inns in testing that we do and and one of the major go to hike near type of advice that we provide back after the year when we’re done with usability testing session is that your things like You’re incorporating nested menu very very frequently confusing and unnecessary for the end you want to sound like a system of yesteryear exactly exactly yeah and then and then the other one is you mentioned it is you’re dealing with

basically handling errors are handling misunderstandings race with you know and being able to do so for example if the user says something or request something that is not immediately understood by the application that could be that the user has requested something for with the application is just not prepared or it could be that the user’s actually requested something with the application can respond just maybe it was misunderstood by the platform or maybe the utterance is not one for which the platform is prepared and so it’s a bit difficult because in those scenarios you can have it you don’t want to say oh I can’t handle that when in fact you can but you also don’t want to keep prompting the user to repeat themselves repeat themselves and repeat themselves when fundamentally fundamentally what the users asking for the program can’t provide so striking

balance between those essentially you know prompting them maybe once if if you if the application doesn’t understand what they’re asking for but then finding a way to move on if it’s misunderstood the second time that can be very important was the one I was looking at today she was moving on like this so that’s an impetus to have a conversation move forward it’s like when you told him someone a party is you feel like it’s stay out since you’re repeating the same things over and over again that okay where we going with it and it didn’t really matter what you’re talking about as long as he is particularly important in skills the design for kids and so I know a lot of the time for skills the design for kids you want to make them interactive in conversational but you want also set things up so that the the kids input isn’t particularly important and what I mean by that is that

yo even if you know even if what they say you know might not make a lot of sense you can still move on and you can still handle that gracefully and you can still basically you know say all right but we’re still going to go and move on and the story and kind of get you where you need to be doing this with voice was open-ended then voice voice apps for kids is going to be the most open-ended absolutely see that’s the whole point of usability testing is to learn about your users what they want their motivations and then the hours and the problems with the encounter I have a sounds like some design practice is it really it sounds like it reveals the best practices for voice design in the process of forming these kind of tasks you must have learned a lot of that the things that what really while invoice apps and then some of the things you should avoid old cost America – are a few of those are there

oh yeah absolutely so I actually did a presentation with with pockets on the sent you this subject at last year’s Amazon reinvent conference has the Evangelist from Celexa run yes exactly he’s the he’s the Alexa van you’re one of these and he’s a he’s a member of the elected one of us got a whole thing of a whole crew but he’s he’s the one that he’s one of the you know Alexa evangelist for for Amazon and doesn’t make them you know the nested menus you know you almost always about it that you want to keep things Broad and flat the other thing is that

in standard human conversation you

if you always get the same response to something using exactly the same words

that get the bit uncanny you know if someone if you’re even if you’re saying something like hello every single day it’s somebody just says hello hello hello is compared to Hi how are you at cetera excetera it seems you know in human and robotic and so if you’re building something if you’re building an application that is going to the in which people can be interacting with it frequently even if it’s a simple transactional application it’s best to provide some level of variability and even customisation or personalization there you’re nothing you nothing that would make it seem like a different experience and experience that they’ve had many times but also you you should want to hear you should try to vary the experience a little bit at least in terms of the words and phrases that you use or else it will it will seem less comfortable there will be less comfortable for you users

attention I guess it’s exactly all right one thing that came to mind was that design challenge to balance the the need to be brought in and flattened is as you said so that you need to have these crazy nested menus by the same time not be a very linear experience which is kind of similar to what you’re saying about that the repetition you got the same thing over and over roll over when a game or not and experiences is Right predictable that was so can lead to to boil them or frustration I guess so. Said to make something sound novel and exciting and unpredictable but the same time I get very few few options every every step of the end of the process

yeah exactly in the way that you probably the the best ways of dealing with that or two

I was thinking maker try to make your application as contextual as you can and to personalize it to your individual user and so remember things about what they have done and kind if you can find a way of even you know kind of injecting that into the experience that will make it seem you’re different and unique to them but in a way that is not necessarily going to be misunderstood and so if you if they can feel like okay I’m getting the same experience but it’s been tailored to me you’re back in kind of that can kind of satisfy the best of both you can swim in the power yeah it sounds as I just sounds like you’re giving advice to a to someone who’s planning to socialize for the past time I was hunting it’s not exactly exactly what Bingo yeah I mean your voice application you sending it out there and you’re saying okay make friends with the world but it’s the first time that they really talk to people outside of their at your home

you got it you got to set it up for going out there when the pool

voice chops Tuesday is a Weekly Newsletter that helps you build better voice apps where they were looking for research chops death chops or design chops that something in there for everyone just go to voicetechpodcast.com newsletter I look forward to your Tuesdays

have you seen some great examples of of these techniques in practice satellite what are there any voice apps that really stand out that you had from from your customers were just out in the in the wide world and that you had to mention yeah absolutely the first off you know one that I think a lot of people that have Alexa that I’ve used that does a lot right is is jeopardy so this is one that you know it’s got a very intuitive than and straightforward user interface and it kind of is exactly the experience funny I think your Jeopardy’s been around for decades and did you know the game show Jeopardy when you say that you know what is or who is better Center and so

but the Jeopardy the game show has been around for decades and they have been trying to find a compelling home version of the game for about as long and have had limited success in doing so until voice platform is that right I mean you know they were Nintendo games that were based on Jeopardy and it was just like one of them yeah yeah I only one of them and it was just a nightmare of an experience you know you really wanted to have a situation you needed a situation where you could respond to these questions verbally and have it be understood by your you know by the application by the Yeah Yeah Yeahs exactly don’t want to say ABC and I remember the Super Nintendo game you actually had 2 with a controller take it out pick out your response letter by letter my God

it was it was terrible but you know with you with Alexa you can actually just hear the question and then

respond vocally with your answer and have it be you know how it be understood and Dean correct or incorrect missing and so that’s acting experience this kind of waiting a long time for Alexa and it’s really it’s really a perfect mess and so that one’s nice and that it’s intuitive it’s fun and it actually manages to set things up really nicely and that it’s an excellent example of a skill that you can monetize in that you can have people play it for long enough that they’re enjoying it they understand it and they want to play more but that they’re not bored with it and so you can basically have them you can offer an enjoyable experience for free but that you can also say if you’d like to keep playing if you’d like to play more and keep enjoying it you can also opt you know basically sign up for a subscription premium tier and get you more content and he can get more into it once I hope so

yeah that’s exactly exactly and so that’s your one example of a skill that I think is done it really well another one in this this sort of leads into your what I think is a is a potential

Aura for Brands out there one of the one of the one of the nice potential applications for voice platforms for Brands out there right now is that what you’re going to have asked if you’re going to build an application that is going to be used at all the time a very frequently it either has to quickly and easily provide a feature that they that makes things better for something that’s already routine example that is setting an alarm setting a timer getting the weather you know how those core function real from 10 x 50 get people to switch is why they can’t just be a little bit better have to be significantly better exactly exactly much much better than the current Trend experienced the current way of doing it and so yeah be being able to say in your Alexa what’s the weather and have it be immediately respond with your local weather is

the best experience you could possibly easy as it could possibly be looking out the window exactly so an example of something that is the people going to be using everyday because it makes something it’s a routine a lot more yo a lot easier another example is there is this

there’s this skill called Chompers that specifically design basically is it as a as a micro podcast or two minute-long podcast with an updated content for children with while they’re brushing their teeth the idea there is it something the kids can listen to and to make a game out of brushing their teeth and also last as long as they’re supposed to brush their teeth cuz it’s really really difficult to get like a three-year-old to brush his or her keys for 2 minutes but if you have something they can listen to him they can they can turn it into kind of a game then it can become a lot more enjoyable and so that’s what are the waves going to getting into one of those routine the thing is he doesn’t like Chompers this requires

be your kind of the continuous production of content and so that you know that can be great but it does require you know kind of continuous resource it can be a commitment exactly and so another way that these applications can be useful is that they can be incorporated into the marketing strategy or around a big release and so you might have a big release of a of a movie or TV show or the season TV show or album or something like that where you know it might not be something that people are going to be a lot of people going to be returning to two years after lunch but it can be a really cool aspect of kind of Vape The Fan Experience during laundry in the lead-up to Lawton and one great example that we saw that

my last year was for the for the TV series Westworld exactly as part of their kind of marketing strategy for the next season of Westworld they had this interactive Alexa game which is really really had a lot of depth to it and it was it was amazed that you had to go through and you’re phenomenal recorded audio from the from the characters and the and the actors in the series and that was a really cool experience that yo it’s it’s something that

you’re not going to be doing everyday but it’s something but it’s kind of a kind of a game that you can you can try out and that you can play that will get you excited about and be part of sort of the experience of you know the fans of the TV show getting psyched about the next season. Clearly Now is it a while ago about the way that they built it I’m feel as other than that the quality of the older that’s the last thing I remember that I was thinking it was because it’s almost like watching the show right but was already sent it to the user that stood out for the west-world skill and so one of the things that was cool about it was that you know again it was this it was this it was this story and one of the ways that they said that they designed it is that

the word a lot of the word a lot of things that would stop you from moving forward you know and so you know there were there were a lot of times where you could where you would be interacting with them or where you could say something but they were very few things that you could say that wouldn’t necessarily move the story forward and so we actually help to help them expand out the response set for a lot of input that you know when somebody would say something and Destroy would move forward but it really didn’t what the other responses to I didn’t make a lot of sense based on what they say we help them kind of Explorer you know input they might not have been expecting and expand out this set of you so they’re set of responses appropriately but they did follow that recommendation that I mentioned earlier and that they really didn’t want to want to stop people from moving on they always wanted to have a response that kind of kept the story going and Captain America

how do you manage that Branch out into a million different possibilities are you the one story right but then handle anything that

yeah so so the way you do that is that so first a few you in this case there are few different possible sores a few ways that you could go okay the way yet but the way that you can you can do that is that you can try to or you can tailor your response to what somebody said but not make that response fundamentally alter the direction you’re going on in the game the free example you come up to a bartender in the bartender says you know what would you like to drink what is a whole bunch of different things you can say and consequently a whole bunch of different responses that they could have but fundamentally these different responses are not going to dramatically change been there before by customisation exactly how I can see where it goes to make it feel like your having this person lies that unique

variant exactly all right so it sounds like a huge amount by doing this this usability testing and I just wanted to know whether any of the stuff is anyone put it down on paper are there any good books or videos or any other resources that we can get condensed form of this advice so that you got Head Start if we’re thinking of doing to me because he testing yeah absolutely so so I would I would recommend a good start there is that the Amazon off actually has some some documents for basically a design and usability testing on that you can get from their kind of Alexa organize their Alexa website developer it about the platform or whatever it’s called exactly and then I actually have a blog post from that talk when I mentioned with pockets

goes through a lot of these principles and talks about you know principles of design and also how you can set up usability test the book designing voice user interface from Kathy Pearl which is a great book on essential just voice user interface design generally has a really good chapter around usability testing or voice that was that the kind of one of The Inspirations for us when we started. So yeah those I think would be some good introductory resources and then you want to go in more depth email me at Dylan. Wicked. Saturday and I’d love to have a conversation with you but I don’t know. I’ll definitely put the links to tour de Vaca see what post are in the book comes up again and again in conversation and everybody had to have to read that book it seems like an all-right I do today to

super pads that had to look at the head of the car for the voices you know developing such a such a clap what would be your advice to to to to get involved basically absolutely so one of the things that I would say is in terms of what what not to do is I don’t take an experience you have on Webber Webber mobile and try to just for that directly to voice and trusting I lost I’ve had the phrase just taking my contact United States that if you got to know about that. So you’re saying it’s an easy way to these way to get something out the door but it’s usually not a great way to get something out the door that people actually going to use and

for example one thing that will see you as well see them say okay what we’ve got this you know we’ve got an API that will you know that we can use to link to a huge amount of content and so we’re just going to kind of put a voice skin on this that’s going to interface with the API and people are going to be able to get in there and explore all that with her voice the thing is is that for almost all of those experiences there’s very very very little reason why somebody would want to do that in a voice on text as compared to on their phone or on their on a website and so what you’re talking about with like an ex for example those sort of experiences definitely do not end of a 10x better in kind of an audio-only environment one of the reasons for this is that there are

so there are certain types of kind of interface responses that

for which audio is not the best medium in so what I mean by that is that

voice can be kind of the best and lowest friction way of a user inputting information but if the response that they get back from that is like a list of 20 things or something like that now it’s almost invariably a bad experience if the response is number one is number two is that at and so if you have an experience like that I would say if you want to make that voice make you really focus on trying to utilize if you can a multi-modal experience so I think there’s a lot of there’s a lot of potential there and I think we’ve seen

year-over-year set 2017-2018 there was almost a 600% growth in the end of the size of the The multimodal Voice market so Echo Show those Google home hubs that kind of scenario is where you can combine yo an optimal user input with a with a much more intuitive output you can say yo show me show me the movies that are close like that are that are playing close to me and then the response can be you know here are some movies that are close to you but it doesn’t actually have to read them all okay so you guys how did the idea of putting a mic on it on a mobile app not that kind of makes sense cuz I send me Janine multimodal I’m just not doing that image phone it’s not your voice and they were quite a bit more. I love it more of a redesign if you were just coming from a website on mobile app to a pure voice experience that not make sense

Bingo yeah and so yeah so the device to put a mic on it if you’re talkin about basically just taking your taking your mobile application and offering voice input to it that’s fantastic device but that’s fantastic advice but that’s not really building a conversation with experience right that that is that is augmenting and providing new features to your mobile application which is a great idea but if you say Okay I want to build a Skiller in action on Alexa I want to build you know a conversational application here then you’re going to want to put in a little bit more probably going to want to to try to find an experience that is a better fit for that particular type of environment and that particular type of interaction

okay let’s Ms listen up this podcast wouldn’t be possible without generous contributions from listen to such as yourself be like see the options my patreon page is that voicetechpodcast.com do night Patron tears are also available for businesses so if you have a voice tech product that you like to promote on the podcast newsletter and blog then you can become a voice Tech Champion head over to voicetechpodcast.com

if I was a Brandon I was considering getting into voice then I asked this question a lot at night how do I sell it to my boss but not what one might sell it to demo it no produce some kind of a proof-of-concept in any organization say love it is another way would be to just point two examples of of companies who are generating value from at her she could getting out of return on investment from their investment in voice would you would you agree and do you have some examples of that happening out in the real world when we mention West Weldon some of these media companies but is it isn’t the case now what boys really does deliver on on the investment or is it more speculative is it more about presenting an image of being at The Cutting Edge and and difficult time in the water so that you can build up a Knowledge and Skills in your team so that you can be ready for when it does eventually move the needle

yeah so so right now things are still very early The Voice ecosystem and I think that a lot of brands are still trying to figure out if still in the process of figuring out what is the what is the best experience here and what is the best experience here for my brand and so I think your music and media companies are certainly the ones that I think of bound that be the most the most natural transition they’re the same things that are coming up that I think you’re going to see the next year is that

voice assistants are going to move

beyond the smart speaker and into an enormous number of additional devices in and run and so one scenario where you know let’s say let’s say ordering food

ordering pre ordering food from a restaurant in Panama voice only contacts might not make a lot of sense in your living room if you have your phone immediately available but can make a tremendous amount since your car and so if you have basically your voice assistant immediately available in your car and you’re driving somewhere then being able to make an order that way be extremely you know you can can can bet the kind of contacts you really unlock that that type of features additional Revenue channels depending on the context during pizzas on the way home from work and your Pizza Company than it really, really come to bingo or if you if you are a McDonald’s or Burger King or something like that and you know somebody can send you can pre-order for your drive-thru

run when they’re you know five blocks away or something you just say you know have my have my usual ready to go when I show up that can be a that can be a 10x sort of experience on voice you know in in the proper context in the right context there and then there’s also

so what I mentioned earlier about your contacts in except for voice you know that the Chompers example which is basically this this experience for kids to help them do to help them find the routine of brushing their teeth a lot more enjoyable as well inserting yourself into a into a routine like that can be extremely effective channel for brands that have you know they’re basically selling Goods that are part of that so if you are Colgate if you are Crest or something like that you want to occupy that moment exactly you want to occupy that moment and you want to have me on that moment be enjoyable to your to your customers on its own but they can also be the be the possibility of hay and if you happen to be out of

toothpaste right now just say this and you can reorder toothpaste in like 20 so I think that the pizza example which is something that people already do and but it’s difficult to order pizza while driving and you probably shouldn’t try it but you you could do it with invoice dining for instance of us has taken all the time you need the voice interface us. Taking an existing problems let me know that hurts it’s a pain point and making it ten times better does the brushing teeth example that’s that’s a whole new fresh ground in a breaking new ground and I was your oyster right now I guess those are the opportunities that the company’s ready to dream about discovering and then the other is this was just announced actually buy Spotify like a couple days ago is the consumption of audio content has been growing rapidly of last year’s

and so your people are listening to things when they’re at home on the move all of these things but you’re almost always listening while you’re doing something else and so advertisements upon these platforms can be effective you have to listen to them and then you have to basically take action on them later on you know you have to remember them after you sit in this hell yeah I thought you don’t really have the ability to you to click on an audio ad

you will very soon and so you’ll be 8 so if you had an experience where let’s see you’re listening to a podcast and there’s an advertisement is being read by the podcast host for example and if he said you didn’t know if your she says and to find out more just say Alexa tell me more and if you’re listening to it you’re not interested you just keep listening if you say Alexa tell me more it can be something as easy as you then get an email sent to you with more information about that product and then you go right on West us so it’s minimally invasive but it lets you immediately take acting upon that I owe ya I’d like to know more about that and then it’s waiting for you when you get to ride seems I guess listening to what you’re listening to is playing it through this must be cool whatever voice mobile device on exactly exact so I can respond to go exactly and so the idea is that essentially be those are going to be you know

the dream you Amazon Google Samsung they’re not devoting all of the resources they are into their voice assistance because they really really really want to dominate the clock radio Market you know exactly for them is that they imagine this is being the operating system of the internet of things they they imagine that you were going to be able to talk to not just your speaker but your car your phone your television your refrigerator all of these are going to have you know that’s how you’d that’s voice is going to be the primary medium if interacting with all these with all of these objects and so when you’re listening to audio pretty much anywhere on any device is going to be kind of this assistant that is there that you can

that you can call upon and will understand okay I know what you’re listening to right now I know who you are you can take action on this really quickly and among other things allow you to cut a double click on on audio advertisement when I listen to that I got again it’s one of those things that excites me and it says my imagination going to that technology and then it will say terrifies me because I think while I know what these companies are going to use it for a soul about the contacts is that the mall contact these devices have on you no more personalized the most powerful that I ain’t never said they can provide to I’m just done that I need to be with Chris Mitchell from my audio another tech and that company does the event detection so that that all about providing other apps and the context of the oldie environment and you know where it’s being used whether it’s in the home or out in the move and at the moment I admit that they training these classifieds for specific events like baby crying and Powerline and stuff

soon it’s going to be like the specific Russell of a of a of a Chris Paquette everything isn’t even the salmon steak and then you got this load voice notification coming in and pushing you in one direction or another proper Balancing Act for that is something that the platforms are very very concerned with getting right and you know they want for these to be fundamentally something that is it is valuable and useful and you know important to the end user and so example that I gave being able to quickly take action upon something that you hear and maybe get a little bit more and maybe you’ll get a little bit more information about in your inbox or something like that yo

get you don’t want to have a situation where it seems intrusive and you don’t want to have a situation where that will take you out of the context of what you’re listening to rent but if you can have it be completely optional on the user side and also relatively light touch but can still let them say oh yeah I’d like to you know I’d like to follow up on that that’s the sort of scenario that I think can be you know you’re the user won’t mind and will and my inner and might really enjoy but they can also really really be beneficial to in a market is Radford possibly talking to Dad is really not the toilet days is good advice does the major developments going to be on invoice of the next to be able to do you have any big predictions of things that you’re hoping to say yes so

yo I think you’re going to see is that just opens up so many new possibilities so I think you’re going to see your Echo Show the Google home hub the ability to that I think is going to be there going to be a lot more application development there and there and that’s going to that I think is going to lock a lot of different about lot of new possibilities I think on your smart speakers the big developments are going to come over around what I just talked about you and maybe that you the ability to provide kind of optionally the interactive audio experience is augment the things that people already doing on them Yo listen to podcast listening to listen to NPR BBC listening to music that sort of stuff and then the moving and smart Voice Assistant off of smart speakers and into other areas of your life you know particularly

take the most exciting when is the car because that’s kind of an inherently Audio Only environment for the type of interactions you can have and so you know things like being able to do you order ahead at a restaurant or something like that that might not be as valuable in your living room will become extremely valuable in your things related to the car and I’m getting Diagnostics on the car at South allow the movement on invoice going to come through Smart TVs and voice like the west-world example really augmenting the TV watching experience because it hands-free have to do anything except exactly and even even you know I mean that was that was your bandersnatch little while ago I got to be able to have actually interactive television shows

was that people love. Yeah I mean I wasn’t too honest I wasn’t as impressed I preferred the other ones and I think you’re going to miss its it’s a balancing act in that one’s that sort of require you to be interactive you know those are those credentialing hybrid between a television show in a video game right now but ones that you can find ways of making optional interactive you know some of the way of saying you if you choose to interact with this you can if you’d rather just sit back and watch you can to figure out how to make both of those experiences great has to be hard but it’s as normal as an option you know it’s that it’s on you all times. That’s why I like all right so I think we’re coming to the end of time it’s been one for conversation what will you and post-op you working on fixing it

she’s on her way over the next six months or so while your priorities

well so we’ve got a lot of this a lot of them is going on as I talked about and what what’s going on for the next 6 or 12 month full slab is going to be there offering base will user testing user research for that we are also looking to expand out or testing up offering to also working on a recertification testing and functional testing for skills and actions so you will see that in the in the near future, and functional test checks you need to do in order to basically go through certification and get your get your Steeler acting into production and so being able to sort of streamline that processing a lot more help with that is something that we will be offering very soon and then but yeah it’s mostly just it’s mostly being there

for all kinds of usability and user research needs that the developing voice ecosystem is is going to require an excellent day I look forward to seeing the same way from from post lab it sounds like you’re at you’re involved in that whole area is if you have voice to die in that he’s going to keep both of y’all ready and then yeah it’s unhealthy relationships with both amazonite Google so yeah congratulations and best of luck why can people find you that I can people find more about UI Online why can why can we contact postal absinthe and find out what some of that the case studies in the what you’ve done in the past that be great. Yeah absolutely so best place to start is just a website www. There that’s blog. Pulse Labs. AI that has a lot of case studies and also a lot of blog posts about basically voice user interface design you can follow us on Twitter post Labs AI

or you can follow me on Twitter I’m a pulse Labs underscore CPO or you can even just send me an email the Dylan. Zwick u i l a n z w i c k at Pulse Labs. I didn’t thank you very much for coming on the show has been fantastic conversation

thank you so much Carla it’s been a pleasure

okay so you just heard from Dylan zwick the chief product officer at Pulse Labs that’s open today very much hope you enjoyed listening as always you can find on the show notes with links to the results is mentioned in this episode. Voicetechpodcast.com future this episode there’s many ways to support me in the show how you can tell one friend will call you about this episode I you can always leave a quick review on iTunes that voicetechpodcast.com iTunes we can write to the blog I have voicetechpodcast.com publish finally you can become a sponsor voicetechpodcast.com. I know that your contribution really does make me Shy possible I’ll be back soon with another upset but until then let me know how to become Robinson thank you for listening to The Voice Tech podcast