Testing product concepts and assumptions is critically important to the success of tech startups who bring new technologies to market. To do it properly a sophisticated approach is often required. However, it is a common misconception that you can’t do product testing at a conference, because it is just too complicated.
Testing at conferences, takes team prep.
It often becomes complicated as a result of lack of team preparation or misunderstanding of Product Testing practices. This inevitably means that founders are often concerned about the value they can get from attending conferences as a startup. Is it worth the price of the ticket plus related expenses? Will our ROI for the event be negative?
In this article, we reflect on our experiences of testing our product at conferences to address some of the common misconceptions, so that our startup peers can get as much value from attending conferences as possible.
Please note that ‘Product Testing’ forms part of the overall Customer Development process, popularized by Steve Blank in his book "The Four Steps to the Epiphany."
Myth One:
"You can only test finished products."
A common mistake startups make is that they are afraid to show their alpha / beta version to the public waiting too long until the product is perfect. But the truth is that perfection isn’t a realistic target, just like chasing the crock of gold at the end of the rainbow, you’ll never quite attain it.
As Reid Hoffman says: "If you are not embarrassed by the first version of your product, you've launched too late."
Typical mistakes.
A typical mistake is that when startups spend too much time on development, they don’t have time to listen to customers to understand their real needs and perceptions. The result is they often need to redevelop large parts of their product, because they aren’t taking quite the right approach to the customer problem.
It is important to keep in mind that most attendees of tech events have early adopter mindsets. They often don't get hung up product flaws and bugs and are instead more interested in the product concept. So no need to be embarrassed if your prototype isn’t perfectly polished by conference day, do your best though!
In most cases you don’t need to test the whole product. In fact it is better to focus in on testing only the features you need to, to inform next steps along your Product Development journey.
When YouTeam attended Startup Grind Europe we were testing only one major product update: our new search results. So we focused all our efforts on polishing the search results page.
The booking process.
The booking process of particular software engineer, for example, didn’t provide a seamless user experience, but rather than addressing issues with the booking process, we placed all our efforts on preparing a new version of the search results page. Our focus for this experiment was to gain insight into what the users were expecting to be shown in our search results in response to their query / search request.
Before you can sell your product effectively, there is a lot of learning and discovery that needs to happen. Conferences can be an excellent source of feedback, provided that you have hypothesized who your target market is and they are present in significant numbers at the event.
Product tests.
There are many types of Product tests you can do before even creating a semi-functional prototype. Here is a list of some of the prototypes you can take to conferences to gather feedback from your target audience:
Problem-Solution Presentations - slideshow listing the problems, their implications and screenshots of the proposed solutions.
Video of the Solution - some products are inherently hard to explain without video, i.e Dropbox & Slack, videos are often quicker and easy to create than developing products.
Product Marketing Collateral to test the comprehensibility of product descriptions or stickiness of Unique Value Propositions etc. (e.g. product fliers, presentation deck, coupons, vouchers).
Low Resolution Wireframes - can be done in grey scale, created within Invision.
Facade Prototype - high-quality, full colour representation of only the workflow(s) under investigation created within Invision or TestFlight.
UI Prototype of the Full Experience - Full color, high definition prototypes of complete workflows through the product including onboarding / sign up for users to explore, created within Invision or TestFlight.
Website Landing Pages - to test product comprehensibility, Unique Value Propositions and decision making related to onboarding / signing up processes.
Fake Door/404 Page - to test user's interest-level in a particular feature, you can add buttons to your product which lead to screens, you haven't even designed yet!
Semi-Functional Prototype - to verify user interactions with certain features which have now been fully developed, the learning process never stops.
Wizard of Oz Prototype - An application which looks like it is fully functional, however, in actual fact, many features are facilitated manually by someone behind the scenes.
Mechanical Turks - Most of the application is fully functional, however some of the processes that appear to be automated are actually done manually in the back office.
Clickable UI Prototype.
The following is an example of a clickable UI Prototype, which depicts of the selection and shortlisting process of engineers which can be hired through youteam.co.uk. It is designed to test what kinds of information are most important in the decision making process; rating, price, availability, tech stack etc.
It is often difficult to anticipate when a startup product company will be able to ship enough value to its target market, so in the meantime it makes sense to focus on testing your most fundamental product & market assumptions. This will assist you to steer product development and also to ensure those pesky assumptions don’t come back to bite you further down the track.
Myth Two:
"Everyone will flock to our stand and be super interested in our product concept."
People are naturally curious about new ideas, however at startup conferences visitors can be overwhelmed by the sheer numbers of stands and everyone wanting to showcase their own ideas/products.
It is also difficult to read an attendee’s mind, so it is hard to know whether visitors to your stand are just browsing or there is the opportunity to connect with them in a meaningful way. People at conferences are typically in a good mood and friendly, which adds the confusion because you don’t know whether they are just being polite or if this an idea that genuinely resonates with them.
If your idea genuinely resonates with an attendee, the next question to ask is which group does this individual fit into:
Leads in our Target Market, who would be interested in:
Helping us testing our Product at the conference,
Becoming a future customer
Potential Investors - Strategic Angel Investors or VCs, who have the right backgrounds/experience
Strategic corporates/partners - who can help us bring the product to market quicker or more easily
Other connectors in the Target Market (Sales Agents, Distributors and Re-Sellers)
Our experience is that providing entertainment for entertainment’s sake does not translate into meaningful engagement. However, don’t let that limit your creativity!
Our horribly designed shirts.
For one of the first conferences we attended, we decided to wear some horribly designed shirts showcasing one of the most important parts of our product. Our product allows for searching a database for the availability and technical profiles of full-time employed software engineers working for Eastern European development companies.
So we wore shirts with our faces and showing our experience & tech stack, as if we were software developers who could be hired based on our credentials.
For the record, none of the team attending the conference were in actual fact developers ;).
The response was fascinating, people were intrigued about why we were doing this & many even took photos of us and posted them on Twitter.
But more importantly, we were recognizable among a sea of attendees and the shirts were interesting/engaging, they provided a reason to strike up conversation organically.
Myth Three:
"Everyone attending conferences is too busy to test our product and give us feedback."
It might come as a surprise but it is often possible to convince the ‘right people’ to come and test your product and be interviewed at conferences, in sessions lasting up to 30mins.
During one of the first conferences we attended we were able to interview more than 20 target customers! Each interview lasted over 20 mins and we were able to record the feedback as audio & video files as well as all of their interactions with our web app! Holy Moly!
So what are the secrets?
Planning. You need to think through the Target Market recruitment process in detail.
Rewards. Appropriate rewards for Target Market participation!
Here is an example of the vouchers we were offering Product Test participants at a conference, 40 hour of free development services!
Here are some other examples of ways to incentivize Product Test participants, which are suggested by getworm.com:
Recognition (profile badges, special mentions on your website/newsletters, moderator rights);
Free upgrades to premium versions of your product
Discounts on your product or services, or free at best;
Merchandise (stickers, t-shirts, coffee mugs)
Skip the waiting list, etc.
Myth Four:
"No need for preparation, I just need to get my product in front of the right people and they will give me the feedback I need!"
Your product and market assumptions.
Startups should always be clear about which product and market assumptions they are testing before attending conferences. The assumptions you will be able to test at conferences typically fall into one of the following categories:
Positioning
Customer/User Persona
Buyer Persona
Target Market
Product Design/UX
Onboarding Process
Execution/ Go-to-Market Strategy
Stakeholder Management
Indicatively, this is the process to prepare for Product Testing at Conferences (at high level):
Agree with your co-founders upon the Product’s:
Target Market(s)
Use Case(s)
Problem-Solution Pairing(s)
Unique Value Proposition(s)
Overall Product Positioning
List all the assumptions you need to test and decide on 1-3 most important assumptions. There is so much you can test but focus is everything.
Create a plan of what and how you are going to test (which product features, elements, how (asking questions, observing the reaction etc.) and where to store the feedback).
Develop the Recruitment Strategy - to maximise outcomes you should start the recruitment 2-3 weeks in advance.
Develop prototypes to allow for carrying out of the Product Tests (Refer to Point 1 for further details)
Create all the supporting documentation to carry out the Product Tests:
Brief information on product positioning
Interviewing Scripts
Pre & Post Interview Surveys
Source all the technology you will need to run the Product Tests. We use Loom for video recording, Typeform for surveys, Invision for prototypes.
From our experience, all of the above takes quite a while to prepare with a lot of sync meetings, but it is definitely worth the effort because Product Testing (Customer Development) needs to run in parallel with Product Development.
Myth Five:
"Product Testing participants are easily converted into leads."
It is often true that "if you ask for money you often get advice, and if you ask for advice you often get money." Our experience is that Product Testing participants rarely convert into leads. This is because the chances that they need your product at the precise moment you introduce them to it, is relatively low. But that is okay, because the primarily objective of Testing Products is to get feedback. Also your product might not be ready to be sold.
With this in mind though, you are still demonstrating your product to your target market and they might gather all the information they need to make a buying decision, which might result in sales. However, it is important not to overestimate the number of sales and website visits which will occur as a result of interacting with the target market at conference.
Myth Six:
"The preparation process for all conferences is more or less the same."
Unfortunately, this is another misconception, because different conferences attract different demographics of attendees.
This may affect the list of assumptions you will be able to test. It makes sense to check who is in attendance to ensure that they match your current needs with respect to Product Testing.
Usually global big conferences provide access to their attendees list in advance (so you can search them and filter according to the type of industry, job title, country etc.). Also, many conferences provide the attendees with useful apps to connect. If you are applying as a startup you might also try and get in contact with organizers to get more info on the attendees list if it's not available online.
Bring it all together.
As a startup you might have different strategic purposes for attending conferences. Many of the outcomes you desire will only be possible if you have a great team and are well prepared.
If you decide to conduct Product Testing, make sure you have at least two full-time team members assigned to carrying out the process.
As a final note.
A lot can be achieved with respect to Product Testing, but don’t set yourself up for failure by setting unrealistic expectations!
Best of luck with your product testing at conferences! Feel free to reach out if you have any questions.
About YouTeam.co.uk
YouTeam is a curated b2b platform that matches businesses with vetted software agencies and engineers around the world.