Experimentation news round-up #6: Domino’s Innovation Garage, Learning to be okay with failure, as well as experimentation job postings from Agoda, Hulu and more!

Welcome back to another edition of the Widerfunnel Experimentation News Roundup! These are the headlines that caught our attention over the last two weeks, along with upcoming events and job postings you may be interested in. Did we miss something major? Let us know in the comments!

Experimentation News: September 2019

At Booking.com, Innovation Means Constant Failure – Stefan Thomke/HBR.com

“It’s not anarchy, but it’s democratized. Anybody can launch an experiment without permission from management, but anybody in the company can also kill an experiment that somebody else launched; they call it pushing the nuclear button. It’s an environment where we trust people to do the right thing, to test out things a lot. At the same time, it’s also an environment that has a lot of checks and balances, where other people can step in. It’s an environment where transparency is key.”

Creating Digital Offerings Customers Will Buy – Jeanne W. Ross, Cynthia M. Beath, & Martin Mocker/MIT Sloan Management Review

“In large organizations, learning does not naturally flow across the enterprise. People with an idea want it to succeed and are often tempted to tweak an unsuccessful idea rather than abandon it. But learning from experiments depends on recognizing what isn’t working and shifting resources to something that might be more successful. And as companies discover what customers do and don’t want, they need to design processes and teams to ensure shared customer insights.”

Why ‘one app to rule them all’ is not the future of digital health – Chris Hogg/Tech Crunch

“Enrolling users is hard. Keeping them engaged is hard. Improving specific clinical outcomes — and proving it continuously — is especially hard. Making a great product requires an obsessive focus on a specific user and problem space, as well as relentless experimentation and iteration. When you don’t have that singular focus, the needs of the patient are deprioritized compared to the needs of the organization, and the user suffers.

To make this approach worthwhile, you’d have to believe that the convenience of one app would make up for a worse user experience by driving higher enrollment or retention rates. You’d have to believe that user experience simply doesn’t matter as much as the convenience of an all-in-one platform.”

Digital Transformation Success Depends on Agile Approach to Change – Peter Bendor-Samuel/Forbes

One of the keys to navigating through the unknowns of the deep change that most digital transformation demands is the willingness to experiment to learn and test. Failures – doing things and finding they don’t work or don’t work as well as we had anticipated – is part of how we move into the unknown, and much of the details of a transformation journey are unknown and evolve on the journey. Failure, or at least some level of dissatisfaction with results, is often the norm.

Delivering results through the magic of experimentation – Jose Hernandez/CIO.com

“I had no idea then, but I know now, that offering an organization a way to experiment on the quick and cheap without taking too much time from those that have other day jobs, is immensely valuable. In fact, our business partners invest an average of an hour a week throughout their engagement with our lab, The Tech Nest. This can lead to unlocking all sorts of value from showing the potential of an idea to validating that a particular technology is just not ready for use by your organization.”

Domino’s opening collaborative Innovation Garage in Ann Arbor – Meredith Bruckner/Clickondetroit.com

“Domino’s Innovation Garage is focused on accelerating Domino’s spirit of creativity and collaboration,” Domino’s Chief Technology Officer Kelly Garcia said in a statement. “As a pizza company with a focus on technology, we want our team members to have a workspace that encourages experimentation and fosters an environment to take risks, test innovations and fail-fast so they can bring their best cutting-edge ideas to life for the brand’s customers and stores.”

These Four Traits Can Help You Become a Better Digital Transformation Leader – Sandra Petrova/Adeva Blog

Building an experimentation culture is vital to surviving. Every chief digital leader must:

• Embrace failure. Otherwise, it would be difficult to create a culture of innovation and experimentation.
• Strive for high returns but launch incremental changes. Step-by-step changes will help you achieve your targets and avoid unrest within the organization.
• Analyze your experiments. Check whether your experiments are working and what can you do to make them more effective. The analysis could help you plan your future experiments.
• Document everything and share the results. Keep track of all successes and failures and share them with your team. Your team should learn from their failures, share their insights, and receive feedback. This is a productive way of encouraging a culture of innovation and broader participation.

Using A/B testing to improve HMRC services – Jennifer Scott/Gov.uk

“We could do an A/B test and find the new design performs worse than the control. This is not a failed experiment. We have learnt what does not work for our users and can use this insight to inform our next A/B test.

We can also learn by trialling new features with a small number of users first. Recently we used our A/B testing tool to send only a small percentage of users to a private beta release of the employment expenses tax relief service. The team could learn and iterate with small user volumes, reducing risks and allowing them to fail fast and learn quickly.”

How today’s news publishers can use data, best practices, and test-and-learn tactics to build better pay-meters – Harvard Kennedy School, Shorenstein Center On Media, Politics and Public Policy

“Our research suggests publishers should invest in capabilities to engage in constant testing and experimentation in digital — to build engagement among digital audiences and ultimately convert engaged readers into paying subscribers. For commercial and for-profit models in particular, publishers should become smart across a range of new strategies, including how to configure meters and rules to calibrate a mix of free and paid access.”

Postlight CEO Paul Ford on Writing, Experimenting and Running A Company – Podcast/Mission.org

Experimenting and being creative is critical to what Postlight does. As such, Paul said it was important to create a culture where you could pitch anything. But after pitching, you needed proof of concept within two weeks. That, he says, lessens the cost of failure. He explains that product ideas tend to come from leadership and the product team, and then the engineers will develop solutions and interfaces.  

This culture has allowed the creators at Postlight to build some really fun tools, APIs, and platforms. There are no bad ideas, and when you build something cool or silly that catches on, it works as a product, but it can also double as a marketing tool and a culture builder.

Solving the digital advertising causality problem – Brad Howarth/CMO.com.au

“There is huge frustration in the entire world of advertising and marketing that they don’t know what they are getting in return,” Nair says. “Digital advertising provides a vastly superior measurement and tracking and therefore that has been its source of success. But while measurability has improved quite dramatically compared to traditional media, there is still a huge problem in the confluence between correlation and causality.”

“It is very hard to assess causality unless it is an experiment, and the experimentation has to be done by the platform,” Nair says. “If I put money into one platform as opposed to some other platform, what do I get? Can it show me that I am achieving my goals? So along with my team I built these external facing self-serve experimentation products, which are now available on JD.”

Dissecting AI hype and FOMO – Finextra.com

Abhijit Akerkar, head of AI business integration at Lloyds Banking Group, reveals that a bank’s journey into artificial intelligence starts with experimentation. For Lloyds, it was around 24 months ago. Experimentation provides a low-cost way to test what works and what doesn’t. This experience shapes the portfolio of use cases and hence, the trajectory of value creation.

Innovations need a mindset that is okay with failures – TheHinduBusinessLine.com

Stating that innovation will not happen in R&D labs, Mariwala said innovations need a mindset that is okay to failures since innovation involves various accreditations, changes and developments before a product can be developed into a viable prototype.

“But if an organisation punishes a person for a failure arising out of an intelligent experimentation, then innovation cannot happen,” he added.

Upcoming experimentation events and webinars

Opticon 2019 North America – San Francisco – Sept 11-13

One of the largest experimentation events in North America. If you are in the experimentation space in United States or Canada, you will want to attend Opticon 2019 in San Francisco this fall.

Opticon Europe 2019: London – Oct 1st

Optimizely’s European conference, hosted out of London and geared towards Europe-based companies.

Who’s hiring?

• Thomson Reuters  – Sr. Digital Testing and Optimization Manager (Carrollton, TX)
• Agoda – Head of Experimentation (Seattle,WA) 
• Avis Budget – Director, Insights & Analytics (New Jersey)
• Etsy – Senior Product Manager, Experimentation Platform (Brooklyn, NY)
• Credit Karma Senior Growth Marketing Manager (San Francisco, CA)
• Dow Jones – Product Manager, Acquisition Experimentation (New York, NY)
• Twitter – Senior Product Manager – Product Insights & Experimentation (San Francisco, CA)
• Facebook- Content Strategy Manager, New Product Experimentation (Menlo Park, CA)
• Hulu – Senior Software Developer – Experimentation (Seattle, WA)
• Hulu – Experimentation Manager  (Santa Monica, CA)
• Williams-Sonoma – Manager, Product Analytics, Experimentation (Brooklyn, NY)
• Lyft – Research Scientist, Experimentation (San Francisco, CA)
• KickStarter – Experimentation Manager, Insights (Brooklyn, NY)
• Change Healthcare – Product Manager, Experimentation (San Mateo, CA)
• GoDaddy – Sr Software Development Manager-Experimentation & Personalization (Kirkland, WA)

Do you know of an upcoming event or current job posting we should include in this round-up? Let us know in the comments section below, and stay tuned for our next experimentation news round-up in two weeks!

Author

Daniel Molina

Sales Development Manager

Enjoy this post? Share with your friends and colleagues

Share on Facebook