This week marked our annual Product Barcamp at XING. While around 40 Product Managers gathered in the New Work Harbour, another 40 or so joined online. As the days get shorter and the cozy winter season is about to start, I sat down to read a story to my fellow peers. It’s a story about obstacles, failure, and even war from past experimentation endeavors of mine.

📖 Story Time!

The story kick-started a valuable discussion about how we Product Managers approach experimentation and in particular AB-testing at XING. It concluded into a new working group that will try to progress on this topic on a regular basis.

Maybe it will spark new thoughts, questions, or impulses for you too? Here it is.

Three Experimentation Pitfalls Experimentation

Pretty much exactly 5 years ago, in the fall of 2016, I worked in the UK on a game called PokerStars. With millions of daily active, paying users, it’s an enormous business.

I was in charge of growing new users, making sure they get to create an account, fund it, understand how to play the game and find a game that suits them so they get the most out of their time and money and come back again in the future.

After doing some research and sparring ideas with stakeholders, one opportunity stood out as the one to pursue further. The idea was to insert a visually engaging screen that features a welcome offer for new users.

Everyone was on board. It was a no-brainer. No questions. But those were just opinions, right. I needed scientific evidence!

So, I got to sit down and prepared the JIRA tickets needed in order to get an AB Test started. Old, obviously shitty screen versus shiny, obviously much better screen. It went into the magic of engineering and as we approached a state of readiness, I got to spend more and more time with my analysts. So far, so good?

Hmm no. From there, it went downhill. We had multiple hypotheses and couldn’t agree on the most important one. With that, we weren’t on the same page regarding which KPI should be the deciding factor. We couldn’t even agree on whether this should really be an AB test. Why? Because we didn’t play by the same rules.

The Analysts goal was to simply answer a question in a clean and accurate way. He wanted to test a hypothesis. My goal was to prove that my grand idea is as grand as I think it was. And I decided to make it an AB test because people know that the results are non-discussable. They are proven by science. In a way, I was asking for any positive number that I can use in a way to make this look like a success. I was asking for absolution and didn’t really care much about the data. What should I have done instead?

At the beginning of every experiment, make sure that there is a clear hypothesis to answer. Be honest with yourself that it’s not about producing some evidence to prove your point. There are easier ways to get that! Evaluate together with your analysts and user insights colleagues what methods to use. If you decide on one, agree on the rules and stick to them. And only let data drive your decisions if your data basis allows it. If it’s broken or you don’t trust it. Invest in getting it right. Finally, be an advocate for your data to make sure everyone understands and uses it in the right way for the right reasons.

Alright, let’s continue with the story because it got worse. The Analyst and I continued to argue and fight over all sorts of details. We were still at war over the nature of the AB test when the implementation hit production. When we checked it, we noticed that the group split wasn’t as random as we thought it would be (because of a technical race condition, I spare you details) and we couldn’t be 100% sure about which user saw which variant.

That didn’t help and as a Product Owner, it was my fault. I saw the analysts as a service provider. I get the stuff built and they will check the numbers if it was successful I thought. That’s not how it works.

The Analyst should have been involved from the very beginning. Even though they might not always join your standups or refinement and planning sessions, Analysts are part of your team. What we talk about as Product Owners might be fascinating to us, but really is Kindergarten for them. They are the experts and if you don’t team up with Analysts, involve them early and most importantly trust and value their contribution, you won’t get the results you are looking for. So, instead of creating an analysis ticket in JIRA alongside your implementation ticket, meet at the discovery stage and continue to collaborate throughout the process. Finally, remember that we are all after the same goal. The questions they ask or the assumptions they challenge are there to ensure better progress towards the goal of creating great products.

Alright, let’s continue with the story because it got worse. Well, we got the issues sorted with some more iterations. But because I was just a little too motivated, I sneaked in a few more variants making it an ABCD test. The Analyst was speechless and went back to his desk. A few days later, he came back with the calculated runtime: 2 months. Of course, I couldn’t believe that. “That long? Can’t you do better than that?” I asked. The analyst bluntly said, “I can’t. But you can certainly do better”. And he was absolutely right.

When we Product Managers demand complex experiments then the result will be more complexity in engineering, planning, design, analysis and so on which not only increases time and effort but also the chance for mistakes.

Less complexity means less time, less effort, fewer mistakes.

If you get results faster, then you can bring value to users faster. It doesn’t always have to be an ABCD test, maybe it’s better to test A versus B, if B wins, get it live and test it against C, and so on. But have that discussion on the best experiment design with your analysts. They will provide you with the best option. Follow it.

Alright, let’s summarise the key messages we had so far:

1. Play by the Rules & Advocate for your Data
2. Team-up with your Analysts
3. Simplify Always

Final Message: Fortune Favours the Brave Have you read Ryan Holidays great new book “Courage is Calling” yet? It’s brilliant.

We should lead by example and focus on the right process. If you feel the pressure to cut corners or maybe to look the other way when there is something wrong with the test design or data then be brave and take a stance. Because you want to base decisions on valid answers driven by data, driven by our users. Everything else is a waste of time.

To wrap things up and give you some closure on how the story ends, I’m happy to report a happy end. The ABCD Test went live and was a wild success. But that’s not the happy end.

My son Lasse was born in August 2017. On my first day in the office, the Analyst welcomed me with the biggest smile and a lovely gift for my newborn baby. And it wasn’t much of a surprise for me. Because we dealt through our conflicts and we got a chance to understand each other’s perspectives. It made us an awesome PO-Analyst duo and we had a much smoother sailing from there.

🏈 NFL Update: We are past week 9 already! While the Cowboys play a great season (6-2), my 49ers (3-5) are last in NFC West. Mixed feelings!

I just finished Digital Minimalism by Cal Newport on Audible.

Recommended Read or Listen? Yes.

While it didn’t blow my mind, I found the ideas in Digital Minimalism worth spending time on.

10 years ago or so, the people who could afford it, where connected all the time. Today, it’s the wealthy people who can allow time for being disconnected, not being available on demand all the time.

10 years ago or so, our devices were able to perform impressive things. But only one thing at a time. Switching from a document to a new browser tab for example took some effort and time. We thought that was bad for our productivity, but what if, the opposite is true?

10 years ago or so, Facebook mobile ad revenue was $0. Today it’s $34.35 billion dollars. Our attention, our data is the new oil.

While many apps aim at getting the most of our attention, we have to manage how we spend it.

Technology isn’t good or bad. It’s what we make out of it.

Some of my highlights from the book include:

1. Digital Declutter: Clean up your devices. Clutter is costly. Remove all clutter. I removed all bad* social apps, any app that I haven’t used within the last month and disabled push notifications.
2. Reclaim Leisure: Schedule seasonal high-quality leisure plans, like crafting something, reading, engaging in a club. Schedule low-quality leisure too (e.g. browsing Facebook, LinkedIn etc.) to fight FOMO.
3. The Bennett principle: We get more energy out of doing something seemingly more demanding than some passive activity. E.g. instead of falling onto the couch and binge-watch Netflix every night, do something real first, like crafting a blog post ;)

Digital Minimalism is about

… the quality of your life in our current age of alluring devices. … it rejects the way so many people currently engage with these tools. …

We cannot unlock this potential (of technology) until we put in the effort required to take control of our own digital life’s to confidently decide for ourselves, what tools we want to use, for what reasons and under what conditions.

The book energized and motivated me to live a more mindful life. I’d say, attention well spent!

*that excludes micro-blog and XING. I explain why in a future post.

Paul Osman’s blog post on Production Oriented Development is pure gold.

With my limited understanding of technology and therefore without knowing all implications, I’m convinced by all of the arguments of that blog post.

I haven’t worked in a company where pre-production environments are on par with production. Hence, any QA performed on such environment can never be fully trusted.

Instead

… teams should optimize for getting code to production as quickly as possible …

Thanks to @jthingelstad for sharing this content piece in the recent Weekly Thing.

Building products with my background in gaming, the topic of “Gamification” comes up regularly. Usually with the idea of making a mundane user task more entertaining and therefore increasing the likelyhood of completion.

First, a bit of theory:

There are three so called brain loads:

1. Cognitive/Mental: Memory, Thinking etc.
2. Motor: Moving hands or arms
3. Visual: Stimulation through visuals

Games raise (= increase) one or more of these brain loads to create an entertaining and engaging experience.

• Visual & Motor: Fruit Ninja, Point and click shooters
• Motor only: Press buttons as soon as you can
• Motor + Mental: Candy Crush, Starcraft
• Mental: Puzzle games

Back in the 80’s, the goal was to lower brain loads to make everything as easy as possible. Now, there are software/web structures that are very established so they can be made more complex again to make things more engaging.

Some fail, and some succeed. I will write about that in a future post.

When developing products, having one team or part of a team focussed on Discovery and then others on Execution is a very bad idea.

It drives silo thinking and a throw-over-the-fence mentality.

People feel less committed when they weren’t there in the room from the beginning.

That’s why “cooperate innovation” labs mostly fail. Product teams have to go through the whole cycle. It will be much faster.

Avoid having teams of mercenaries. Have teams of missionaries.

———— Wise words that I paraphrased from a recent AMA session with the godfather of Product Management, Marty Cagan.

Today, I attended a short webinar on “Conversational Interfaces” hosted by Applause in collaboration with lautmaler. Conversational Interfaces offer great possibilites to onboard and guide users.

It wasn’t a sales event which I was afraid of but actually had some interesting content on the topic. Even though it was rather high level.

My most valuable takeaway is a better understanding of the anatomy of conversational systems.

I also appreciated their separation of the user intent versus the trigger of a message. My intent could be to eat a burger. What triggers a conversation could be me opening the DeliveryHero app. Identify your user intents first, then think about the triggers.

Here are my notes.

1) Conversational UI offers the opportnuity to interact with machines on human terms.

2) There are two formats: Voice interfaces that allow to talk. Chatbots that allow to type.

3) Six factors should inform our strategy to “unmute our brand” to give it a voice: Use Case, Context, Channel, Language, Knowing the User, Brand impersonation

4) Use Cases: Voice for when hands are busy. Chat for others.

5) Consider how your users talk. What could be their intents to communicate.

6) Develop a brand persona e.g. tone of voice, jargon, what kind of connection should be established (long-term or short-term; professionally distanced or amicably familiar…)

The anatomy of a voice Interface looks like this.

(ASR Automatic Speech Recognition) –> (NLU Natural Language Understanding) –> Context User Management and Situational Routing) –> (Logic Implementation of Business Logic) –> (Output Text and Visual CMS Integration) –> (TTS Text-to-Speech)

Remove ASR, NLU, TTS which leaves you with the anatomy of a chatbot interface.

7) Voice is more difficult to get right as it needs a wake up phrase (a la “ok Google”) and a “No Input” timout. A chhat is simple and save as the user just writes a message. The chat can theoretically wait indefinitely.

8) When designing system responses, we need to - identify the users intent - trigger backend calls - keep track and update dialogue context - execute business logic

9) Managing the dialogue could be handled via simple if/then statements and/or machine learning

10) The System should be tested, especially when several use cases exist to ensure proper conversation flow. Pretty sure that’s the Applause pitch here but it’s also true

When optimizing user onboarding flows, the completion rate number of users who started the flow / number of users who finished the flow isn‘t the right KPI.

If that wouldn‘t be true, you could reduce the number of screens to a single one with a big fat „click here“ button and your completion rate will be 98%. 2% always drop-off, no matter how easy it is to click through 🤷‍♂️.

That‘s not why there is user onboarding. It exists to achieve a specific goal of getting your new users to their Aha!Moment - the moment when they realize the value of your product for them.

That‘s not at the end of a series of explainer screens, walkthrough tours, setup screens or other-things-to-get-out-of-my-way-to-finally-see-and-use-what-I-came-for. It‘s when I understand and use your product so it delivers meaningful value. Try to measure that instead.

—— We recently re-designed the onboarding flow of XING and ended up with twice the amount of screens in the flow. Our onboarding completion rate dropped 😎 Here is how it looks like.

A UX Designer actually suggested we should reduce the screens to get more people through the flow. I suppose he didn‘t understand the aim of the flow. It‘s not about maximizing the number of users at the end but to maximize the number of new users who turn into happy, engaged members of our network.

Not sure if we‘ll achieve that it‘s currently running as an AB test but if we don‘t, I‘m sure it won‘t be because of the number of screens but because their content isn‘t the right fit for purpose.

I‘ll let you know when the results are in 👨‍💻