This week marked our annual Product Barcamp at XING. While around 40 Product Managers gathered in the New Work Harbour, another 40 or so joined online. As the days get shorter and the cozy winter season is about to start, I sat down to read a story to my fellow peers. It’s a story about obstacles, failure, and even war from past experimentation endeavors of mine.

📖 Story Time!

The story kick-started a valuable discussion about how we Product Managers approach experimentation and in particular AB-testing at XING. It concluded into a new working group that will try to progress on this topic on a regular basis.

Maybe it will spark new thoughts, questions, or impulses for you too? Here it is.

Three Experimentation Pitfalls Experimentation

Pretty much exactly 5 years ago, in the fall of 2016, I worked in the UK on a game called PokerStars. With millions of daily active, paying users, it’s an enormous business.

I was in charge of growing new users, making sure they get to create an account, fund it, understand how to play the game and find a game that suits them so they get the most out of their time and money and come back again in the future.

After doing some research and sparring ideas with stakeholders, one opportunity stood out as the one to pursue further. The idea was to insert a visually engaging screen that features a welcome offer for new users.

Everyone was on board. It was a no-brainer. No questions. But those were just opinions, right. I needed scientific evidence!

So, I got to sit down and prepared the JIRA tickets needed in order to get an AB Test started. Old, obviously shitty screen versus shiny, obviously much better screen. It went into the magic of engineering and as we approached a state of readiness, I got to spend more and more time with my analysts. So far, so good?

Hmm no. From there, it went downhill. We had multiple hypotheses and couldn’t agree on the most important one. With that, we weren’t on the same page regarding which KPI should be the deciding factor. We couldn’t even agree on whether this should really be an AB test. Why? Because we didn’t play by the same rules.

The Analysts goal was to simply answer a question in a clean and accurate way. He wanted to test a hypothesis. My goal was to prove that my grand idea is as grand as I think it was. And I decided to make it an AB test because people know that the results are non-discussable. They are proven by science. In a way, I was asking for any positive number that I can use in a way to make this look like a success. I was asking for absolution and didn’t really care much about the data. What should I have done instead?

At the beginning of every experiment, make sure that there is a clear hypothesis to answer. Be honest with yourself that it’s not about producing some evidence to prove your point. There are easier ways to get that! Evaluate together with your analysts and user insights colleagues what methods to use. If you decide on one, agree on the rules and stick to them. And only let data drive your decisions if your data basis allows it. If it’s broken or you don’t trust it. Invest in getting it right. Finally, be an advocate for your data to make sure everyone understands and uses it in the right way for the right reasons.

Alright, let’s continue with the story because it got worse. The Analyst and I continued to argue and fight over all sorts of details. We were still at war over the nature of the AB test when the implementation hit production. When we checked it, we noticed that the group split wasn’t as random as we thought it would be (because of a technical race condition, I spare you details) and we couldn’t be 100% sure about which user saw which variant.

That didn’t help and as a Product Owner, it was my fault. I saw the analysts as a service provider. I get the stuff built and they will check the numbers if it was successful I thought. That’s not how it works.

The Analyst should have been involved from the very beginning. Even though they might not always join your standups or refinement and planning sessions, Analysts are part of your team. What we talk about as Product Owners might be fascinating to us, but really is Kindergarten for them. They are the experts and if you don’t team up with Analysts, involve them early and most importantly trust and value their contribution, you won’t get the results you are looking for. So, instead of creating an analysis ticket in JIRA alongside your implementation ticket, meet at the discovery stage and continue to collaborate throughout the process. Finally, remember that we are all after the same goal. The questions they ask or the assumptions they challenge are there to ensure better progress towards the goal of creating great products.

Alright, let’s continue with the story because it got worse. Well, we got the issues sorted with some more iterations. But because I was just a little too motivated, I sneaked in a few more variants making it an ABCD test. The Analyst was speechless and went back to his desk. A few days later, he came back with the calculated runtime: 2 months. Of course, I couldn’t believe that. “That long? Can’t you do better than that?” I asked. The analyst bluntly said, “I can’t. But you can certainly do better”. And he was absolutely right.

When we Product Managers demand complex experiments then the result will be more complexity in engineering, planning, design, analysis and so on which not only increases time and effort but also the chance for mistakes.

Less complexity means less time, less effort, fewer mistakes.

If you get results faster, then you can bring value to users faster. It doesn’t always have to be an ABCD test, maybe it’s better to test A versus B, if B wins, get it live and test it against C, and so on. But have that discussion on the best experiment design with your analysts. They will provide you with the best option. Follow it.

Alright, let’s summarise the key messages we had so far:

1. Play by the Rules & Advocate for your Data
2. Team-up with your Analysts
3. Simplify Always

Final Message: Fortune Favours the Brave Have you read Ryan Holidays great new book “Courage is Calling” yet? It’s brilliant.

We should lead by example and focus on the right process. If you feel the pressure to cut corners or maybe to look the other way when there is something wrong with the test design or data then be brave and take a stance. Because you want to base decisions on valid answers driven by data, driven by our users. Everything else is a waste of time.

To wrap things up and give you some closure on how the story ends, I’m happy to report a happy end. The ABCD Test went live and was a wild success. But that’s not the happy end.

My son Lasse was born in August 2017. On my first day in the office, the Analyst welcomed me with the biggest smile and a lovely gift for my newborn baby. And it wasn’t much of a surprise for me. Because we dealt through our conflicts and we got a chance to understand each other’s perspectives. It made us an awesome PO-Analyst duo and we had a much smoother sailing from there.

🏈 NFL Update: We are past week 9 already! While the Cowboys play a great season (6-2), my 49ers (3-5) are last in NFC West. Mixed feelings!

Last week, on the day of my birthday, we went back to the office. After more than 1,5 years of working from home, without seeing most of my colleagues face to face, this was huge.

⛴ Our new office, the “New Work Harbour.” is described as one of Germanys most interesting places to work at and I couldn’t agree more. Overseeing ships cruising through the Hamburg harbour, it’s fabulously designed, featuring a variety of spaces that invite to collaborate and have fun. We have a band room, gym, pub, library, fire place and family rooms. A canteen that grows it’s own herbs inside the building, sports classes every day and even a very good barista coffee bar. There are so many spaces designed specifically for collaboration that I really really really want to have as many workshops as possible in the future.

It’s the office from the future.

🤝 Going back to the office also means seeing people again. In any kind of organisation, humans and their connections are the most important thing. And that connection really took a hit during the pandemic. While the bond within our team, grew even stronger, the whole company suffered. When you don’t see the human side, it’s so easy to dismiss others and judge them. Especially our leadership team had a very hard time. Even though they made plenty of mistakes, they are fighting for the same mission as every one else at New Work. And as every human, they make mistakes too. And they will learn from it and do it better next time. It’s difficult to see that with distance. It’s easy to see that with proximity.

🙌 That’s why I’m a big fan of our Mobile50 policy. We ask everyone to work at least 50% of the time from the office and the rest can be used working from home. It’s not controlled, checked or tracked. It’s a trust-based ask. Because we believe that our great company culture is part of our way of new work. And our culture consists of shared experiences, putting post-its on the wall, having kitchen chats, partying until the sun sets and last but not least to conquer obstacles together, as a team. How can this ever be done in a sustainable way if everyone sits in front of their screens? How will a new starter ever get to really know her colleagues? How will we ever trully celebrate our wins and learn from our failure if we don’t even see each other?

👏 Imagine if you are alone in a room, clapping by yourselves. Feels wrong, doesn’t it? In our monthly Cluster review this Monday, we clapped after every presentation. And we enjoyed it so much! Because we were a group of people in the same room. Such a good feeling.

Follow the sign to find Growth

🚴‍♂️ Of course there will be challenges. My legs still hurt from cycling an hour to the office and then back for an hour. This means 2 additional hours that I have to find in a day. But also, 2 additional hours I spend outside, exercising on my bike.

I didn’t even know I can cycle at 45 km/h!

My new hybrid working week will look something like this:

• 🏢 Monday: Wake-up before anyone else to get some work done, bring kids to Kindergarten, cycle to work, shower and change, meet the team (Monday is our office day when most of the team will be present), collaborate on challenges as much as possible, meet people, share coffees, change, cycle back home.

• 🏡 Tuesday: Wake-up before anyone else to get some work done, then bring kids to Kindergarten, then working from home - Focus Mode with a few meetings like planning sessions.

• 🏢/🏡 Wednesday: Wake-up before anyone else to get some work done, then bring kids to Kindergarten, then sometimes working from home for focus work, sometimes going to the office for more collaboration work.

• 🏡 Thursday: Wake-up before anyone else to get some work done, then working from home - Focus Mode. I try to have no meetings on Thursdays except for the team standup and our Leadership Team weekly.

• 🏢 Friday: Wake-up before anyone else, then be at the office before 7 a.m., exercise at the gym, shower and change, followed by a mix of meetings and using the various spaces at the office for some inspired, creative work. Fridays are my favorite office days. Because the weekend is near, people are less stressed and therefore I can get more done than on any other days.

I’m already looking forward to Monday.

🏈 NFL Update: In Week 2, the 49ers beat the Eagles and Dallas defeated the Chargers in a very close game. So far, so good.

Days get shorter and nights longer. Mornings start with crisp, fresh air until the first ray of sunlight. Summer officially didn’t end yet, but you can certainly feel that something is about to change. 🍃

I hope you had a fabulous summer! After the long, cold and - thanks to the pandemic - lonely winter, we all deserved a proper break, didn’t we?

The Neubauer’s household spent a lot of time outside with many long camping weekends and a proper multi-week vacation in 🇩🇰 Denmark. The whole extended family made their to Kerteminde. It’s a lovely little coastal town on the island of Funen, north of Odense. Nice beaches, high quality design stores, great restaurants and the most delicious liquorice ice cream! We rented this Airbnb and were once again super happy to use Airbnb instead of more traditional rentals. You just get so much more bang for your buck.

Our VW Bus “Kalle” ready to board the 🇩🇰 Ferry

Even though the Covid-19 pandemic is far from being done, it feels like we have it under control. At least here in central Europe. Two thirds of Germans are vaccinated and pretty much everything is open again. There are strict measures of course, like social distancing, wearing masks, getting tested and so on. That’s a bit annoying but so much more better than lockdowns. Also, borders are open, giving us back our freedom to move as we please. Denmark even just dropped all Covid-19 measures!

For us, the timing couldn’t be more perfect. After maternity and parental leaves as well as “home office only”, our life as a family is changing drastically. Rike just started in her new school, as the head teacher of a new first class. Lasse goes to his Waldkindergarten and Jonte did a fantastic job at getting used to his child minder. When I drop him off, it takes a minute and he waves toward my direction. That’s his way of saying “Daddy, please leave now” 😢.

Next week will be the Grand Opening of the New Work Harbour, our fancy new office in the Hamburger HafenCity district. I couldn’t be more excited as it also means that I will return to the office for at least 50% of my time. Working with my team face-to-face is the best. We actually met last week already as I invited everyone for a 🥩BBQ in my garden. That was the first time we all saw each other since 1,5 years. Insane!

Do you spot our 4 colleagues from 🇵🇹 Porto who joined remotely?

It’s been quiet on this Blog in the past months. I could blame it on summer, family and work busyness. But that would be too easy. Truth is that Blogging on a regular basis for an infinite time is hard. Probably too hard to keep it enjoyable and creatively inspiring over the long run.

Hence I decided to go for a seasonal publishing schedule. And the first season starts now. Since I’m an 🏈 NFL fan ever since the NFL Europe came to Berlin in 1998, the NFL season will guide my Blogging schedule.

This week, Tampa Bay kicked things off against the Cowboys. And so will I. Welcome to the season. Welcome to Blog S1!