Have Product, Will Travel

Time 2022-08-20 11:15:13

Web Name: Have Product, Will Travel

WebSite: http://www.zumsteg.net

ID:287283

Keywords:

Product,Have,Travel,Will

Description:

Have Product, Will Travel

Derek Zumsteg

Unchecked AB Testing Destroys Everything it Touches

Leave a reply

Every infuriating thing on the web was once a successful experiment. Some smart person saw

Normal site: 1% sign up for our newsletterThrow a huge modal offering 10% off first order: +100% sign ups for our newsletter

…and they congratulated themselves on a job well done before shipping it.

As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money.

As an industry, we are blessed with the ability to do fast, lightweight AB testing, and we are cursing ourselves by misusing that to juice metrics in the short term.

I was there for an important, very early version of this, and it has haunted me: urgency messages.

I worked at Expedia during good and bad times, and during some of the worst days, when Booking.com was an up and comer and we just could not seem to get our act together to compete. We began to realize what it must have felt like to be at an established travel player when Expedia was ascendant and they were unable to react fast enough. We were scared, and Booking.com tested something like this:

Next to some hotels, a message that supply was limited.

Why? It could be either to inform customers to make better decisions. Orrrrrr it could instill a sense of fear and urgency to buy now, rather than shop around and possibly buy from somewhere else. If that’s the last room, what are the chances it’ll be there if I go shop elsewhere?

There’s a ton of consumer behavioral research on how scarcity increases chances of getting someone to buy, so it’s mostly the second one. If a study came out that said deafening high-pitched noises increased conversion rates, we would all be bleeding from our ears by end of business tomorrow, right?

So we stopped work on other, interesting things to get a version of this up. Then Booking took it down, our executives figured it had failed A/B and thus wasn’t worth pursuing, so we returned to work. Naturally Booking then rolled it out to everyone all the time, and we took up a crash effort to get it live.

(Expedia was great to me, by the way. This just a grim time there.)

You know what happened because you see it everywhere: urgency messaging worked to get customers to buy, and buy then. Expedia today, along with almost every e-commerce site that can, still does this —

It wasn’t just urgency messages, either. We ran other experiments and if they made money and didn’t affect conversion numbers (or if the balance was in favor of making money), out they rolled. It just felt bad to watch things like junky ads show up in search results, and look at the slate of work and see more of the same coming.

I and others argued, to the more practical side, that each of those things might increase conversion and revenue immediately and in isolation but in total they made shopping on our site unpleasant. In the same way you don’t want to walk onto a used car lot where you know you’ll somehow drive off with a cracked-odometer Chevrolet Cavalier that coughs its entire drivetrain up the first time you come to a stop, no one wants to go back to a site that twists their arm and makes them feel bad.

Right? Who recommends the cable company based on how quick it was to cancel?

And yet, if you show your executives the results

Control group: 3% purchasedPop-up modals with pictures of spiders test group: 5% purchased 95% confidence

How many of them pause to ask more questions? (And if they have a question, it’s “is this life yet why isn’t this live yet?”)

And the justifications for each of the compromises are myriad, from the apathetic to outright cynical: they have to shop somewhere, everyone’s doing it, so we have to do it, people shop for travel so infrequently they’ll forget, no one’s complaining.

There’s two big problems with this:
1) if you’re not looking at the long-term, you may be doing serious long-term damage and not know it, and you’ll spiral out of control
2) you’ll open the door to disruptive competition that you almost certainly will be unable to respond to as a practical matter

Let’s walk through those email signups as an example case.

What this tells me as a customer is they want me sign up for their email more than they want me to have an uninterrupted experience at the very least. It’s like having a polite salesperson at the store ask if you need help, except it’s every couple seconds of browsing, and the more seriously you look the more of your information they want.

They’re willing for me to not buy whatever it was I wanted, or at least they are so hungry to grow their list they’ll pay me to join, which in turn should make anyone suspect they’re going to spam the sweet bejeezus out of their list in order to make back whatever discount they’re giving out.

As a product manager, it means that company has an equation somewhere that looks like

(Average cart purchase) * (discount percentage) + (cost of increased abandon rate) > ($ lifetime value of a mailing list customer)

…hopefully.

It may also be that the Marketing team’s OKRs include “increase purchases from mailing list subscribers by 30% year over year”

So there’s some balance you’re drawing between cost of getting these emails — and if you’re putting one or two of these shopping-interrupting messages on each page, it’s going to be a substantial cost — in exchange for those emails. Now you have to get value out of those emails you mined.

You may think your communications team is so amazing, your message so good, that you’re going to be able to build an engaged customer base that eagerly opens every email you send, gets hyped for every sale, and forwards your hilarious memes to all their friends.

Maybe! Love the confidence. But everyone else also thinks that, soooooo… good luck?

As a customer, I quickly get signed up for way too many email lists, so my eyes glaze over. I’m not opening any of them. Maybe I mark them as spam because some people make it real hard to unsubscribe and it’s not worth it to see if you made opt-out easy…

Now your mailing list is starting to have trouble getting filed directly to spam by automated filters, so by percentage fewer and fewer people are purchasing based on emails. Once your regular customers have all signed up for email, subscription growth even with that incentive is slowing. And if you’re sharp, you’ve noticed the math on

(Average cart purchase) * (discount percentage) + (cost of increased abandon rate) > ($ lifetime value of a mailing list customer)

Is rapidly deteriorating, and now you’re really in trouble.

What do you do?

Drive new customers to the site with paid marketing! It’s expensive even if you manage to target only good target customers. These new customers want that coupon, so you juice subscriptions and sales. And hey, that marketing spend doesn’t affect the equation… for a while.Send more emails to the people who are seeing your emails! They’re overwhelmed with emails so you need to be up in their face every day! You see increased overall purchase numbers, and way more unsubscribes/marked as spam, and people are turned off to your brand. Which also doesn’t affect that equation… for a while.Increase the discount offered!Well everyone, it’s been a good run here, I’ve loved working with you all, but this other company’s approached me with this opportunity I just can’t pass up…

This is true of so many of these: if you think through the possible longer-term consequences of the thing you’re testing, you’ll see that your short-term gains often create loops that quickly undo even the short-term gain and leave you in a worse position than when you started.

But no one tests for that. The kind of immediate, hey why not, slather Optimizely on the site and start seeing what happens testing will inevitably reveal that some of the worst ideas juice those metrics.

How many executive groups will, when shown an AB test for something like “ask users if we can turn on notifications” showing positive results that will juice revenue short-term, ask “can we test how this plays out long-term?”

As product managers, as designers, as humans who care, it is our responsibility to never, ever present something like that. We need to be careful and think through the long-term implications of changes as part of the initial experiment design and include them in planning the tests.

If we present results of early testing, we need to clearly elucidate both what we do and don’t know:

“Our AB test on offering free toffee to shoppers showed a 2% increase in purchase rate, so next up we’re going to test if it’s a one-time effect or if it works on repeat shoppers, whether our customers might prefer Laffy Taffy, and also what the rate of filling loss is, because we might be subject to legal risk as well as take a huge PR hit…”

Show how making the decision based on preliminary data carries huge risks. Executives hate huge risks almost as much as they like renovating their decks or being shown experiment results suggesting there’s a quick path to juicing purchase rates. At the very least, if they insist on shipping now, you can get them to agree to continue AB testing from there, and set parameters on what you’d need to see to continue, or pull, the thing you’re rolling out.

It’s not just the short-term versus the long-term consequences of that one thing, though. It’s the whole thing, all of them, together. When you make the experience of your customers unpleasant or even just more burdensome, you open the door for competition you will not be able to respond to.

I’ll return to travel. You make the experience of shopping at any of the major sites unpleasant, and someone will come along with a niche, easy-to-use, friendly site, probably with some cute mascot, and people will flock to it.

Take Hotel Tonight — started off small, slick, very focused, mobile only, and they did one thing, and you could do it faster and with less hassle than any of the big sites.

You’re paying for customer acquisition, they’re growing like crazy as everyone spreads their word for free. It’s so easy and so much more pleasant than your site! They raise money and get better, offer more things, you wonder where your lunch went…

If you’re a billion-dollar company, unwinding your garbage UX is going to be next to impossible. The company has growth targets, and that means every group has growth targets, and now you’re going to argue they should give up something known to increase purchase rates? Because some tiny company of idiots raised $100m on a total customer base that is within the daily variance of yours?

I’ve made that argument. You do not win. If you are lucky, the people in that room will sigh and give you sympathetic looks.

They’re trying to make a 30% year-over-year revenue growth target. They’re not turning off features that increase conversion. Plus they’ll be somewhere else in the 3-5 years it takes for it to be truly a threat, and that’s a whole other discussion. And if they are around when they have to buy this contender out, that’s M&A over in the other building, whole other budget, and we’ll still be trying to increase revenue 10% YoY after that deal closes.

There are things we can try though. In the same way good companies measure their success against objectives while also monitoring health metrics (if you increase revenue by 10% and costs by 500%, you know you’re going the wrong way), we should as product managers propose that any test have at least two measurable and opposed metrics we’re looking at.

To return to the example of juicing sales by increasing pressure on customers — we can monitor conversion and how often customers return.

This does require us to start taking a longer view, like we’re testing a new drug, as well — are there long-term side-effects? Are there negative things happening because we’re layering 100 short-term slam-dunk wins on top of each other?

I’m less sure then of how to deal with this.

I’d propose maintaining a control experiment of the cleanest, fastest, most-friendly UX, to use as a baseline for how far the experiment-laden ones drift, and monitor whether the clean version starts to win on long-term customer value, and NPS, as a start.

From there, we have other options, but all start from being passionate and persistent advocates for the customer as actual people who actually shop, and try to design our experiments to measure for their goals as well as our own.

We can’t undo all of this ourselves, but we can make it better in each of our corners by having empathy for the customer and looking out for our businesses as a whole. And over the long term, we start turning AB testing back into a force for long-term

…improvement.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Hinge’s Standout stands out as a new low in dating monetization

6 Replies

Hinge’s new Standout feature pushes them further into a crappy microtransaction business model and also manages to turn their best users as bait, and if you’re a user like me, you should be looking for a way out.

I understand why they’re looking for new ways to make money. First, they’re a part of the Match.com empire, and if they don’t show up with a bag of money that contains 20% more money every year, heads roll.

Second, though, every dating app struggles to find a profit model that’s aligned with their users. If you’re there to find a match and stop using the app, the ideal model would be “you only pay when you find your match and delete the app” but no one’s figured out how to make that work.

(Tinder-as-a-hookup-enabler aligns reasonably well with a subscription model: “we’ll help you scratch that regular itch you have”)

Generally, monetization comes in two forms:

ads, to show free users while they’re browsing, and selling your data

functionality to make the whole experience less terrible

Which, again, presents a dating business with mixed incentives. Every feature that makes the experience less painful offers an incentive to make not paying even more painful.

For example: if you’re a guy, you know it’s going to be hard to stand out given how many other men are competing for a potential match’s attention. So sites offer you a way to have your match shown ahead of users not spending money. If a customer notices that their “likes” are getting way more responses when they pay for that extra thing, they’re going to be more likely to buy them… so why not make the normal experience even more harrowing?

Dating apps increasingly borrow from free-to-play games — for instance, setting time limits on activities. You can only like so many people… unless you payyyyy. Hinge’s “Preferred” is in on that:

They also love to introduce different currencies, which they charge money for. Partly because they can sell you 500 of their currency in a block and then charge in different increments, so you always need more or have some left over that will nag at you to spend, which requires more real money. Mostly because once it’s in that other currency, they know that we stop thinking about it in real money terms, which encourages spending it.

One of the scummiest things is to reach back into the lizard brain to exploit people’s fear of loss. Locked loot boxes are possibly the most famous example: you give them a chest that holds random prizes, and if they don’t pay for the key, they lose the chest. It’s such a shitty thing to do that Valve, having made seemingly infinite money from it, gave up the practice.

Hinge likes the sound of all this. Introducing:

Wait, won’t see elsewhere? Yup.

This is a huge shift.

Hinge goes from “we’re going to work to present you with the best matches with paid features make that experience better” to “we’re taking the best away and into new place, and you need this new currency to act on them or you’ll lose them.”

If you believed before that you could use the app’s central feature to find the best match, well, now there’s doubt. They’re taking people out of that feed. You’ll never see them again! That person with the prompt that makes you laugh will never show up in your normal feed! And maybe they’ll never show up on Discover!

Keep in mind too that even from their description, they’re picking out people and their extremely successful prompts. They’ve used data to find the most-successful bait, and they’re about to charge you to bite.

$4. Four bucks! Let’s just pause and think about how outrageous this is. Figure 90% of conversations don’t get to a first date — that’s $36 per first date this gets you. And what percentage of first dates are successful? What would you end up paying to — as Hinge claims to want to do — delete the app because you’ve found your match?

Or, think about it the other way: if Hinge said “$500, all our features, use us until you find a match” that would be a better value. But they don’t because no one would buy that, and likely they’ve run the math and think that people are more likely to buy that $20 pack, use the roses, recharge, and they’ve got a steady income, or the purchaser will give up after getting frustrated, and that person wasn’t going to spend $500. More money overall from more people spending.

If you’re featured on this — and they don’t tell you if you are — you’re the bait to get people to spend on micro transactions. This just… imagine you’ve written a good joke or a nice thing about yourself, and people dig it.

Now you’re not going to appear normally to potential matches. Now people have to pay $4 for a chance to talk to you.

Do you, as the person whose prompt generated that rose, receive one to use yourself?

You do not.

Do you have the option to not be paraded about in this way?

You do not.

This rankles me, as a user, and also professionally. As a good Product Manager, I want to figure out how to help your customers achieve their goals. You try to set goals and objectives around this — “help people’s small businesses thrive by reducing the time they spend managing their money and making it less stressful” and then try to find ways you can offer something that delivers.

Sometimes this results in some uncomfortable compromises. Like price differentiation — offering some features that are used by big businesses with big budgets at a much higher price, while you offer a cheaper, limited version for, say, students. The big business is happy to pay to get the value you’re offering them, but they’d certainly like to pay the student price.

Or subscription models generally — I want to read The Washington Post, and I would love not to pay for it.

This, though… this is gross. It’s actively hostile to the user, and you want to at least feel the people you’re trusting to help find you a partner are on your side.

I can only imagine that if this goes well — as measured by profit growth, clearly — there’s a whole roadmap of future changes to make it ever-more-expensive to look for people, and to be seen by others, and it’ll be done in similarly exploitative, gross ways.

I don’t want to be on Hinge any more.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Chemistry.com fizzles: a product manager attempts online dating, pt 3

Leave a reply

So far: match.com was not fun, then EliteSingles looked at Match.com’s heterosexual bias, said “hold my privilege,” and set out to make the experience even more coercive, white, and hetero-normative. I did not have a good time. Then I took a couple-month break there because I got an insane flu and then met someone delightful I dated for a couple months, and I didn’t want to revisit this.

Still, we

Next up I went to Chemistry.com. Chemistry, like OkCupid used to, claims to do matching based on a huge number of questions and science. It’s got Dr. Helen Fisher, who I’ve heard on podcasts and seems great!

Chemistry claims their test is “fun, engaging, and provides an in-depth look at who you are and what you want in a relationship.”

I’ll spoil it for you: it is none of those things, and Chemistry offers some clear signs that you shouldn’t trust them.

Anyway, let’s get started? Sure match and EliteSingles were white and heteronormative, but a science-based site like this is going to have a more diverse and —

DAMMIT.

(And I am again using VPNs to test these things from cities with wildly different demographics, that’s not just them guessing I’m straight and in Portland)

I’m sure Chemistry will have a more nuanced set of who can look for what, right?

Nope. You’re straight or you’re gay.

😐

So let’s get into the meat of this. Let’s kick off this personality test.

😑

I kinda gave up immediately. Was the next question going to ask me to feel the lumps on my head and pick the diagram closest to it? What could this possibly indicate about one’s personality?

That critical question answered, you’re introduced to the bulk of the test. It’s 45 minutes of questions, often in succession asking for almost the same thing:

and

Occasionally with a curve ball like this:

or

These moments were welcome breaks from the world of bubbles. Eventually you’re granted questions with different numbers of answers:

When you’re through that ordeal, you get to describe yourself.

Again, I’m really hoping for some better options than we’ve seen in our last two adventures.

Eye color… hair… build…

Hmmm.

…also an interesting set of choices…

Again, hate this question, hate the “marriage is the most important thing” and you’re either not in one, you’re on your way out, out, or you were involuntarily taken out of one. In a loving long-term partnership? Nope! Doesn’t matter… ughhhh.

It takes the “forced choice” approach to getting you to pick some interests. You have to have three, and only three count.

Now to upload your photo. You have two choices. Facebook, or upload.

Wait, what’s that tiny small grey text there? “Skip this step.”

Look, it’s voluntary to sign up for a site like this. If it’s that important to their success, and to the success of everyone else, that there be a photo there, make it mandatory. Maybe don’t spring it on them this late in the process — which is another thing, Chemistry does not tell you it’s going to take so long to sign up.

Then you get the sell on subscribing —

Okay, well, thanks for telling me. I’m curious what those features are — it’s pretty vague what “enhanced search” means, and having the two communication features makes it seem like you might not be able to contact people. It’s an odd choice — I’d really think they’d want to do a better job expressing what the value is here before they make you the pitch.

BUT THIS IS THE PITCH!Continue is actually sign up — now you’re asked for payment. Did you want to skip? Hidden grey text again. Note that here it’s not next to the continue button, but all the way over on the left. This is… intentionally deceptive.

This page is so jarringly different from the design you’ve seen to that point I thought for a moment that I’d clicked on an ad or gone awry somehow. Clearly this is some vestigial code owned by a troll under a bridge, or something.

However I want to focus on a huge breach of trust here.

Let’s say you want that “special profile highlight offer” they’re pushing. $38.94, right?

No!

No.

There is an extra $4 added for no reason. “All new upgrade orders” — is this an upgrade? It’s a new account. What are they talking about? Why does that say “upgrade now?” Am I even in the right place?

What are the chances you realize you’re moving forward with a different amount, given this confusing presentation? This like a hidden fee on your hotel bill where if you look up at the person at the desk they immediately remove it out of embarrassment?

You’re prompted to set up some things that people can ask you, what you’re looking for… I was out by this point, though. However, I’d been sent

The results of my personality test!

What, all those questions about whether I’m into new experiences told you whether I’m into new experiences? THAT IS AMAZING.

Truly a marvel of science. Who knows what the future might bring us?

Yeah, this very much rubbed me the wrong way. It felt like a particularly sophisticated “What Zootopia character are you?” Where all the questions are “do you like carrots?” “Are you good at multiplication?” “Do you have over 1,000 people at your family reunions?” “OMG YOU’RE JUDY HOPPS”

Still, this was — as personality tests can be — an interesting break before I had to face:

The cancellation test!

One of the best ways to learn about a company is by how they act when you cancel. Do they make it difficult? Do you have to call someone? Do they make you go out under a full moon and hold up a solved Rubik’s Cube with both hands and turn three times counter-clockwise, so that you end facing South-by-South-East?

Probably an account status, right?

“Other account status changes” is cryptic…

Oh there it is, the last option.

Why is Date capitalized here? Why is the distinction between casual/serious made here? Why would you stop if you made a friend — isn’t Chemistry about serious people here to meet their partners?

Why aren’t you allowed to tell them you don’t like their site? That’s not a “Technical issue”

Anyway, so pick a reason…

We’re into bad breakup territory here, where everything you say requires more explanation. So you type something in —

You have by my count gone through at least six (and probably a lot more, possibly including looking up a help article on how to remove your profile). You’ve just told them more about why want to remove your profile. And you get this last “wait” modal. It’s just..

I will say it’s nice that they clearly tell you what each of those do, but it’s probably deliberately confusing if someone’s going through this thinking “cancel my account” at each step, gets to the end, and — because Chemistry’s been trying to divert them the whole time — sees “cancel” as the option they want, and “Remove Profile” as a different, non-deletion step. This is not helped by how many other sites — see Match for one example — very much want to keep your zombie self up and boosting their numbers, and try to dance around what profile and account mean.

The end

I’m disappointed. I thought given the association with Dr. Fisher that Chemistry might actually be more… on the up-and-up? More inclusive? By the time I got through the questions, though, I had no desire to see what the rest of the experience was like, and getting out of it only reinforced my impression that I didn’t want to do business with Chemistry. I continue on.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

EliteSingles: a Product Manager attempts to date online, pt 2

Leave a reply

Fresh off our Match adventure, let’s check out EliteSingles. First, scan the page, who do we think their target market is?

 

Huhhhhh.

Even the stock image of the customer support person — I’m presuming, yes — 

😐😑

Moving on. They make some bold claims:

High success rate? Compared to what? What’s the rate? Weird they won’t tell me.

 

It’s a little odd they claim elsewhere that EliteSingles is all about serious dating, then here it lists out some dating phrases (for SEO, presumably) followed by “or find love, idk 🤷‍♀️ ”

I’m on board with the approach conceptually — if they can intelligently select the people, only showing a few could be a huge win. It would let you consistently put effort into it, keep from being sucked in (on a site like Tinder or OkCupid or wherever, you can put in regular, incredible amounts of effort). You would hopefully be able to know that you did your best, that it went to the right places, and feel good about having put the time and energy into it.

I’m hyped! Let’s run through some warning signs!

Disappointing, in exactly the same way Match was. However, this is even more aggressively coercive than Match: once you pick what you’re looking for, you’re auto-taken to sign up:

Please note the creepy results of fading between pictures. I didn’t even notice this until pasted the picture in here.

If you’re, for instance, bisexual, and wanted to click both check boxes, nope. You gotta go. EliteSingles, I know you want to get people through the process and make it quick and easy, but this is gross.

Do Match and Elite do usability testing with a diverse set of testers? At this point I’m willing to bet that Elite’s decided their target customers are heterosexuals, predominantly white heterosexuals, and maybe — maybe — they think about homosexual people occasionally. And if not, they seemingly made a conscious decision to keep same-sex couples off their home page. Why would you do that? Do you think that there are enough heterosexual customers who’ll leave if they see a gay couple? Why?

Anyway. I’m writing a blog, I persist.

It’s weird they ask you for email and password and then take you to this page, which is asking you the same things, including getting you to agree to the T&C. Why would you duplicate this step, especially knowing this kind of early gate is going to have a huge impact on conversion?

Despite having said you’re new — or at least, having passed up the chance to say you’re a returning customer and login — the “already a registered user?” is the most prominent box, the first thing people are going to see on this page. Do that many people enter their email and password on the first page as if they’re new? Regardless, this feels like something you could clarify in one step, rather than have two confusing duplicative pages.

There is also at the top a duplicate “email/password” path to login. This page feels like it might be a vestigial page that’s still in the path after a redesign because it’s load-bearing.

A big point about a small thing: note it would not let me use a + in my email address, which is a cool way to do cool things with your gmail address. It claims it is not valid. It is! It’s in the spec for email addresses! Sure, many sites don’t let you use some special characters, but they are lying that it is not valid. This is not a great way to go for a site you’re going to trust with incredibly intimate information.

I know, it’s an email validation thing, and maybe they have problems with people using valid special characters and it bouncing (I would wager this justification is not backed by good data). 

How you act in the small things is how you act in the big things. I don’t like this. 

Sigh, let’s go.

Please note that one hand has painted nails and a smooth, seemingly hairless arm, and the other — it’s another white heterosexual couple is my point. Do you see this if you’ve said you’re homosexual? Place your bets…

It does. Of course it does.

Anyway, I too am excited to be provided with exciting matches.

What a pointless, offensive screen. One, you’ve already identified yourself as “man” or “woman” in the first screen — a question asked without making it about gender.

But here you go! You have to confirm your gender. Your gender has to be one of those two things. What a crock of shit. I can’t even imagine what it feels like to have dealt with gender identity discrimination and come across a gate like this… when you’re trying to meet people.

Why would you do this?

Anyway, I was mad enough about this that I emailed their support address and said

Hi!
The first question in the registration process is to pick a “gender” of
male/female. It’s 2019, gender’s a spectrum and biological sex is kinda
irrelevant. Why are you asking this as if it’s a binary? What happens
to people who don’t identify with either binary?

A little glib, maybe. Anyway, they got back to me:

Dear Derek,

Thank you for your message.

I appreciate what you are saying and that opinions differ around this
subject currently. We currently don’t have any plans to change this part
of the process however I will be sure to pass your comments along to
the relevant department.

Let me know if you have any further questions.

Please note that the question is not answered. But it’s the “opinions vary” that infuriates me. To have this page, to have this page like this, is a clear choice on which opinion you’re lining up with, and it’s the most regressive and hurtful one. This is such a crazy, dismissive, bullshit response. I want nothing to do with them. I abandoned the first time I hit that screen, and now… I’m writing this. Afterwards I’ll go into one of those sci-fi decontamination chambers where a burst of radiation destroys the outer layer of my skin entirely, hopefully before this seeps in.

How bad does this get? It gets pretty bad doesn’t it? Some highlights!

I’m supposed to confirm their gender? Look, you’re the ones asking intrusive questions so I don’t have to. 

Why, EliteSingles. Why. Are you doing this because Match does it?

Beliefs. Okay, that’s better than asking what your religion is and forcing everything under that label. I’m at least grateful —

WHY WOULD YOU DO THAT.

Is this the touch screen kiosk at a checkpoint in a fevered sequel to the Turner Diaries? What… whyyyyy… ugh.

Can you pick more than one? Guess. No, go ahead. You’ll be surprised at this answer.

I lied, you’re not going to be surprised: one. You’re auto-advanced to the next screen after you pick one. This is yet another point I wanted to throw in the towel.

After answering enough questions to get to about the one-third mark on the progress meter, it’s time for an intermission slide!

Reader…

Is it a picture of what is almost certainly a heterosexual couple?

What do you think?

What are the odds that it is not? What would I have to offer you if you were going to pay me $1 if it was not a heterosexual couple?

$10?

$100?

$1,000?

You’re still not taking $1,000.

Because of course it’s a heterosexual couple. Of course it is. Are they white? Your call, but… yes? It seems like a safe assumption.

Many questions later…

Why not one line that says “Choose as many terms as apply to you” or another single sentence, rather than two?

Then… aren’t many of these terms everyone wants to ascribe to themselves? And who is going to check “unsuccessful” besides people who meant to click other bubbles and missed?

“Yeah, I’m good looking and attractive, but I also want to click honest, soooo distant, cold, argumentative, both dominant and dominating, irritable…”

You’re free to click as many as you want, but then they make you focus:

This is an interesting approach and I’d love to see the data of what people initially pick to what they narrow to. I’d also be interested in how they’re using each of those: are they put to different purposes? Are (as I’d suspect) the second set used strongly in weighing potential matches while the first is not?

Hey there’s another intermission!

Quick, is it a white heterosexual couple looking towards the future?

Don’t roll your eyes at me. 

There’s a ton of multiple choice questions in the process like this:

Interspersed with free text answers like:

 

It’s unfortunate that these are so late. At this point you’ve been in the process for at least 30 minutes from the start. Who’s going to answer these well?

Their distance question is interesting to me:

Even if you want 200, you have to move the slider before ‘OK’ is usable, which I don’t understand.

But my point: defaulting to 200 allows them to mitigate a huge problem for non-Tinder sites. If you join a site like this, especially if you’re paying — and do note that at no point in this process has there been a glimpse of “there are actually people on the other side, see?” as Match.com did — you need to come out of the gate with something. So it’s framed as “are you willing to travel in your search for a partner” and it’s anchored cleverly: the top option is infinite! Are you a true romantic or do you have blinders on in your search?

I’m a little surprised the language isn’t stronger: “How far would you travel to meet your partner?” Any direct suggestion that not traveling farther might mean you’re not going to meet your match. And the “I don’t mind” for infinite distance seems tepid. “I’ll travel anywhere” or “We’ll figure it out” would be stronger language.

I will now reveal what happens when I’m tired, have spent 45 minutes on this process, and come across some baffling UX. It’s…. It’s not flattering.

First, they make it seem like you’re just doing this for funsies, while they’re lining up matches for you. But there’s no progress meter, no updates that they’ve got 1, 20, -4 matches in the queue. So why hang around indefinitely? And why take these seriously, if they’re already able to go find your matches?

I don’t have answers.

Second, let’s talk about how confusing this is. 

There is a “Next Question” button. What do you think “Next Question” does? You type your answer, you hit next question, right?

Because there’s a button there that says “Save & Continue” that, presumably, saves how far you’ve gotten and you come back to it. That’s a reasonable assumption. It’s even labeled “Later.”

No. “Later” is a link, it bails you out entirely. It is a third, different action, which does something larger than those buttons, but is in smaller text. 

If you enter an answer and then hit “Next Question” you are not given a “Discard answer and go to the next question?” warning or anything. You just get the next question, as if everything’s fine. 

Why would you do this? You have three actions:

move to the next question, discarding any answer the person’s entered

move to the next question, saving the entered answer

quit answering questions entirely

I’m baffled why you’d choose the UI they did, where the first button… argghhh.

As you can probably guess by now, I spent minutes answering questions and hitting “Next Question” until — 

I bailed on questions, and as a reward received this screen!

I found this page incomprehensible.

“Member Favorite” like it’s a mobile game asking you to purchase qDollaz or something.

First, the tiers make no sense to me (and I will not be typing them in all-caps). Premium can’t be Light. Premium Classic, sure, and Premium Comfort. But the distinction is Light to Classic? Light to Comfort? Light has fewer features, but there’s seemingly no difference other than length of contract between the other two. But they have different color schemes! Classic is in italics!

Comfort has that gold sparkle on the name! And is otherwise shown in a drab color scheme that isn’t like what we see elsewhere in the site. Why is that one not done in the EliteSingles green? Why is even the “Continue” in a different color — one that hasn’t meant ‘go’ in the process so far? ! Whyyyyy?

Classic is also the only one with a discount! It gets a red badge! And a red price!

Second, this pricing is just wild. As far as I could decipher:

Light has fewer features, but you’re only on the hook for 3 months, so it’s $174

Classic has all the features, for six months, so $210

Comfort runs for twice that at a slightly lower price, so $384

 

I don’t get why it’s framed this way either. The worst quality product is presented as if it’s the most expensive. Maybe they’re trying to anchor people with that higher price, but that’s confusing as hell. “Wait this one is terrible… and expensive?” And then the one they’re pushing, which has the “just right” middle price and middle term, shares the color scheme with the crappy one, features with the Comfort, is discounted, but is still more expensive…

It’s like they wanted the reaction to be “Ugh, what? Oh hey, Goldilocks option before it gets all drab over there, I can’t be seen buying something where the buy button isn’t even enabled…”

This isn’t how anchoring works, though. If you do anchoring well, it’s more like…

Luxury! Our platinum toothpicks… $50,000 for a set of two. Monograms available, inquire.

Good! These reusable toothpicks are made of stylish graphite! $10.

Enh! Toothpicks, like you get in the store. Box you’ll spill on the floor before you get through it, $2

Awful! Made of the worst, most splinter-prone Ash we could source, you’ll hate using this only a little less than your dentist hates you for using them! Crate of millions, $1

 

Right? You can experiment with pricing, number of options, and features, but this is what you want. You want the customer to say “Can I afford the better version I want?”

Here, the worst version as presented is also presented as 

I can only think that they’re doing this because they want everyone to pick that option. If that’s the case, why present options? Or, why not just present one product, one price? Or perhaps just show different term/rate combinations as the way to offer choice?

I don’t get this screen.

I also didn’t get EliteSingles Premium Classic, or any of their options.

…

 

I feel like I need to offer some kind of conclusion, some neat summary of the experience and recommendations from a Product Manager-y perspective.

I can’t. I’m going through this both as someone who has lived in this world, and who would so much like to find one of these services that is not terrible. When I go through something like EliteSingles’ onboarding, I’m just sad. Why not pay attention to things like diversity of appearances? Or whether you’re enforcing beliefs that — even if you truly, fervently believe that there are only two genders, why be a jerk about it to people who think they’re not? What’s the point? Also, gender’s not binary, it’s hugely complicated, I’ll acknowledge we as a society haven’t figured out how to work with that complexity. We can try, though. We can at least do that.

I also always feel like when you notice something like that — and I’m a privileged heterosexual cis white man, I am not nearly as good at seeing things like this as I could be, but I’m trying, and if I see it — it’s a huge red flag that the people and/or the company as a whole are lacking in empathy.

Dating is incredibly difficult and stressful. You don’t want to go into that with someone you don’t trust, and don’t trust to be honest and understanding.

I don’t, anyway.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

A Product Manager attempts to date: Match.com onboarding

Leave a reply

I’ve been in and out of online dating for ages, sometimes paying, often not. After hearing I should be on a pay site for years, a friend joined one, met their partner, and moved in. Since that’s what I want, it seems worth a try. As a Product Manager by trade who has worked on onboarding before, I’m finding the experience fascinating.

Let’s talk about Match.com

Okay, that’s a ton of white people, and all the women have long hair and all the guys have short hair and… I wondered if they were IP sniffing and know I’m somewhere that’s been gentrified to hell, but I VPNed in from other locations (Atlanta was one where I figured it’d have to be different) and it’s the same picture.

That’s a great sign.

So what are our options?

Sigh. That’s not… great. So if you’re bisexual, you have to pick one? If it’s even more complicated than that, you’re entirely screwed? This seems so immediately exclusionary, and to what end?

I’ll continue on because I’m curious if this is going to stick and I enjoy torturing myself with hope. It’s worth noting that the prompt to go forward is “View Photos” which isn’t the user’s goal when they come to the page. They want to find a match at Match, not view photos. If they wanted to make themselves feel awful by looking at beautiful people they’re not going to get with, they’re not going to come here.

Nowwwww they’re putting the carrot in front of you. Here’s the interface you’ll see, there are 1,596 matches if only you’ll give them your Facebook or email.

Once you do that, though, the carrot is removed. There’s work to be done!

That this is not “Relationship Status” but “past relationship status” always pisses me off. I’m divorced. Why does that matter? Why is that different than someone who lived with someone for a decade? Why is this question framed around marriage?

Further, this entirely neglects that there are a ton of other possibilities just within the boundaries of marriage. I don’t like this one at allllllll.

But I really want to talk about something in the background. 

That background image of the (seemingly) hetero couple? Shows on all the paths. If you’re looking for a same-sex relationship and signing up for Match, you get to stare at those two while you’re answering all the signup questions. There will be many questions.

I wonder if the non-hetero abandon rate is higher than it should be just on this. I hope so.

This is so low-effort, too! You need two more pictures so gay men and women see couples that reflect them. Is this just… no one who works on Match is gay? That can’t be true.

This isn’t a potentially painful question or anything, thanks. Also, another question where you’re forced into an answer and maybe you don’t know, or maybe the answer doesn’t fit into these. This whole thing feels… vaguely coercive.

Interestingly, and we can chalk this up to progress or Match just having one path — the questions appear identical between paths. I admit I was afraid I wouldn’t be asked this when I tried those paths.

“Of course they’re asking gay people if they want kids,” you might say. “It’s 2019!”

And again, you have to identify in step one as a man or a woman, seeking a man or a woman. 

Anyway check out this progress bar:

You’ve made progress, for sure, but how much longer? We have no idea. If it was soon they’d tell us, though.

What about only showing dots?

It must be deliberate to not reveal the icons as you go — if you know there are more steps, you forge ahead, but if you knew you were about to get asked about babies and smoking, that’s a look into a future that requires energy and is likely to be unpleasant. I would love to see their A/B results if they tried this.

I hate that ethnicity is always a thing. Why? Because people are searching for or to avoid other ethnicity? Oh right, that’s exactly it.

Framing here is also interesting: you must have a religion. Athiest? Agnostic? That’s a religion. I know, I’m being crotchety now.

They only show the top six options initially, and you have to expand to get to more. Why six? 

Your thought process to even get to those six has to be something like

“Protestants, of course, they’re nearly half the population, and Catholics, another quarter or so. Then it’s people who say no religion, but we can’t just put that, because we’re making them choose a religion. Ugh. Let’s break that down into “non-believers” “don’t care” and “New Age hippy-dippy people” and what’s that leave us? Non-christian religions and Judaism as a whole? Ugh, just… put it behind a more options bonus and call it a day.”

And that “Christian/Other” is first, when that wording makes it seem like a thrift store red tag item. Demographically, you’d think “Christian/Protestant” would be first — it’s almost half the US population. 

This interests screen intrigues me:

One, you’re forced to pick five and only five. There’s a lot of implications in this approach. If you’re like me, you look at this and start to trade off what you’re going to put in there if you only have five. You have to pick the things that you see as most-defining, because you can’t do a laundry list like OkCupid. But also, what if nothing I like is on here? Do I just look at my overfull glass of Pinot and click “Wine Tasting” while sighing?

Also, Pinterest is its own thing? Facebook isn’t. Instagram isn’t. I’m curious how they got to that list.

Now we get to the search.

Nah, I’m not looking for anything specific. I’ll just root through your dumpster or whatever. Don’t mind me.

Derek, how would you word it then?

I’d try something more goal-oriented, like “What do you want in a match?” 

Also, yes, this exercise is wearing on me.

 

Again, their “marital status” one makes me want to scream:

“Hey how are you doing?”

“I used to be bad.”

“Okay, but now?”

“I can only tell you what happened in the past.”

“Cool.”

 

What proportion of people seriously want only widowed or currently separated matches? Is it really enough that adding this to both the profile and the search flow are worth the cost? That seems so unlikely.

And again, I hate that they let you search by ethnicity. It’s just gross.

Now that you’ve picked the kind of flesh you’re interested in, time to flesh out your profile ha ha ha ha ha I’m dying inside, help, please, help.

 

You’re forced to add one topic, and then you have the option to do two more… and wait! Now there’s only one dot remaining at the bottom? We’re ALMOST DONEEEE.

Now! Upload that image!

 

I dig this, actually. It’s clear about why you should upload a picture, it’s clear what the next step is — but again, do people come to Match to view and be viewed, or to find a partner? If it’s a partner, then how much do my chances improve there?

The actual experience fell down for me, too: when you upload, nothing happens for a long time, so you might upload the same image repeatedly, and then it only complains on the next page… anyway.

Hey wait, I’m not a full member? I just did all that work! Continue…

 

This concludes my examination of Match.com’s onboarding process.

Of course it doesn’t.

I will also note that if you have an adblocker on, throughout the process you’ll see you’re avoiding a staggering amount of crap. Disabling the ad blocking does not get that page to load though.

I’ll do them the solid of heading back to the base site. Oh sweet, CAPTCHA in order to log back in.

Kinda feeling like that fire hydrant actually. Fiiiine —

 

Ha. This concludes my look at Match.com’s onboarding.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Learning from uncooperative A/B testers

Leave a reply

One of the joys of working at a tiny startup packed into an ill-equipped, too-small space was running an account at Khaladi Brothers, the coffee shop across the street, because all small meetings had to be done outside the office. As the top coffee nerd, I took on running fresh vacuum pots over (and yelling “Fresh pot!” as I entered) and exchanging the empty ones. When we moved to a newer, spacious, swanky, and quite expensive office space (hot tip to startups: don’t do this) with an actual kitchen and drip coffee maker, I was put in charge of deciding which coffee beans we’d order. We had many options and an office of high-volume consumers with strong opinions on everything, and needed to get down to one or two for the recurring bulk order.

Naturally, as a Product Manager, I decided to do selection through a series of A/B tests.

Must-have for the tests:

end up with a winnerclear methodology, publicly exposedeasy to participate — or nottakes as little as my time as possible, because this was amusing but everything’s on fire all the timekeep momentum out of it (so the first voter didn’t disproportionately determine the day’s winner)

I discarded forced-choice, so coffee drinkers didn’t have to vote if they didn’t feel like trying both or didn’t find a winner, I decided against setting up a dedicated “today’s test” table or doing “three-sample can you tell if one’s different” type advanced testing, I didn’t try to timestamp votes to determine if one did well fresh and one did well through the day… nope!

I went straight single-bracket, winner-advances, random seeding at each round. Every day I tried to get to the office before everyone, and made two giant pots of coffee labelled “A” and “B”. If someone wanted to vote for a winner, they could write it down and drop it in a tin, which I tallied at the end of the day. I will admit that having come out of Expedia, where our A/B tests were at colossal scale with live customers, this whole thing seemed trivial and I didn’t spend as much time as I might have.

You may already see where some of this is going. “I know! I too am from the future,” as Mike Birbiglia says.

It was not trivial, and I ended up learning from the experience.

Test assumptions, set baselines: I didn’t have 32 coffees, which was good, because some days I did an A/A test to see what the difference would be. I was surprised, on those days voting for winners was down, and results were remarkably close to 50%/50% — and the highest split was 58% (10/17), which was a vote off a straight split.

Know that blind tests mean subjects may reject results or: Starbucks did really well. I don’t know what to say. I figured they’d barely beat out the clearly awful generic ones and Tully’s, but some of their whole beans did well got all the way to the semi-finals. Participants were not happy to learn this, came by to ask questions, and generally were reluctant to accept that they’d preferred it. If a Starbucks bean had won but it had made people unhappy, would I have gone through with ordering it? I’m glad I didn’t have to confront that.

Also… yeah, Seattleites have issues with Starbucks.

Consider the potential cost of testing itself. The relatively small amount of time I thought it would take each day turned into way more effort than I’d hoped. Doing testing in public is a colossal hassle. Even having told everyone how I was doing it, during the month this went on, there were those offering constant feedback:

it should be double-blind so I don’t know which pot is whichit should have three pots, and they might all be the same, or differentno, they’re wrong…it’s easy to come in early and see which one I’m making

…and so on. By week two, getting up early to make two pots of coffee as someone offered methodological criticism was an A/B trial of my patience.

If testers can tamper, they will — how will you deal with it? For one example, I came into the kitchen one day to get a refill and a developer was telling everyone he knew what pot was which because he’d seen me brewing and had poured an early cup off that, and so knew the pot with the lower level indicator was that batch. He was clearly delighted to tell everyone drinking coffee which one they’d picked. I honored the day’s results anyway.

This kind of thing happened all the time. At one point I was making the coffee in a conference room to keep the day’s coffees concealed. In a conference room! Like a barbarian!

I was reminded of the perils of pricing A/B experiments, which Amazon was being called out for at the time — if customers know they might be part of a test and start clearing their browser cookies and trying to get into the right bucket, how does that skew the results? “People who reloaded the page over four times converted at a much higher rate… we should encourage refreshing!”

Think through potential “margin of error” decisions when structuring tests. There was a coffee I liked that dominated early rounds and then in the semi-finals lost by two votes to a coffee that had squeaked by in previous rounds by 1-2 votes each time. What should I have done in cases where the vote was so close? I’d decided the winner by any margin would advance, but was that the way it should have been? Should I have had a loser bracket?

In the end, we had a winner, and it was quite good — and far better than what the default choice would have been — but I was left unsatisfied. I’d met the requirements for the test, it’d been a pain in the ass for me but not taken that much time. I couldn’t help but think though that if I’d just set up a giant tasting session for anyone who cared, and let them vote all at once, I’d have saved everyone a lot of trouble and possibly had a better result.

But more importantly, like every other time I’ve done A/B testing in my product management career, the time I spent on the test and in thinking through its implications and the process helped me in every subsequent test, and was well worth it. I encourage everyone to find places to do this kind of lightweight learning. Surely there are dog owners out there wondering what treats are best, and dogs who would be happy to participate (and cheat, if you’re not wary).

Go forth!

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Two dual-track agile attempts, and lessons for future generations

Leave a reply

I’ve been through two tries at implementing dual-track agile, and I’d like to offer some perspective on the travails, the pros, the cons, and offer some advice to those who might attempt it.

What’s dual-track agile?

In short — you pull a product manager, a designer, and a developer with broad knowledge of the area you’re working in forward in the process, taking on scouting and evaluating work, which then drops into the backlog of whatever agile methodology you’re using.

This is intended to solve the problems of how you develop the stories that a team will work on — the design sprint, the reset sprint, the investigation card, or in carving capacity for those tasks out of the team’s capacity — which in themselves often provoke both a set of methodology challenges and a level of bikeshed argumentation about methodology that can be immensely draining.

Here’s Marty Cagan on this in 2012
And here’s this good Mind the Product article

Which includes this good gif:

Implementation one: everyone jump into the pool at once

At Simple, we’d been through a couple of massive crash re-platformings that hadn’t delivered new features to customers. We two product teams (one, mine, working on what would become Shared Accounts, and another working on long-neglected customer service features, so we were going after things that would create value for our customers, but we were still not shipping.

We brought in Silicon Valley Product Group to do an assessment and recommendation. One of their number came in, did interviews with key stakeholders, and then in a day-long session in which almost everyone was present (we had our recruiters there!), told us what they’d seen as problems and offered a set of prescriptions. The biggest and most systemic one was to go to dual track agile.

Our leadership then declared we’d adopt dual-track agile, and so it was.

It didn’t take. We adopted dual track in name, but in practice, we couldn’t get the developers with the required knowledge to participate, and so discovery withered. Ours could not move into doing that work, being continually pulled into on-call, supporting retiring the deep technical debt, and doing the architecture work that would keep a team working.

Without developer participation, the discovery track could evaluate whether work was valuable to the end user, and whether that value could be realized by customers, but it still meant that items about to go to the team did not have a ROI, because we still needed to figure out the I. And in turn, that meant that there still needed to be an additional step in what should be the “delivery” track to do even high level development investigation.

What could we have done?

no methodology exists in a vacuum — consider the people and teams that will be doing the work. The people may be willing, but if circumstance can’t permit, you’re set up to failif there are changes you can make that would allow you to use a methodology, you have to make the changes or wait — don’t do the reorg or change your approach and just hopea top-down mass implementation wasn’t the way to go — ever the pragmatist, I’d have rather found a team where it was a particularly good fit, done it there, and then learned lessons that could spread. When we started doing Scrum on teams at Expedia, we got approval to try it in three teams with good conditions and very different problems (flight search, hotels, and packages) and were able to learn from each other and measure our success against other teams and spread the good word.

Two: easing into it

As Lead Product Manager at Manifold, I was able to drive methodology. I decided to follow my own advice and do it real subtle-like. We started to use Product Pad, doing more and more discovery activities… and my intention was to use more and more of the process until we’d be using dual-track across the teams without having to talk about it.

During this time, my boss, the VP of Eng, encouraged me to just do it! We’re small, you can do a lunch-and-learn and just go!

I, having been burned by the previous experience, and encouraged by success at gradual adoption of Scrum at Expedia, declined and decided to go ahead with moving forward. Let this be a lesson to us all:

If you have the opportunity to jump ahead, and the support to do so, maybe just do it

Dual-track took… kind of. Unfortunately for dual-track, the company made a hard business pivot and organized around long-term contractual obligations (so product teams organized around delivering promised functionality, rather than pursuing objectives of their own). There’s a little bit of work to be done within that, but you’re not going out to do basic user research, address problems, etc.

Fundamentally, dual-track exists to support testing ideas, learning, and exploration. If you’re not doing that as a business, don’t adopt a framework that supports it. It’s like the difference between the road-trip and doing a regular commute. One requires research, planning, friends, snacks, a good dog to hang out the window if you’re lucky, and the other requires you to figure out the vehicle and route once and then pay attention.

I also encountered more resistance in the form of nearly-endless tool and process questioning than I’d expected or was prepared for. I found myself answering questions like “What we do when robosaurs attack local food distribution centers is a good question, and we’d have to talk about what that would mean for ticket handling…””

Now, what was going on? There were two culture clashes with Manifold’s stated values of transparency and autonomy, where anyone should be able to do anything as long as there was an audit log and the action was reversible.

1: dual-track itself seemed to clash with culture: that there would be three people who, outside of a product team’s normal activities, would be making decisions on the direction of the team and what work there was to be done.
2: tooling: first, introducing another tool for people to use raised the “why not just do all this in (JIRA/github/___)?” and in particular, the tool I’d introduced, Product Pad, is great for a lot of things but has a ton of restrictions on roles (for instance, a normal user can’t go add stories to an idea filed by someone else) that rankled people. I had not done enough to consider this.

What happened?

We started, ideas and feedback go into the hopper, there are processes to do discovery, and it’s being used on some more-nebulous pieces to create a roadmap.

I feel like I should have abandoned it when we made our pivot — I should have considered that a change in direction and how we work as a company was worth wiping the white board clean and evaluating the process challenges against what the kind of work we’d be doing over the next 12-18 months.

Questions to ask and things to do as you weigh dual-track.

First, think about this:

What’s your elevator pitch for why the change is worth it — what will you gain, or what known harms would it have prevented?What’s that pitch for each of the stakeholders in the process? What’s your pitch to your Marketing partners, for instance?Who’s your champion at the exec level?Consider the culture: will people react poorly to the appearance of a trio of people taking over the direction of the team where once it seemed open to all? What can you do in designing the process and communication to address those concerns?

And then, block out your calendar for a long time so you can concentrate, and

Map out the process from end-to-end before any public rollout or discussion.

How will an idea or customer request go through the process? What tools will be used? Consider the maximally complex cases —

A developer has an ideaWe write up the idea — where? How is it tracked?How do you decide that that idea is worth investigating out of all the other ideas? Who makes that decision? Where?We need to do some user research on the idea — who does it? Where is that request tracked? How does the result get tracked?We need to show a prototype to users, and it requires both design and a little bit of dev. How do you start and track that work? How do you record the work?You now know that the idea has merit and can be made usable. How do you track that?How do you cost out the work? How is that work assigned, and tracked?How do you choose what goes onto the roadmap from all of the ideas that have cost/benefit ratings? How?If you’re doing regular user research or usability testing, how will that fit? Where will those results go?How will you communicate each decision out to everyone who is interested in the results?

Then for each person in the process, list how many tools they must use including any new ones, and how they’ll track their work at each stage. If they live in a world of GH issues and you now require them to put their head into Product Pad regularly, or participate in discussions there, you’re adding complexity into their life and they’ll have to see a lot of value come out of this extra effort — which may fall to you as a product manager.

Now you’ve got a rollout plan, of sorts, with an idea of the cost and what will need to happen for the process to be a success, and you can make an informed decision.

In summary, dual-track agile is a methodology of contrasts

Having been through two attempts to implement it, I’m much more likely to look at existing processes and find ways to build in things like regular user testing and fast feedback loops, and see if they improve things — or start to create the need for dedicated discovery process and activities.

I would love to hear other people’s success or failure stories implementing dual-track agile (or any of the latest hot methodologies) — please, drop me a line if you’ve got them.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Obsess over teams to drive organizational change

1 Reply

In “Stop Obsessing Over ‘The Teams
John Cutler argues “we shouldn’t focus on “The Teams”, until the conditions for effective teams are in place.”

Cutler begins:

Most Agile methodologies/frameworks focus myopically on “the teams” (front-line teams of individual contributors). Meanwhile, organizational dysfunctions (those causing 80% of the drag) persist.

And that you have to solve organizational problems first.

In a tweet, Melissa Perri adds

Companies usually ask me to “transform teams” but many of the problems they need fix stem from the org level. Local optimization over global rarely works. You have to work on both.

Everyone’s right — the trick is, sometimes it’s agile methodologies are what let you find the global problems, and make wide organizational change necessary.

Cutler continues:

The dreamland utopia of “The Business” and “The Developers / The Teams” just working it out over mugs of coffee … is ultimately unfeasible in any org — with >1 teams — that simply tries to graft something like Scrum on to their existing structures, dependencies, and project culture.

No.

When faced with something that seems intractable, you start with what you can control, improve what you can, and force the organizational change that seems unfeasible.

I worked at Expedia way back during a rough patch for the company, and if you’d asked “what’s going on here?” you’d get a litany of entirely different and well-supported answers from different organizations. Or teams. Or people at adjacent desks on the same team with different roles. Top-down “let’s unlock transformation” attempts went nowhere. The business deteriorated and everyone knew why, but no one could agree on the reasons, much less what to do.

Three of us Program Managers (Adam Cohn, Yarek Kowalik, me) got permission to turn our teams to Scrum from the Microsoft process Expedia kept after spin-off, which went:

1) Strategy comes from somewhere
2) Business crunches numbers, hands a direction to team in Business Requirements Document
3) Program Manager turns that into a full specification down to a really deep technical level
4) Spec is reviewed and iterated on until it gets stage-gate approvals
5) Development goes until it hits “Dev complete”
6) QA until it hits “QA complete”

…the whole thing.

There were all kinds of organizational and business problems, all the stuff Cutler points out. The teams weren’t really autonomous, the Program Managers were effectively Product Owners, Project Managers, and deputized Product Managers, it took forever to ship, what did ship didn’t do well — all of it.

We had to hand-lobby leadership and make ourselves pests until we got permission to try Scrum just on the team level, just on those teams. Adopting was rough, and there was a time where transparency and the ability to produce stats meant we were the only teams admitting we weren’t making progress.

But it worked. It transformed everything.

It started with the immediate team and radiated out:

showing the teams’ work created a sense of increasing success where there’d been frustrationseeing the effect of gates on productivity let us remove the “toss over the wall” steps and get everyone closerreporting how having design do one set of “red-lines” in the spec and then only respond to critical issues delayed iteration got us more and better collaboration throughoutdocumenting the problems with releasing code helped drive efforts towards continual delivery

With the “engineering can’t ship” out as an excuse, it kicked off organizational change. Once, no one could agree on why things weren’t working in our organization. Now we could prove them, and when people agreed on the causes, we could solve them.

For instance, tracking the work done and the cost of it meant we could show executives the productivity-draining effect of business-level randomization. I could go and say “as the business, you told us to prioritize quick-wins in response to this metric, nothing happened and we made no progress on what you want for four weeks” and the leadership of that organization had to grapple with what that meant.

That started a conversation first about how much time we put into tech debt, and on-call, and soon it was a business-level reckoning of “are we doing the right things at all?”

Solving that drove change through the once removed business organizations: it required Product Managers who could act quickly, pragmatically, who were available and collaborative (I wrote about what I learned from working with one of them ). And for the first time, you could tell who was an effective Product Manager, and that organization transformed.

All of those changes were painful and it took way longer than anyone wanted. And if you’d started omniscent, knowing what all the barriers were going to be, sure, it’d have been way easier to make a list. But we’re not. Instead agile methodologies let you see and prove the barriers to the team’s work and force action, and then you get to see the next one.

When large organizations are faced with holistic questions, you don’t get progress. At best you get a weighty consultant-produced report on how you’re deviating from best practices, and each of the groups tasked with work finds some way to argue their bit of the findings don’t apply, or feet are dragged and no progress is ever made.

Agile methodologies allow you to flip that: to not have to solve the whole thing at once. To find the things that are small, and fixable, and get better as you build clear and incontrovertible cases for the next implementable organizational change. You don’t know what the effect of a set of wide-ranging best practice announcements will be, but you know having a designer embedded on product teams will make everyone more effective.

Obsess over teams. Not only is it the often the best way to create change, in large organizations beginning with with the team is the only place it can originate.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Review of Jason Calcanis’ Angel: Looking Down at You from a Useful Vantage

Leave a reply

Calacanis’s Angel contains useful insight if you’re interested in dipping your toes in funding startups, if you’re thinking of starting one and want to know what the other side sees, and for those of us who are around or just interested in the world.

That insight costs putting up with Jason himself, and that’s enough to make this hard to recommend to anyone without this advice:

Do not under any circumstances read Chapter 1
Skip anything about how great Jason is or his history

The rest of the book is relatively tolerable, but I wrote “fuck you” in the margins repeatedly in this chapter. It kicks off some themes that keep coming up in the book that are problematic.

I’m going to entirely ignore Calacanis’s opinion of himself and his abilities, and how important he seems to find it that you too believe in him. I’m personally extremely skeptical of the kind of rabid, persistent self-promotion he engages in throughout. The book’s here, we can roll our eyes to the sky, and move forward. There are much larger issues here. You can check out this profile of him in the NYT titled “Should the Middle Class Invest in Risky Tech Start-Ups?”

Skip past this section if you’re already familiar with the problems with Calacanis (and much of Silicon Valley)’s attitude.

Jason thinks you’re pitiable moron if you for some inexplicable reason aren’t also a VC

First, privilege

Our parents and grandparents took factory and white-collar jobs and rode them for all they were worth in the last century. Now the robots — who never sleep and self-improve on an exponential curve — are taking these jobs. Meanwhile, we humans, with our nagging psychological and emotional needs, struggle to keep up.

Most of you are screwed.

But you’re here, so you’re clearly willing to learn and I can radically improve your odds if you do the work.

First, note his use of “you.” It’s “our” for the parents, but when it comes to who’s screwed, he’s clearly comfortable on his lifeboat seat.

This is important. In order to participate in this new economy, he suggests starting in a “syndicate” where someone like Jason finds investments, you and his crew of small investors jump in with him, and Jason takes a fee for making the match. To do this, you must be an accredited investor. Here’s the standard for that, which changed in 2011.

You must be:

worth more than $1 million dollarsyour primary residence doesn’t count towards that

Angel was published this year. A million dollars is about 90th percentile in the net worth of US families (and I believe that counts home equity, but figure it doesn’t so we can forge ahead with even 10% qualifying). There’s an assumption throughout that you, the chump, can just start doing this. He refers to it as “taking the red pill” at one point (and let’s for a moment ignore that term’s association with some deeply toxic people).

For 90% of people, there’s no option on the investing path. Calacanis can’t “radically improve your odds” if you can’t participate. He has another option for those people, I’ll get there.

Second, that’s paired with contempt.

Calacanis goes back and forth in the book from being insufferable to being able to look at his own history and even at times in his own luck to heaping shit on people who aren’t in the tech industry.

Cafe X and other startups will also eliminate millions of jobs in which humans get paid to stand behind a counter and repeat back your seven precious little instructions on how to prepare your morning libation, before pressing one button and masturbating a milk-frothing pitcher for two minutes.

I’m not going to argue even that he’s wrong that baristas could be endangered. His tone is fucking unacceptable though. Think about someone you know that has a “service” job and does it amazingly well. Doesn’t even have to be a barista, though it would warm my heart if it was entirely counter to Calacanis’s example. It’s almost certain that what they do beyond the core and potentially mechanizable task is what makes them great.

What does a great city bus driver, for instance, do beyond drive the bus? They’re getting people in wheelchairs on or off and secured, they’re doing customer service like helping people navigate a public transit system, and infrequently but critically they’re defusing dangerous situations or dealing with crimes as they happen. Why shit on them for not having a million dollars and investing in startups that will put their fellow co-workers on the street?

Now, to his credit, Chapter 8 is “How to be an Angel Investor with Little or No Money” and then 9 is “The Pros and Cons of Advising” — but this chapter’s about signing on as an advisor, a probably unpaid position in which you work for equity. Which, and again, Calacanis doesn’t seem to realize this — requires you both have skills useful to a startup, like his internet marketing expertise, which very very few of the 90% of people without a million dollars will, and also the connections to the startup, which that 90% won’t have, and time to make that happen, which the 90% are probably not going to have — particularly since the jobs being created by Calacanis & Co. are frequently high-effort, no benefit gigs that I’m sure Calacanis would express contempt for if he thought to (“…eliminate millions of jobs where humans get paid to pick up your overly-precious sushi order and ride their overly-burdened bikes in their ridiculous helmets, weaving to dodge potholes and confusing my autonomous car’s prediction sensors causing it to slow and making me several moments late for my next pitch meeting…”)

This is — and despite being written in first-person, it’s possible he did not write this — a stark contrast to what the book promises, that if you’re a “mom in the bookstore with screaming kids, the sales executive in the airport exhausted from layovers, or the kid graduating from college wondering, “What now?”” that you can be like Calacanis and “live in a big house with a bunch of Teslas in the driveway and an ATM balance receipt that makes me smile from ear to ear every time I see it.”

They can’t. That college student’s almost certainly graduating with crippling debt and joining an economy with little to offer them. The mom with the screaming kids? Is she going to take them to the pitch meetings? Tech bros don’t need daycare to meet with founders. And — yeah.

What’s good here?

Once we’re well into it, there’s some legitimately worthwhile stuff regardless of Calacanis himself.

Chapter 11 starts into how you can start by making tiny deals in angel syndicates, and use that to gain experience in how to evaluate founders and make connections with other investors and people in the community, and how you can help your founders.

The sections on evaluating startup pitches and what you should do to prepare for and how to be in those meetings are great.

Chapter 22, “Why Angels Should Write Deal Memos” is great advice for almost any kind of decision like investing in a startup (investing generally, for instance) — that you need to create a clear case for why you’re acting not only as a clarifying thought exercise but so that you can look back at it in the future and compare it to where you are, offering a valuable reality check.

Then the walk through of what it’s like to be an investor of a startup in those early years, and how it will be challenging, are all excellent advice.

This is also where Calacainis’ experiences can make the story — when you hear about him getting screwed over in a deal, or acting poorly, it’s easy to sympathize and learn from him. When he’s rude in response to something, you get it, and there are lessons there both in how to conduct yourself as an investor but in what to watch for to avoid being in that situation.

For founders, you’ll be able to see it all through the investor’s eyes, and walking through the pitfalls of investing (in particular, that the money is not going to get you where you think it will and everyone outside the startup knows this). Knowing the expectations of a good investor in both communication and what they want to hear so they can try to help is excellent to set yourself up well.

On balance

I was well into this book still angry about its opening attitude and considering a pointed email to Calacanis. Once I took a walk and could set it aside, I returned to get into the nuts-and-bolts, and it was entirely worthwhile.

If you take my advice and skip that first chapter and then gloss over any of his “I’m amazing” self-reassurance, you’ll have a much more pleasant experience in getting all the good stuff.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

A long review of Phil Town’s 2006 Rule One Investing

1 Reply

Who’s the author and do we care?

Phil Town is the author of two books, 2006’s Rule #1 and Payback Time

Town’s origin story is that casting about after serving in Vietnam as part of the Green Berets, he worked as a river guide in the Grand Canyon, where an older investor he met offered to teach him the ways of the stock market.[1] He then claims to have turned $1,000 into $1,450,000 in five years and became a full-time investor[2].

Now, he’s got a popular podcast where he’s teaching his daughter Danielle Town (who is so great as a foil and voice of the listener), and wants you to attend his seminars and subscribe to his stock research tools from him.

Should I read this or not?

Totally worth reading if you don’t already have a valuation strategy or process you’re into, with the caveat that theres’s some confusing additional material that might make you scratch your head.

If you do, or you’re more generally familiar with value investing (why it can work, particularly), there may not be a lot here.

The meat of the book: how to find good businesses on sale

Distilled, Rule One tells us to invest in good stocks that are 50% off. He’s got the “Four Ms” for you

MeaningMoatManagementMargin of Safety

And then walks you through each one. Some are easy to understand: for “Moat” he talks about competitive advantages you can count on over time. For “Management” you’re looking for some key metrics, and there are good explanations of why you want to look at each one.

“Meaning” isn’t quite the right term (and to his credit, on his podcast Town’s joked about how the neccessities of book marketing forced the names so there would be a catchy “Four Ms”) — but here, do you understand what the business is doing, and would you buy the whole business?

Where the real meat is for people will be in how you value and price stocks, looking for the discount. And here’s where a lot of math comes in, and I respect that: this isn’t Greenblatt’s “Magic Formula” where you just rank all the stocks. For every stock, Town walks through how to estimate the future value of the stock, and compare it to alternatives, like investing in U.S. Treasury bills, or against expectations.

I dug this a lot, but I also had to break out the spreadsheets, and it’s way more of a pain to get all the data together (this as much a problem with the web in 2017 though). Handy solution though — you can just subscribe to his website and get all those numbers in one place.

You end up with what Town calls the “sticker price” for a stock, the current fair value for a company given expectations for its growth. The explanation for the concept is excellent — that price should be a starting point, and you want a sale.

Running those numbers was an eye-opener for me and taught me a lot about how to think about stocks as both being a company’s value in the present but an expectation of future growth as well. For instance, I’d been griping that one company seemed wildly under-valued based on their price/earnings ratio. When I ran through the process here, I understood that it was priced perfectly reasonably.

Once you can calculate the sticker price, you want stocks that are at least 50% off. For this to be true, either the company is deeply out of fashion or something has gone horribly wrong and the stock’s being beaten up over bad news. As I write this, the CAPE (Cyclically Adjusted PE Ratio) is at 30.62, which is historically very high, and you’re not going to find many stocks on sale for half off.

Going off the rails: add technical analysis

Chapter 12 is “The Three Tools” and here Town talks about “red light” and “green light” indicators for whether to buy a stock. Here we diverge entirely from the value of a stock and get into some entirely different territory, and where I think the book loses people. Buying quality businesses on sale is simple, and while it takes a little work, it makes sense as you’re doing it.

In Chapter 12, suddenly we’re not just looking at business, we’re now trying to figure out what everyone else is doing, and things get way harder to understand.

The three tools:

Moving average convergence divergence. As a measure of whether there’s “pressure” pushing the stock up or downStochastics. Measures if something is “overbought” or “oversold” — and the explanation of this one is not greatMoving average.

I’m going to say up front I didn’t get this chapter at all. I feel like I’m missing the point, so this will be a bit more uncertain.

This chapter confuses the issue by . For instance, you’re supposed to look at Stochastics for crossings of the 14-day and 5-day lines. For one, most places you look you don’t get a ‘buy’ and a ‘sell’ line as the book describes them, you get a “%k” and “%d” and it’s confusing. When I got through the instructions for setting one up using Town’s settings for time frames, I looked up some stocks, and it was a coin flip on whether the stock went in any discernable direction.

I don’t know if I was just unlucky or if I’m even using the tools right. My point is that if you’ve gone through the book and you’re working with pretty clear, understandable numbers (earnings per share is pretty much earnings per share), this shift to difficult to understand technical indicators is confusing and exaperating.

Moreover, there’s no convincing why here. The explanation of what we’re doing in this chapter doesn’t make it clear why it’s important to pay attention to it. It seems like the fact that you’ve found a company that’s on sale for 50% off should far outweigh whether others are getting into or out of the stock at the moment.

I was left thinking there are important signals I shoud pay attention to in addition to basic valuation, but unsure how I’d do that.

This is also where the book falls down on the cover promise of “successful investing in only 15 minutes a week!” Even if you’re only monitoring the few stocks you’ve bought, using these three tools once a day to see if you should bail on a stock, you’ve burned your 15 minutes squinting at whether one line crossed another.

It all feels like Town, as a long-time experienced investor familiar with tools like these, was unable to explain these or their importance in a way that made sense to those without the same experience, which is unfortunate because I got a lot out of the valuation sections.

And from there, much good is undone

Chapter 13 then walks through a possible example as a couple considers buying The Cheesecake Factory, Inc. The valuation all makes sense, and they buy. A chart of spectacular returns is shown — CAKE goes from $18.90 to ~$36 in two years, an 90% gain. Nice. The couple’s example $20,000 is now $38,000. Except… here’s what Town writes:

By getting in just below $19, and then moving in out 11 times in two years with the big guys, and by adding $500 a month they were saving, by July 2005, CAKE gives Doug and Susan a nice compounded rate of return of 56% and their $20,000 is now worth $78,000

He’s claiming twice that return rate.

Nope. Nope nope nope nope. Just… not true. At all.

1) Is the regular additional contribution counted as part of the return? It’s unclear. But why mention it otherwise? You can’t have them put more cash in and count that as what it’s worth now. I could have an $100 investment that’s totally flat and add $1m and wow, I beat the stock market over two years because now I have $124. This is, at best, confusing. At worst, he’s counting over $10,000 and associated gains in value as part of the return that shouldn’t be there.
2) They moved in and out 11 times? How do we know that? Does this assume that they perfectly timed each of those? When were those 11 moves? What did the technical indicators say at the time?
3) They’re going to get creamed on short-term taxes. The investor who held for more than a year pays way less.

Investor A: bought at the same time and held. +$18,000, may be paying zero in taxes
Investor B: bought, as a beginner times moving in and out 11 times, up $58,000, might be $40,000 after taxes.

If you think Town is counting the additional contributions in taxes, we’re down to around $30,000 for Investor B.

It’s not okay that this important calculation is so unclear, and the handling of the additional contributions makes me extremely suspicious. It doesn’t make sense as written, and I don’t understand what the point is. 89% over two years is great! Why confuse things?

This raises a larger question: does this all work? We’re presented with a couple examples of Company A compared to Company B, walked through the Cheesecake Factory example, but beyond those hand-picked examples, all we have to rely on is Town’s accounting of his track record. I’d have felt much better about the whole thing given a wider accounting of his trades, or of studies done on historical data — like we have for Greeblatt’s Magic Formula book.

Put it all together, what’s it mean

I dug reading it, and particularly found it useful in making me do real work putting things together and looking at companies. But I also found the technical tools section confusing and hard to understand, and the example math particularly worrying. It occupies a weird place in an investment bookshelf: I can see recommending it to someone who is interested in Warren Buffet’s investment philosophy and wants to know how to crunch the numbers to find those great companies at attractive prices.

[1] The story, as told, is so perfectly mythic that it’ll probably raise at least one eyebrow if you read it. But whatever.

[2] (I spent a little time looking at this, but besides him raising money in 2013 for Rule One Capital, I couldn’t find anything on early investment proof or his returns as a manager).

[3] Similarly, Town claims early that as a Vietnam veteran, on his last day in uniform he was at the Sea-Tac International airport when someone spat on him and ran away. This… I don’t know. That spitting at veterans happened at all is disputed: there’s a whole book on this, Spitting Image by Lembcke, discussing how no incident like this has ever been documented as part of a larger case that antiwar activists and veterans were allied far more than is now generally believed.

I thought about looking into this in greater detail (particularly, why would he be at Seatac on his last day in uniform, when it seems like he’d have flown into McChord Air Force base?), and stopped. This has been hashed out many other places, and it doesn’t seem relevant. Let’s just grant Town this.

Share this:

ShareClick to share on Twitter (Opens in new window)Click to share on LinkedIn (Opens in new window)

Hi! I’m Derek Zumsteg

In current order of time spent, I'm a Product Manager at large, followed distantly by biking, reading, and writing.My last truly cool trip was to South Africa, which I highly recommend.

Recent Posts

Unchecked AB Testing Destroys Everything it TouchesHinge’s Standout stands out as a new low in dating monetizationChemistry.com fizzles: a product manager attempts online dating, pt 3EliteSingles: a Product Manager attempts to date online, pt 2A Product Manager attempts to date: Match.com onboarding

Follow me on Twitter

My Tweets

Subscribe, for some reason

RSS etc feeds

TAGS:Product Have Travel Will

<<< Thank you for your visit >>>

Hot Websites