A/B Testing on Etsy: Rendered Pictures

Imagine I put two pictures in front of you:

Which one do you like better? That’s the question I posed to my customers recently. The business significance of finding an answer to this question was massive.

Business Context

Let me explain. When creating listings for products on Etsy, it’s highly recommended you add pictures. After all, the platform tends to be more visually driven and it helps customers know what they’re getting. In my segment (3D printed D&D miniatures), it’s even more important. One of the key differentiating factors between me and other 3D printers are the designs of the models we sell. Taking great pictures is a great way to articulate the value in these differences.

But there’s a downside to the conventional approach. For every new listing I launch, I have to print the design and stage a photo shoot. The total time to print a design could be ~4 hours and staging a photo shoot is another 30 minutes. This might not seem like much for a one-time deal that pays dividends in perpetuity, but there’s no guarantee that a given model will be successful. In fact, the distribution of sales across models resembles a long-tail.

Now - what if instead of having to print out each model and run a photo shoot, I could just render the model via software? It would mean:

Sounds great! It’d free up considerable amounts of my time and allow me to quickly expand my listings into the triple digits - greatly increasing my keyword penetration on the platform.

Experiment Setup

Of course this all assumes customers wouldn’t mind - so I decided to run another A/B test. The same one I had you take at the beginning of this post.

The rest of the setup is quite similar to my previous experiment on pricing:

In this experiment, I was mostly worried about click-through rates as opposed to conversions - since there were no changes to pricing - so I ignored sales.

Results

Here are the descriptive stats for views:

Mean Std Dev N
A 0.51 0.72 326
B 0.51 0.78 326

**p-value = 0.93

Next Steps

Looks pretty damn similar to me! The p-value is quite high and leads me to conclude that I can move to using rendered pictures without any effect on my click through rates.

This is a pretty massive win for team MiniPrinter and opens the pathway for 100+ listings active on Etsy in short order. It also opens up interesting questions to ask going forward:

All extremely exciting questions I plan on answering as I continue down this journey.