My first post is essentially a planning document for a marketplace monetization experiment. My aim was to create a case study that can serve as a framework for myself when setting up experiments in the future. A reference that can remind me of key questions to ask, how to go about sizing scope, how to imagine plausible outcomes by creating simple models, and how to group and interpret metrics.

The write up is..long. In an effort to be comprehensive, I have traded in brevity. I would not impose a document of this length on any unfortunate stakeholders; rather something like this would serve as the source document from which I would offer them bite-sized chunks. Such a document would be free to access (depending on context specific restrictions!) for anyone curious to take a look, however.

To make a long post digestible, even to myself, I am breaking it up into a series as follows:

  1. First I share the context of the experiment and do a simple sizing exercise to estimate the number of target users. Does reducing fees increase revenue? Context and TAM
  2. Next I create a simple model to conceptualize the results of testing various allotments. Does reducing fees increase revenue? A conceptual model
  3. Then I examine the external impact of the test beyond the treatment group. This is followed by a look into the ASP and conversion combinations required to breakeven. Does reducing fees increase revenue? Externalities and breaking even
  4. The impact of the test on marketplace dynamics over time is considered next. Does reducing fees increase revenue? Dynamics over time
  5. The testing approach, the ability to infer a result from others, the steps to be taken if the test fails are then discussed. Does reducing fees increase revenue? Testing approach, inferring results, next steps if test fails
  6. The above is followed by a detailed discussion on metrics. Does reducing fees increase revenue? Metrics
  7. And finally I end with the takeaway section. Does reducing fees increase revenue? Takeaway

The charts and the data behind them can all be found in this Google sheets.