Research in Public #04: Post fee influences post quality
- Introduction
- Recap: Territory posting fees affect quantity of posts
- What about post quality?
- Interpretation?
- Next steps
Introduction
In #1243188 I suggested to @k00b that we engage in a research project using SN data. The idea would be to use this data to study: A) how micropayments with real money affects internet discourse; and B) barriers to the adoption of self-custody. I also promised @Undisciplined that I'd carry out the research in public, since many people might not know what economics research looks like, and may be curious as to how the process plays out. You can follow all of the updates here.
Recap: Territory posting fees affect quantity of posts
Last time (#1253062), I showed using a twoway fixed effect regression that territories with lower posting fees attracted a greater number of posts.
-
The correlation could not be explained by territory subject matter, since the model allowed every territory to have a different baseline. Thus, the relationship is identified off of changes to territory posting fees and the change in subsequent quantity of posts.
-
The correlation could not be explained by global time trends in SN usage, since the model allowed every week to have a different baseline effect. Thus, the relationship is also identified off differences in the size and timing of posting fee changes in different territories.
The results were both surprising and unsurprising. They're unsurprising because demand slopes downwards. But they're surprising because the costs are small, yet people are still demonstrably responsive to them.
What about post quality?
Ok, so territory posting fees affect the quantity of posts. Does it affect the quality of posts? As it turns out, the answer seems to be yes.
Measuring quality
First, let me explain how to measure post quality. I use two methods:
- The sat value of zaps within 48 hours of posting,
- The number of comments within 48 hours of posting.
I use a 48 hour window so that older posts do not have an advantage over newer posts, simply for having been around longer. Moreover, we can demonstrate that 95% of zaps are earned in the first 48 hours, and 90% of all comments:
In addition to the 48 hour requirement, I ignore posts that:
-
Where the poster is the territory owner.
-
Bio, freebies, and territory-less posts like Saloon posts.
The resulting dataset contains about 180,000 posts with which we can regress posting cost on post quality.
Posting fees and zaps
I run the following regression:
\log Y_{istu} = \beta \log X_{istu} + \mu_{s} + \delta_{t} + \eta_{u} + \epsilon_{itu}where
i indexes a post, s indexes the territory that the post was made in, t indexes the week when the post was made, and u indexes the user who made the post. Y_{istu} is the sat-value of zaps in the first 48 hours, and X_{istu} is the posting cost paid.\mu_{s}allows for arbitrary baseline differences in post quality across territories\delta_{t}allows for arbitrary time trends in post quality across weeks\eta_{u}allows for arbitrary baseline differences in post quality across users
The regression results, with various inclusions of the fixed effects, are shown here:
===================================================================================================
Dependent variable:
-------------------------------------------------------------------------------
log_sats48
(1) (2) (3) (4)
---------------------------------------------------------------------------------------------------
log_cost 0.335*** 0.226*** 0.237*** 0.104***
(0.003) (0.003) (0.007) (0.007)
Constant 2.775***
(0.008)
---------------------------------------------------------------------------------------------------
Territory FE N Y Y Y
Week FE N N Y Y
User FE N N N Y
Observations 179,653 179,653 179,653 179,653
R2 0.079 0.159 0.181 0.406
Adjusted R2 0.079 0.158 0.180 0.390
Residual Std. Error 2.268 (df = 179651) 2.167 (df = 179539) 2.139 (df = 179313) 1.846 (df = 174835)
===================================================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
The results show that posting fees are positively associated with zaps, even after controlling for arbitrary territory effects, time trends, and user effects.
-
The relationship cannot be explained by territories with higher posting costs having subject matter that is zapped more frequently. This is accounted for by the territory fixed effects.
-
The relationshp cannot be explained by global time trends on SN. This is accounted for by the week fixed effects.
-
The relationship cannot be explained by better users preferring to post in territories with higher costs. This is accounted for by the user fixed effects.
My preferred specification is actually model (3), that doesn't include user fixed effects, because isn't part of setting territory fees about related to what kinds of users you attract? The coefficient of 0.237 in model (3) implies that posts that paid double the post cost can expect to have have 18% more zaps within the first 48 hours of posting.
You might be interested in the territory fixed effects, to see which territories attract the most zapped posts. Here they are:
Posting fees and comments
I run the same regressions above, but with number of comments in the first 48 hours as the dependent variable instead of zaps. The results are as follows:
===================================================================================================
Dependent variable:
-------------------------------------------------------------------------------
log_ncomments48
(1) (2) (3) (4)
---------------------------------------------------------------------------------------------------
log_cost 0.065*** 0.042*** 0.059*** 0.025***
(0.001) (0.001) (0.003) (0.003)
Constant 0.644***
(0.004)
---------------------------------------------------------------------------------------------------
Territory FE N Y Y Y
Week FE N N Y Y
User FE N N N Y
Observations 179,653 179,653 179,653 179,653
R2 0.017 0.107 0.117 0.351
Adjusted R2 0.017 0.106 0.115 0.334
Residual Std. Error 0.971 (df = 179651) 0.926 (df = 179539) 0.921 (df = 179313) 0.799 (df = 174835)
===================================================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
Using again, my preferred specification (3), the coefficient of 0.059 implies that posts that paid double the post cost can be expected to get 4.2% more comments in the first 48 hours.
Again, the relationship cannot be explained by baseline territory differences, time differences, or user differences.
Here is a chart of the territory fixed effects for comments, so you can see which territories attract the most comments:
Interpretation?
What's driving the result? As I already mentioned, it can't be because higher cost territories simply have better subject matter. It also can't be because of anything happening across time to SN as a whole. And lastly, it can't be because better users tend to self-select into higher cost territories.
Instead, the result may be driven by one or more of the following channels:
- High posting fees discourage low quality posts by bots
- Users want to post in relevant territories, but if the territory cost is high, they won't post there unless they know the content is good
- Users are willing to pay extra fees (i.e. 10x fees for posting twice in 10 minutes) if they know their content is good
- Territory founders who run high-cost territories are more generous with zaps (I realized that the territory fixed effects don't rule this out, because founders can change throughout a territory's lifetime).
Channels 1-3 are consistent with the hypothesis that pay-to-post increases post quality. Channel 4 doesn't, because it is less about post quality and more about the behavior of the territory founder. In future work I will need to try and rule out option 4, probably using a founder fixed effect.
Next steps
As mentioned above, an immediate next step is to re-try these regressions with founder fixed effects.
Another interesting angle to establish is whether the moneyness of the zaps really matters. That is, does it matter that zaps are real money? Or would users be responsive to these micro-incentives even if they were just points on a scoreboard? (i.e. Reddit reputation, etc.) It's plausible that they moneyness doesn't matter, and that people treat SN as a game to earn points. But it's also plausible (and more likely) that the moneyness does matter. This may be hard to test, but I think it's a worthy question to explore.
Anyway that's all I have for today. Anyone who wants to vet the code can go to https://github.com/ed-kung/sn-research. I'll keep posting any time I spend a day doing substantial work on this project.
-0.246, #1253062) and on zaps per post (+0.237, this post), are pretty close in magnitude. It may net out to nearly zero in terms of total sats earned on posts.