Does Jon Skeet have mental powers that make us upvote his answers? (The effect of reputation on upvotes)

2011-08-04 by . 11 comments

Post to Twitter

Of course since we all know Jon Skeet does have various powers, I will move onto unanswered questions, whether a users reputation makes them receive more upvotes for answers. I’ve seen this theory mentioned in multiple places (see any of the comments to Jon Skeet’s answer that are along the lines of “If this was posted by someone other than Jon Skeet, would this have gotten as many upvotes?”). It is similar to the question we were supposed to address in the currently dormant Polystats project as well.

Examining user and post data from the SO Data Dump, first I looked at the correlation between users reputation and their most recent posts. Below is a scatterplot, with the users current reputation (as of the June-2011 data dump) on the x-axis and the current score on their most recent non community wiki answer on the y-axis.

 

One can discern a very slight correlation between reputation and the score per answer (score = upvotes – downvotes). While consistent with a theory of reputation effects, an obvious alternative explanation is simply those with higher reputation give better answers (I doubt they bought their way to the top!). I’m sure this is true, but I suspect that they always gave higher quality answers, even before they had their high reputation. Hence, a natural comparison group is to assess whether high rep users get more upvotes compared to answers they gave when they did not have as high reputation.

To assess whether this is true, below I have another scatterplot. The x-axis represents the sequential number of the non community wiki answer for a particular user, and the y-axis represents that post minus the mean score of all of the users posts. The scatterplot on the top is all other SO users with a reputation higher than 50,000, and the scatterplot on the bottom is Jon Skeet (he deserves his own simply for the number of posts he has made as of this data-dump, over 14,000 posts!)

In simple terms, if reputation effects existed, you would expect to see a positive correlation between the post order number and the score. For those more statistically savvy, if one were to fit a regression line for this data, it would be referred to as a fixed effects regression model, where one is only assessing score variance within a user (i.e., a user becomes their own counter-factual). Since the above graphic is dominated by a few outlying answers (including one that has over 4,000 up votes!) I made the same graphic as above except restricted the Y axis to between -10 and 25 and increased the transparency level

As one can see, there is not much of a correlation for Jon Skeet (and it appears slightly negative for mortal users). When I fit the actual regression line, it is slightly negative for both mortal users and Jon Skeet. What appears to happen is there are various outlier answers which garner an incredibly high number of upvotes, although these appeared to happened for Jon Skeet (and the other top users) even in their earlier posts. It seems likely these particular posts attract a lot of attention (and hence give the appearance of reputation effects). But it appears on average these high rep users always had a high score per answer, even before they gathered a high reputation. Perhaps future analysis could examine if high rep users are more likely to receive these aberrantly high number of upvotes per answer.

This analysis does come with some caveats. If reputation effects are realized early on (like say within the first 100 posts), it wouldn’t be apparent in this graph. While other factors likely affect upvoting as well (such as views, tags, content), these seem less likely to be directly related to a posters reputation, especially by only examining within score deviations. If you disagree do some analysis yourself!

I suspect I will be doing some more analysis on behavior on the Stack Exchange sites now that I have some working knowledge of the datasets, so if you’ve done some analysis (like what Qiaochu Yuan recently did) let me know in the comments. Hopefully this revives some interest in the Polystats project as well, as their are a host of more things we could be examining (besides doing a better job of explaining reputation effects than I did in this little bit of exploratory data analysis). Examples of other suggested theories for voting behavior are pity upvotes, and sorting effects.

For those interested, the data manipulation was done in SPSS, and the plots were done in R with the ggplot2 package.

Filed under viewpoints

11 Comments

Subscribe to comments with RSS.

  • sbi says:

    Very nice analysis.

    There’s one thing I disagree with, though: IME there’s a rather strong correlation between tag and upvotes. Whether you’re in an obscure tag where about 200 users even look at questions or in the realms of c#, java, android, and iphone does make a huge difference to your upvotes.

    • andyw says:

      @sbi , Sure that seems reasonable, and that is exactly one of the reasons I looked at score - (mean_score). You could have certain individuals that always have high numbers of upvotes due to certain tag participation, and (assuming) those are the individuals with the highest reputation, it would create a spurious correlation between reputation and score. When taking the difference from the mean score for all the answers within a person (assuming their tag participation stayed constant throughout their posting history), it essentially controls for that.

      I have found this paper (Allison 1990) as a useful reference to illustrate this point.

  • James says:

    John, I stand corrected. ;)

  • mbq says:

    Have you removed the +100 bonus? This may be responsible for this “curve” on the first plot.

    • andyw says:

      @mbq , see the comments me and whuber made to Iterator’s post on my follow-up question. The curve isn’t because of 100 bonus, it is due to people gaining all their reputation from one answer.

  • Brad Larson says:

    It doesn’t seem like the post number charts take into account the fact that older posts will have been around longer, and thus have more of a chance to be voted up or down. I certainly see a very long tail on the voting for answers I’ve given over the last couple of years.

    This has come up a few times on Meta, where people have tried to state that questions and answers overall were receiving fewer upvotes nowadays, but they didn’t take into account the time people have had to find and vote for those older posts. No easy way to look at voting over, say, the first month a post had been live had been found from looking at the data dump.

    Of course, if you were to localize the voting to just the first month a post had been live, you’d have to account for the increase in popularity of the site over the years, which might have an effect on voting patterns at the times the posts had been made.

    • andyw says:

      @Brad Larson, It is a reasonable critique about how the older posts have more exposure in time, and hence could be given more upvotes. This would make the correlation between post number and score decrease (and using the fixed effects model probably wouldn’t help with that any). I believe your suggestion of limiting the score to within some specified window would be a good idea to prevent this from happening (and fortunately the data would permit it, as post voting histories are released). I suspect this effect would be trivial, but it is worth taking into consideration in the future.

      The increase in popularity in sites is a completely seperate issue (albeit still pertinent). Although, if the average number of upvotes is increasing as the site grows more popular, it would induce a positive correlation between post number and score (the opposite of what I find). I would have to think about how to take that into account some more (perhaps a variable indicating the average score per answer site-wide for some time period before/during that post) but as is it wouldn’t be a valid explanation for why we don’t observe any reputation effects.

  • [...] a cool analysis on the CrossValidated blog on the effect of reputation on upvotes, which is related to the [...]

  • [...] a cool analysis on the CrossValidated blog on the effect of reputation on upvotes, which is related to the [...]

  • [...] that older posts have a long time to accumulate more posts. For instance, in my analysis of reputation effects it is problematic that older votes have alonger exposure time. Recently a question on average votes [...]

  • Selva Prabhakaran says:

    Great post! Thank you so much for sharing..

    For those who want to learn R Programming, here is a great new course on youtube for beginners and Data Science aspirants. The content is great and the videos are short and crisp. New ones are getting added, so I suggest to subscribe.

    https://www.youtube.com/watch?v=BGWVASxyow8&list=PLFAYD0dt5xCzTQHDhMPZwBoaAXWeVhZzg&index=19

  • Leave a comment

    Log in
    with Stack Exchange
    or