The payments algorithm and gaming

Hey, all, we recently passed 100 lifters, and I thought with all the new people it might be a good time for a refresh on how we decide who gets paid what, and address a concern (gaming) that came up in a couple of hallway conversations at the event in Boston.

Payments: the basics, and the potential future

Havoc explained the basics of how we do payments a few months ago, and not much has changed.

The tl;dr is that we currently take into account two big factors:

  • usage by customers
  • package size, normalized (stripping out tests, adjusting relative to package manager averages, etc.)

If you’re interested, you should definitely read Havoc’s post for lots of deep details! We also hope to surface how we do normalization in the UX at some point (though admittedly that’s not currently high on the priority list - probably easiest to ask us here if you’ve got questions about it!)

We’ve considered other factors like code complexity metrics (like the venerable SLOCCount), direct input from subscribers, or weighting by package type. So far, we haven’t seen any benefits from these that produce substantially different outcomes and/or far outweigh the complexity cost. However, we do expect that (like any metric) this one will evolve over time, so we’re always open to other suggestions!

Gaming the metric?

One problem with any metric that has money attached is, of course, gaming of the metric. We haven’t seen any serious attempts at this yet, but of course we expect it’ll happen eventually — and the more popular we get, the more it’ll happen.

When it does happen, our response will vary based on the nature and intent of the attack. Here’s a quick reminder/refresher on both our general policy and a specific example.

The good kind of gaming

First, a quick reminder: the current weighting algorithm is heavily based on which customers use the package. So, the best way to “game” the system is to get Tidelift subscribers to use your code. Make your code good! Make it fast! Make it robust! And then convince your users to become Tidelift subscribers! (More on convincing users coming Real Soon Now.)

This is the most gaming-proof system we can think of — unlike Google Page Rank, it is hard to fake real companies using your code, and writing all of us a check. And it naturally reduces other kinds of gaming. Since the other factor is package size, if you’re going around stuffing your package with crap to game our package size measurement, eventually subscribers will notice and stop using your package in favor of something lighter.

Policy on the bad kind of gaming

That said, of course there are ways a dedicated person could try to game this metric (or any other one). So quick refresher on our policy, from the lifter agreement:

Either party may terminate this agreement immediately, without notice, if the other party … materially defaults. Material defaults may include: …

  • any attempts to manipulate or game compliance or scoring systems (including either those created by Tidelift or by host open source projects).

This is very clear, simple, and black and white: deliberately game the algorithm and we will cut you off. If you see anything like this, please let us know.

Specific example — copying code

I want to get into a specific example of potential gaming, since it is close enough to a real world example that it could be confusing.

As you all know, part of development is copying code around. Often this is completely legitimate, and not gaming. For example, we remove vendored libraries when calculating payouts, but they aren’t bad-faith attempts to game the system.

Of course it’s possible to do non-standard forms of vendoring, or copy and paste files/functions/snippets without respecting the upstream license. This can happen for innocent reasons, or for more malicious ones.

If this came up (and it hasn’t yet!), it’s important to remember that the more malicious ones aren’t just gaming Tidelift — they’re a legal risk to our customers too, who might be distributing code without a proper license to it. So if we became aware of this, we’d work very quickly to figure out what happened, and either work with you to remove the problematic code or (if it was intentionally deceptive) remove you from the platform.

Be excellent to each other

We truly believe that a rising tide must lift everyone, not just a select few at the very top. If we work together to bring subscribers to the platform, every lifter will see significant funding gains without the need to game the system. We are currently running experiments to attract more subscribers - please let us know if you’d like to join in on some of these!

6 Likes

Excellent

Thanks for writing this Luis!

2 Likes

I’m curious why tests aren’t counted, as they can be a big part of what makes a project high quality.

fwiw, the main reason in my mind is that some projects put them in the shipping package and some (most?) don’t. The intent of adjusting which code we count is mostly normalization, that is, we’re trying to make projects comparable.

Also, I’m hopeful that quality in general (as achieved via tests, for example) would affect package usage. We’ve tried to avoid making judgments about which technical practices to reward in favor of hoping packages with good practices would see the most adoption.

1 Like