[GW2] Small Salvaging Samples

After paying an absurd amount of attention to salvaging rags, something of a consensus has emerged from the official forums, gw2guru, youtube, and reddit on how salvaging kits work. (I'm calling it basic chance theory after Zoodokoo.)

As noted in the reddit thread, one way to account for the minuscule differences in gossamer returns between salvage kits is that instead of increasing the amount of gossamer, the kit modifiers multiplicatively increase the chance for getting gossamer.

So, supposing that gossamer has a base 11% chance of being salvaged from a rag, the chance of getting gossamer with each kit is:
Crude:      11% * 1.00 = 11.00%
Basic:      11% * 1.10 = 12.10%
Fine:       11% * 1.15 = 12.65%
Journeyman: 11% * 1.20 = 13.20%
Master:     11% * 1.25 = 13.75%

As I noted in the reddit thread, this would be entirely consistent with the data I've collected so far.

While it is great to have a theory to test (other than just "everything is consistent with a bug") there are two things that are troubling about this theory.

If the theory is correct, then most kits aren't worthwhile (NeckNeckNeck's observation).

Supposing gossamer is worth 3.5 silver (350 copper) and silk is worth around 20 copper. So getting 1 gossamer is a gain of about 320 copper. A 1% increase in the chance to salvage gossamer is worth about one hundredth that, or about 3.2 copper. Getting a 1% increase in the chance of getting gossamer compared to crude kits must cost less than 3.2 copper to be worthwhile.

Therefore, assuming the base chance of gossamer is 11%, it is only worthwhile to upgrade to basic kits (for rags).

The bigger trouble is that statistically verifying the theory is going to take absurd amounts of data. The effect sizes here are tiny. As I discussed elsewhere, statistically verifying a tiny effect is tricky business.

Here, I've simulated the results of 1000 different attempts at salvaging 2000 rags, split evenly between crude and mystic kits. For each attempt, I recorded the size of the effect that was estimated and whether it was statistically significant. Sadly, I only find a statistically significant effect 33% of the time. And when I do find an effect, its overestimated.

What does it take to get it right?

Looking at different combinations of kits and sample sizes, its only after 10k salvages, or using absurd amounts of black lion kits, that results can be found reliably and accurately.

I don't even want to think about how much gold that represents. (Dear ANet: can we get a Tyrian NSF going?)

I'm taking a different approach. I've started to collect data on everything I salvage. I'm hoping that I can see a clear pattern where `rare' means 'higher tier of materials'. Hopefully, I'll also find a kind of item with a fairly high chance of being salvaged into the higher tier. This will make for a larger difference between the kits, and therefore something easier to estimate.

Wish me luck!