[POE] Building Better Builds: Forum Posts, Part i

I've become interested in making my Path of Exile builds better.

One way to do this is to see how other players are creating their builds. Since this is a pretty big topic, I'll first focus on passive skill tree choices by players.  I'm particularly interested in these because many players post guides to their builds - including passive skill trees - on the forums. (Data time!)

I grabbed the 600 most recently updated threads in each class forum and extracted build links on the first page of each of these threads. Because PoE's passive skill tree is constantly changing, I removed builds that are no longer valid with the current skilltree or were less than 80 points. Because sometimes authors post multiple versions of a build, I selected only one build from each thread (the build closest to 100 points). This left me with 914 valid builds.

	Duelist	Marauder	Ranger	Scion	Shadow	Templar	Witch
valid builds	101.0	106.0	144.0	123.0	166.0	112.0	162.0
all builds	707.0	540.0	657.0	732.0	633.0	591.0	615.0
average posts	68.8	79.3	47.7	68.2	49.1	58.8	80.6
average views	25071.7	33006.9	17866.2	18892.1	14617.9	15087.8	32962.1

Curiously, Templar has the least build options to chose from, combined with relatively few posts / views for each thread. I'm curious whether this will lead the developers to consider rebuilding the templar starting area.

Defensive specialization: 
Whether in a hardcore or softcore league, one of the biggest challenges in Path of Exile is simply staying alive. For this section, I'll focus on player choices to get life, energy shield, armor, and evasion. When evaluating life- or energy- based builds, I omitted any build with less than 50 of that stat (because, well, death).

A few observations:

• Life-based builds tend to get around 200 life, with marauder builds getting slightly more life than the other classes. I'm actually surprised at how many builds get less than 200 life, which I considered an absolute necessity. 
• Rangers and shadows tend to focus on evasion, while duelists and marauders focus on armor. 
• Of all the classes, only witch has multiple builds getting a 'reasonable' amount of energy shield for CI / low-life. 

There is, of course, a potential problem with this data. The wide range of life / energy shield values suggests some of these builds simply aren't viable. For example, a large number of builds from new or inexperienced players might tend to undervalue defensive stats. To correct for this, I weighted the data by the number of posts in the thread (on the assumption that high-quality builds are more likely to get attention).

While weighting the data did remove a large number of the low-defense builds, the basic story is unchanged: players tend to get at least 200 life, energy-shield based builds are uncommon, and strength-focused classes get armor while dex-focused classes get evasion. It is notable that a few high-attention builds (like cyclone marauder) tend to dominate the data.

Weapon specialization:
Turning to offense, here I looked at whether a build grabbed a damage specialization node (+damage with a weapon type / spell).

A few features that stand out (to me):

• Wow, caster options are popular (even omitting elemental damage builds).
• Ranger is overwhelmingly the most popular choice for bow builds. The lack of a bow-templar makes me consider running one.
• One-handed builds are more popular than two-handed builds for all classes.
• Swords are slightly more popular than axes, despite the focus on soultaker in the current meta, though this is potentially due to buzzsaw builds. 
• Specializing into wands / staves are remarkably uncommon options (2.6% and 1.4% of builds, respectively). Staves make sense - other than Pillar builds, not many characters go for a staff (they're also devilishly hard to roll). But wands? I seem to remember everyone trying wander early in the league.

Thoughts:
I'm frankly a bit surprised at how few defenses many builds grab, particularly given the common refrain that 200% life is a bare minimum for PoE. I'm equally surprised at how many builds choose to specialize in a weapon, given how that limits purchasing options at endgame.

There are plenty of places I'm considering taking the analysis. I'm definitely going to look at keystones. I'm also interested in some general descriptive-stats for the passive tree (shortest paths, nearest clusters) and how this influences the build meta.

So many fun data adventures to be had.

[POE] Maps, Levels, and Quantity

I love maps in Path of Exile. But I've always been curious about the details of the map drop system. Does it get harder to get higher level maps? Exactly what does quantity do?  What about the desirable mods (pack size, more magic monsters, more rare monsters) - how much quantity are they worth?

As usual, the forums have an abundance of speculation about these answers. But I wanted math.

So I started to collect data on the maps that I've been running in PoE. Unfortunately, this kind of data gathering takes time, so I was only able to gather data on around 500 maps. Which just isn't enough data to study processes that are based on rare events.

To take this further, I combined my data with a few other projects that had collected data from maps into one, larger, spreadsheet of doom. (Note: all of these sources were gathered since 1.0, which substantially changed the mapping system.)

• My data - link
• /u/dansenMONSEN - link
• /u/jddogg - link
• /u/IcyRespawn - link
• /u/Shadowclaimer- link
• /u/pyrodan2 - link
• /u/Skyl3lazer - link

This gave a total 4076 map runs, covering a wide range of map levels and map quantity.

From this data, I can take a look at the basics of map drops.

Map Level

One finding is painfully apparent for anyone that runs maps in PoE: it gets harder to progress your map pool on higher level maps.  For every increase in map level, the average level of maps dropped increases by 0.71. For 66 maps, the average drop is an increase in map levels (66.5).  By level 70, the average drop is only 69.1, and by 78 maps the average drop is down to 74.6. Maintaining a high level map pool gets harder with each level.

Map Quantity
Declining average drop level makes getting high quantity critically important for maps (since higher quantity means more maps, which increases the odds of getting equal-or-higher level maps).

On average, I estimate that every 100% quantity increases the number of maps dropped by 0.85 maps. (Results from a negative binomial regression - full table after the break.) This, in turn, improves the chance of getting a higher level map.  The same 100% quantity increases the chance of getting a higher level map by around 20%.

Pack Size / Magic Monsters / Rare Monsters
But what about the desirable mods (pack size, magic monsters, rare monsters) - what effect do they have on map drops?

• Pack Size: aids in getting both more and higher level maps. However, the data suggests that pack size isn't as game-breaking as others suggest. From the models, every 2% pack size is worth about 1% quantity. So a 20% pack size roll is worth only about 10% quantity - less than the other available suffixes that could roll on a map.
• Magic / Rare monsters: have no effect on the number of maps (or, whatever effect is there is lost to noise / too small to estimate). Rare monsters increases the likelihood of getting a higher level map (it is likely that magic monsters does too, just that the effect is fairly small).  Again, the mod is about half as valuable as quantity.  So a low rare monsters roll is equivalent to about 13% quantity.

Thoughts
These results are... surprising.

While I've always been told to roll for the desirable affixes, these results suggest that they may be overrated. Of course, there is one caveat: some maps - like tunnel - don't have much room for large packs to spawn, diminishing the effect of the mod. I'll try and gather more data to evaluate this claim.

I'm really surprised that the magic monsters mod didn't show any statistical significance, no matter how I looked at the data. I'm thinking this might have to do with selection effects. The problem is how highly the mod is rated: if players immediately run a map with +magic monsters regardless of the other mods, this would result in +magic monsters being associated with worse maps and fewer drops. I'll try some quasi-experimental methods out to get around this problem.

I'm also surprised that 1% quantity doesn't equate to another 1% chance to get a map... this makes me think something else may be going on with the models, so I'll take another pass at the topic soon.

So very excited to have data to play with!




For the stats nerds:  full table after the break.

[GW2] Server Transfers

I've been tracking players on the GW2 achievement leaderboards.

The boards enable me to track when a player transfers off of a server: players that disappear from the boards - but have enough points to appear on them - have likely jumped ship.  (Or, alternatively, been banned: I'm not sure what happens with the boards then.)

I was able to track players on 23 EU servers for one month [February 20th to March 23rd]. In that month, I observed 1456 players that likely transferred servers. That makes a 0.2% chance that a top 1000 player server transfers on any given day - or - a 6.6% chance that a top 1000 player changes servers over the course of a month.

Caveat: some of my data collection was corrupted - from both my ISP (TimeWarner <glare>) and the ArenaNet website (the website got angst-y partway through my data collection). So I'm missing some servers. Take the resulting numbers with salt.

Transfers are highly clustered: on most days on most servers, at most one player out of the top 1000 transfers (65% of all server-days look like this). However, on some servers, on some days, large clusters of players leave the server (record observed: 57 of the top 1000 players left Underworld on March 20th).

Transfers also differ by day: on March 7th, ArenaNet announced a reduction in the cost of server transfers starting on March 18th.  This resulted in a considerable rise in the number of transfers. Before March 18th, an average of 1.9 top 1000 players transferred per server each day. After March 19, that number jumped to 4.7.

	Pre March 18	Post March 18
Gold	1.1	1.4
Silver	2.0	9.2
Bronze	2.4	4.8

But while these transfers were designed to encourage players to move to lower-tiered servers, they had something of a bandwagon effect.  After the update, very few top 1000 players left gold-tier servers. Most of the movement came from the Silver / Bronze tiers.

Thoughts
I'm actually surprised at just how common server transfers are. Post-March 18, nearly 10 of the top 1000 players in Silver tier on every server changed servers each day. That is a remarkable change in the population of each server.

I'm also not surprised that most transfers originate on Silver / Bronze servers.  WvW match-ups on these servers can be frustrating, and it makes sense that players would seek a stronger world environment.

[GW2] Server Populations and Achievement Scores

After observing large differences in Marionette success rates across servers, I've become interested in understanding a bit more about server populations.

So the achievement point leaderboards were of particular interest to me.

Players earn achievements by completing milestones, participating in living story events, and completing dailies. They're a great measure of player activity, since you get achievement points for just about everything. Even better, ArenaNet publicly tracks achievement points for the top 1000 players on each server.

Here, I've plotted the achievement points of the top 1000 players on each server.  The servers are organized by the average achievement point score of these players (highest top left to lowest bottom right), and colored by their WvW season 1 tier (gold / silver / bronze).

There are vast differences between servers.

As a snapshot, the leaderboards are interesting.  But tracked over time, they contain much more information. For this, I collected data from the leaderboards during the Edge of the Mists update. This gives three measures of server activity:  (1) transfers, (2) inactive players, and (3) points gained among the active players.

Server Transfers: Jumping Ship
Server transfers show up on the leaderboards in two ways. A player could appear on one server, and later on another. So I can track accounts that move between servers.

Other players disappeared from the leaderboards. So, for example, if a player on Eredon Terrace had over 4217 points at the start of the Edge of the Mists update, they should appear on that server's top 1000 list unless they transferred off the server.

Overall, 1.6% of players listed on the leaderboards transferred servers. Given the small number of transfers from each server (and the reasonable expectation that some of the 'large' moves are due to guilds) it is hard to make a general pattern from the server transfer data, but I'll look at it again in future updates.

However, of the transfers I observed, 84% are to servers with higher average achievement point scores than the starting server.

Inactive Players
Overall, 17% of players on the server top 1000 didn't earn any achievement points during the Edge of the Mists update, and therefore likely didn't play GW2 during the update. Here are the percent of inactive players by server:

Inactive players are more common in lower-tiered servers, with the Bronze tier averaging over 23% of players inactive, compared to an average 12% for the gold servers.

Points Gained by Server
During the two-week update, active players on the leaderboards gained an average of 170 points, around 12 points per day. Pretty impressive, considering that the daily requires only 5 points. (The average top 1000 player earned about one-third of the total possible points to be earned from living story / daily content.)

Across all servers, players with more achievement points earned more points than those with fewer points (rich get richer faster).

Several Thoughts
By now, it is clear that there are large differences in server populations and activity. It is simply a myth that there are 'pve servers' that just happen to do poorly in WvW. Players tend to transfer to higher-tiered servers. Those servers have fewer inactive players. And the players on higher tiered servers tend to be more active.

Time to think about a server transfer myself...

Sidenote: the results presented here don't change under more complicated statistical models (which I ran, but excluded from the presentation because this is a long post already). Though, I did have to run a nonparametric model due to the interesting structure underlying the data.

[GW2] Professional Voting

Today, Guild Wars 2 extended their Collaborative Development Initiative (CDI) to ask players to post their feedback on the ranger profession.  (Note: 'professions' are what most MMOs would classes).

Ranger was chosen in this thread, where the developers asked players to list 3 professions (in order of priority) they would like to see discussed first.  But isn't choosing a profession based on posting in a thread open for 24 hours on the forum problematic? Particularly active forum users would be likely to overwhelm the thread. And its reasonable to expect that posters would be biased toward their own class, instead of looking out for game balance as a whole.

So how reasonable was the thread as a method of getting feedback?

The results:

Ranger was the most mentioned profession, followed by elementalist and then necromancer to round out the top three. Rangers are particularly weak in GW2's pve, and one playstyle (bearbow) is consistently viewed with scorn.

Breaking down the votes by order of priority indicates that not only was ranger the overall top choice, it was most frequently listed as priority #1 by voters. Most of the ordering between professions remains the same.

But are these votes contaminated by bias?

Prior Posting Activity

One objection to this poll might be that not only are forums generally unrepresentative of player opinions, but frequent forum posters might be likely to overwhelm any thread. Hardcore and casual players might be expected to disagree on which professions need the most attention.

While generally valid objections, they don't apply particularly well to this poll.

For nearly 20% of voters, voting in this thread was their first post on the forums. For others, posting is a pretty frequent activity. Voters had an average 205 posts before the thread (with a maximum of over 4600 posts!). 

Curiously, however, prior posting activity doesn't change the results. Both first-time and prior posters overwhelmingly agree on their preference order, favoring rangers, elementalists and necromancers. (Though mesmers do a little less well among first-time posters).

Profession(al) Bias

Another way to look at the votes is to examine what other forums voters are active on. Players might be expected to be biased toward voting for their own profession, which could be reflected by their posting in profession-specific subforums.

Here, I took a look at the past 50 forum posts of each voter (excluding first-time posters), and recorded whether they'd posted in one of the profession-specific subforums.  I compared this against their listed vote preferences.

In general, the votes are only slightly biased toward forums voters are active in.  For example, 30% of those who ranked elementalist as priority #2 had posted in the elementalist subforums, compared to only 15% of all voters. But this pattern doesn't hold generally. Those who picked engineer, mesmer, thief or warrior were no more likely to be active in those professions' subforums. So, in general, forum posting activity is at best weakly related to vote choice.

The exception - as ever - is rangers.  While only 5% of voters had posted in the ranger forums previously, 45% of those who listed ranger as their top pick had posted in those forums. Among top choices, ranger is the only profession that shows such a strong pattern. However, even excluding these voters, ranger would still be the top pick.

Thoughts

Clearly, a 24-hour poll isn't the best way to generate clear feedback on which profession requires the most attention.  In this case, however, the ranger profession is so overwhelmingly in need of attention that the effect of these (otherwise valid) concerns is overwhelmed.

Fingers crossed for some pretty excellent fixes.

Disclaimer: I have a ranger at 80, but I only use it to loot jumping puzzle chests since returning to GW2.

[GW2] Back! And Marionette Success Rates

Hello Tyria!

We're back with a blast. After over a year without MMOs, I decided to take a quick check at what was new in GW2.

... and quickly wandered into another data-driven gaming argument.

In the Origins of Madness update, ArenaNet introduced two world raid bosses: The Twisted Marionette and a three headed Great Jungle Wurm. While the encounters have difficult mechanics, they're part of the open world - you simply cannot choose the other players you'll fight alongside. Since each boss has mechanics that make it possible for a single player to cause the whole server to fail the encounter, players have taken to complaining about the mechanics of both fights on the forums (shocker!).

I'm particularly interested in the Marionette because of this comment, made by Josh Foreman (Environment Design for ANet), in responding to complaints about the fight's difficulty:

if after a week the Marionette is still only being beaten 1 out of 10 times, then I would say we may have tuned it a bit too difficult.

That was 14 days ago. A week after he made that comment, I started tracking the success of the Marionette event via the GW2 API. The Events section of the api tracks the status of each event on every server, noting its outcome (success/fail). This gives me six days of data on Marionette kill attempts. Since the Marionette fight occurs once every two hours, that is data on just over 70 Marionette attempts.

The overall Marionette success rate for the past week is 6.6%.

• on North American servers the success rate is 6.1%.
• on European servers the success rate is 7.0%.

Server Population
One issue with the Marionette is that the fight takes a large number of players to be successful. The fight is divided into five lanes, which are periodically divided further, into five platforms. Most guides for the fight mention 20-25 players per lane (100-125 players for a successful attempt), though a coordinated group could get by with less. So active, well-populated, coordinated servers should do better on this boss.

There are vast differences between servers.  The top EU server (Desolation) succeeds on 63% of its attempts, and the top NA server isn't far behind (Blackgate - 59%). On the other hand, I have never observed a successful Marionette attempt on half of all servers (25 of 51 servers).

As might be expected, servers with a higher WvW rank are much more successful at the Marionette. While top 5 WvW servers average a greater than 20% success rate, all other servers have a less than 5% sucess rate at the event.

Prime Time
In addition to vast differences between servers, it matters what time the event is being attempted. NA primetime (starting at 7 EST) has a much higher success rate than afternoons (students getting off school?).

With this in mind, I have two caveats:

1. GuildWars 2 has an overflow system, where if too many players are in a zone, an overflow server is created for any additional players that arrive (or several overflows, if need be). I don't have any information about what happens on these servers (or how many there are).
2. It could be that the GW2:Events API is not accurate. Completely unsuccessful servers are pretty shocking, so much so that I doubt the source. Then again, I do the Marionette at least once per day, and the outcome of each of those attempts is accurate, when I check against the data. 

Further Questions
So what does this mean for events like the Marionette?

Feedback based on a particular server (or time) isn't really representative of the experience of the event as a whole.  Someone trying the event at 7 or 9 pm EST on Blackgate is virtually guaranteed a successful run. But there are 25 servers that I've never seen succeed at the event.

On the other hand, vast differences between servers (and times) suggests that the fight isn't balanced across the wide range of situations GW2 players are likely to encounter. Hopefully, future events will be open to a wider range of the community, instead of being limited / exclusive content.

Time willing, I'll take another look at the Marionette sometime this week, and see if I can break this apart even more.

[POE] Hardcore

Time for some ARPG shenanigans.

I've been playing Path of Exile.

Its probably the best game I've picked up in a years. I could write pages and pages on why I love this game. The races. The skill trees. The complexity. But this blog is about being a video game data nerd, not persuading others to play awesome games. So instead: a video.

Great, now that everyone is playing, here comes the data.

Like most ARPGs, Path of Exile has a hardcore league. A single death kicks you out of the league, and into default. Since staying alive gets comically difficult, the hardcore league is super competitive. Players speculate on the best builds, watch the top characters, and get global announcements when a top 25 player dies. They debate whether duelists are viable, marauders overpowered, witches death-prone...

Of course, the best way to answer these arguments would be to track the progress of characters through hardcore, recording when they die, how quickly they gain experience, and whether class choice matters.

Fortunately, the PoE website lists the top 15000 characters in the hardcore league. By periodically recording this information, I can get a timeline of characters, as the league progresses.

Classy Breakdowns
Here is the breakdown of default and hardcore characters, by class:

While witches are the most second-most popular option on default, they're the second-least-popular class in the hardcore league. Hardcore is dominated by Templars, Marauders, and Rangers. Duelists, who have the weakest starting position, are the least popular class in both leagues. (They've recently been buffed, though.)

But why are there differences between the two leagues?  Is hardcore just harder for some classes? Are witches just too squishy? Or do players just prefer some classes over others?

Changing Classes
Minimally, if hardcore was harder for some classes, you'd expect to see dramatic changes in the proportions of each class over time.  So, if witches were really weak, at the start of the league there would be a ton of witches, they'd die, and the result would be fewer witches over time.

Here is the distribution of classes over the past month.

Over the past month, there have been small changes in the class distribution of players. Witches have increased their share, while duelists have steadily declined. Marauders have fallen in popularity, only to rise again. Rangers and templars have remained remarkably stable at the top.

Overall, though, class proportions have remained fairly stable. This suggests that whatever imbalances exist, they are small, and being amplified via player choices.

The Final Death
Another way to look at difficulty across classes is via deaths.  If POE is harder on some classes, then those classes should die more. In particular, if a class is harder in the mid or late-game, then there should be relatively more deaths for those classes at the higher levels.

Here, I'm looking at the average chance a character dies for every 15 minutes that they're online, by level and class.

Overall, gameplay becomes less lethal as characters level up. Characters in the 60s are less likely to die (per minutes played) than characters in the 40s.

While *slight* differences exist between the classes (witches die more frequently than rangers, for example), they're relatively small, and not large enough to explain overall class divisions on their own.

There are, of course, other ways to look at differences between classes: experience gains, survival analysis, etc... but for the moment, I need to keep leveling in merciless difficulty. =D

[GW2] Crafting Cost

Crafting Costs: Even Scarier Behemoths

EDIT:  Update 1/2 - instead of using GW2wiz, take a look at ErrantQuest for crafting guides.  Much cheaper.  For cooking, Xanthic has a dynamic cooking guide (always chooses the cheapest route).

One of my favorite features in GW2 is the ability to level a new alt entirely via crafting.  Since each crafting discipline gives around 10 levels of experience, it is possible to level a character to 80 by advancing through each crafting profession. Leveling a new alt takes hours, not days.

In the early days of the game, this was fairly inexpensive. Alternating between crafting, story quests, and hunting down skill points, I was able to level 4 alts to level 80 for about a gold per crafting discipline. Recently, I've become interested in leveling another character via crafting, but wondered how crafting costs have changed since the early weeks.

To track the cost of crafting, I used crafting shopping lists from GW2 wiz and price data from GW2 spidy.

Here is the cost of each crafting discipline over the past few weeks:

Armorsmith, leatherworking, and tailoring have all dramatically increased in price over the past few weeks (and are much more expensive than I'd remembered!).  Breaking down each craft by materials cost, the culprit appears to be the fine (blue-quality) crafting materials:

While gatherable components (metal, leather, cloth) have held steady in cost, fine materials (scales, totems, blood,...) have risen at an alarming rate.

The rise in crafting costs is distributed fairly evenly across all tiers of materials, though Tier 5 goods have increased in price more rapidly than Tiers 1-4.

For the curious, over the past week, here are the average prices of leveling each crafting discipline:

armorsmith   artificer    huntsman     jeweler 
       7.80        8.69        9.30        7.59 
    leather      tailor weaponsmith 
       7.23        8.65       10.06 

This leaves cooking as the cheapest craft to level (estimated at 1g!), followed by leatherworking, jewelry  and armorsmithing.

Personally, it looks like I'll be leveling any new alts the old-fashioned way!

[GW2] Data Mining and Magic Find

I've been busy data mining in GW2.

This time, I was curious whether magic find affected the chance of getting gems from mining nodes.*

I've seen plenty of questions about magic find and gathering. I've seen speculation on the topic. I've seen assertions given (many assertions). But the wiki isn't much help. And I haven't seen evidence.

So I set about mining.

Every day for the past two weeks, I took each* of my level 80 characters around Tyria gathering orichalcum and any mithril along the way. All characters used orichalcum mining picks and had the guild gathering bonus. One character had 101% MF, the others only had the guild magic find bonus (10%). By the end, I gathered 1027 mithril and 1483 orichalcum.**

Data can be found here.

All told, I didn't see a difference in gems between different levels of magic find.

Node        MF     N    Gem Chance
Mithril     10%:  696   9.5% (1.1)
Mithril    101%:  331   9.4% (1.6)
Orichalcum  10%: 1065  12.0% (1.0)
Orichalcum 101%:  418  12.7% (1.6)

While orichalcum nodes have a higher chance of yielding gems than mithril nodes, I see no difference in the chance of getting gems between 10% MF and 101% MF. (For the stats-nerds, checked with a probit.)

Of course... this is a relatively small sample. So how likely is it that magic find affects the chance of getting gems from mining, but I just didn't notice it? (In stats-language, what is the power of the test, given expectations about the size of the effect being studied.)

To check this, I simulated the results of gathering between 70 to 4200 mining nodes (increments of 70), assuming that magic find works multiplicatively. For each simulation, I checked whether I found that magic find had an effect. (I also included differences between node types, and recorded those.)

From the plot, its clear that after around 1500 mining attempts I'm nearly certain to find an effect from magic find, if one exists. At about 2500 mining attempts (the data I've collected), assuming magic find works, I'd have less than a 1% chance of observing no effect.

TLDR: magic find doesn't affect the chance of getting gems while mining.

But, of course, I'll keep gathering data, and check back in if any results listed here change. Because science is the joy of revision.

Related links:
On this blog: MF doesn't affect bags.
From reddit: MF doesn't affect salvaging.
Dev: MF only works on kills.

* Horray for leveling via crafting. I love going from 10 to 70 in like 2 hours.
** Just to be safe, this data excludes any ruined ore / gems from ruined ore. Which is why neither total is divisible by 3.

[GW2] Small Salvaging Samples

After paying an absurd amount of attention to salvaging rags, something of a consensus has emerged from the official forums, gw2guru, youtube, and reddit on how salvaging kits work. (I'm calling it basic chance theory after Zoodokoo.)

As noted in the reddit thread, one way to account for the minuscule differences in gossamer returns between salvage kits is that instead of increasing the amount of gossamer, the kit modifiers multiplicatively increase the chance for getting gossamer.

So, supposing that gossamer has a base 11% chance of being salvaged from a rag, the chance of getting gossamer with each kit is:
Crude:      11% * 1.00 = 11.00%
Basic:      11% * 1.10 = 12.10%
Fine:       11% * 1.15 = 12.65%
Journeyman: 11% * 1.20 = 13.20%
Master:     11% * 1.25 = 13.75%

As I noted in the reddit thread, this would be entirely consistent with the data I've collected so far.

While it is great to have a theory to test (other than just "everything is consistent with a bug") there are two things that are troubling about this theory.

If the theory is correct, then most kits aren't worthwhile (NeckNeckNeck's observation).

Supposing gossamer is worth 3.5 silver (350 copper) and silk is worth around 20 copper. So getting 1 gossamer is a gain of about 320 copper. A 1% increase in the chance to salvage gossamer is worth about one hundredth that, or about 3.2 copper. Getting a 1% increase in the chance of getting gossamer compared to crude kits must cost less than 3.2 copper to be worthwhile.

Therefore, assuming the base chance of gossamer is 11%, it is only worthwhile to upgrade to basic kits (for rags).

The bigger trouble is that statistically verifying the theory is going to take absurd amounts of data. The effect sizes here are tiny. As I discussed elsewhere, statistically verifying a tiny effect is tricky business.

Here, I've simulated the results of 1000 different attempts at salvaging 2000 rags, split evenly between crude and mystic kits. For each attempt, I recorded the size of the effect that was estimated and whether it was statistically significant. Sadly, I only find a statistically significant effect 33% of the time. And when I do find an effect, its overestimated.

What does it take to get it right?

Looking at different combinations of kits and sample sizes, its only after 10k salvages, or using absurd amounts of black lion kits, that results can be found reliably and accurately.

I don't even want to think about how much gold that represents. (Dear ANet: can we get a Tyrian NSF going?)

I'm taking a different approach. I've started to collect data on everything I salvage. I'm hoping that I can see a clear pattern where `rare' means 'higher tier of materials'. Hopefully, I'll also find a kind of item with a fairly high chance of being salvaged into the higher tier. This will make for a larger difference between the kits, and therefore something easier to estimate.

Wish me luck!

[GW2] Salvaging Scraps: Which Kit to Use?

Apparently, science takes quite a few salvages (6k and counting...).

After opening 1000 bags to check whether they're affected by diminishing returns / magic find (spoiler: they aren't), I started wondering about salvaging. In particular, Lord Signis found evidence that "items may have their own rare salvage modifier value other than 1. (ie .52 from the item and .25 from the salvage kit means salvaging will yield rare materials 13% of the time) Further testing is needed."

Challenge accepted.

I bought 1500 rags from the trading post. Because I apparently have no desire to wear exotics. Ever.

To salvage the rags, I purchased 500 Crude, Basic, and Fine salvage kits. I recorded the number of silk and gossamer from each salvage, and appended this to Signis's data on 500 salvages with Mystic salvage kits.

For reference: Crude kits have no listed chance of giving rare items, a 10% chance for Basic kits, 15% for Fine kits, and 25% for Mystic kits.

Data can be found here.

As before, I see no effect from the anti-bot code. Salvage away without diminishing returns.

The expected number of silk / gossamer remains the same, whether on the first 100 salvages, or the last 100 salvages.

Visually, here is the average number of silk and gossamer from each salvage, by number of items salvaged:

The average gossamer yield from each kit was:
Crude: 8.6% (1.3%)
Basic: 13.0% (1.5%)
Fine: 12.8% (1.5%)
Mystic: 13.0% (1.5%)

(Standard errors in parenthesis.)

So, while crude kits had an 8.6% chance of gossamer, all other kits had a 13% chance.

While it is always possible that I was exceptionally unlucky with crude kits... it is exceptionally unlikely. The probability that I'd observe gossamer 8.6% of the time, when the true chance was 13%... is about 2%. So possible. But not likely.

This contrasts with the claims in this video of "100% certainty" that crude and mystic have identical salvage rates. Hopefully he'll be willing to share the data, and I can take a look at what is going on.

The crude kits leave me a little baffled. Without them, I'd say that gossamer had a 13% chance of being salvaged from a rag and that kits don't matter. With them... kits matter (at least on the choice of crude vs non-crude). But the rate doesn't match what is listed on the item. Hopefully I can salvage these results over the weekend. I've put in more orders for rags to check whether my crude data is just an unlucky draw, and started to collect other salvageable items to check those too.

TLDR: Salvaging rags with Basic, Fine and Mystic kits has a 13% chance of giving gossamer. Crude kits have an 8% chance. More testing required.

Update: I've done another 500 salvages with crude kits... still at about 8.7%. Now going to check other materials. I've decided to make it clearer that the I'm presenting standard errors of the means here, because otherwise I find +/- ambiguous / unfounded.

[GW2] Opening Bags: the Anti-Bot Code and MF

The Guild Wars 2 subreddit has been invaded by science, and I love it.

Recording the results of 500 rag salvages, Lord Signis found that salvaging items isn't affected by the anti-bot code. Marko did a similar setup, and found that salvaging isn't influenced by magic find.

This got me thinking: what about loot bags?

TL;DR: The loot obtained from opening bags isn't influenced by either magic find or the anti-bot farming code.

Setup:
In order to obtain a desirable sample size, I grabbed 1000 Bags of Pinched Goods from the Trading Post.

Doing this has one significant drawback. I'm essentially assuming that loot is determined at the time I open the bag, not at the time the bag is generated. I'm really hopeful that ANet decided to save on storage space. [Further testing required... its something I'm working on.]

On the upside, It felt like Asuran Christmas. With fewer explosions.

Controls:

• The first set of 500 bags was opened with 0% MF after a day of opening no bags.
• The second set of 500 bags was opened with 111% MF 24 hours later. (MF sources: 30 on jewelry, 53 on runes, 18 on gear, 10 from guild bonus)
• I stood in the same place without moving on each run.
• The bags were opened at 7pm EST 9/26 and 7pm EST 9/27.

Results:

Raw data can be accessed on this spreadsheet.

Each bag has a chance to contain a gathering tool (ex: axes, mining picks), a cooking item (butter, blueberries, blue meats), white crafting components (cloth/leather), or blue crafting components (claws, totems, etc).  In what follows, I'll break those categories down into gathering tools, white crafting materials, and blue crafting materials. I'll note whether the items are cooking components or not.

The Anti-bot Code.

The number of items contained within each bag does not decrease as more bags are opened.

For example, taking a look at the blue line in the center panel, across all 500 bags, if the bag contained butter, it averaged just under 4 butter.  This was as true for the first 100 bags as it was the last 500 bags. The order the bag was opened didn't change the amount of loot inside. [For the more stats-minded, the order the bags were opened was not statistically significant across several specifications of negative binomial regressions.]

The chance a bag contains a white vs blue item does not change as more bags are opened. Here, following Lord Signis, I'm taking a look at the change in the probability that an item contains a blue crafting material as the number of bags opens increases. While I had a run of good luck at the beginning, and a spot of rough luck around 250 bags, its fairly clear that the probability of a blue item hovers around 50%.

[Again for the more stats-minded, I confirmed this with a few specifications of probits of blue vs white on order with various controls.]

Based on the the first set of 500 bags, I didn't find any evidence that the anti-bot code changes the rewards from bag opening.

Magic find results. Here, I'm comparing the results of the first 500 bags (discussed above) to a second set of 500 bags I opened under identical conditions, 24 hours later, this time with 111% magic find.

Cooking Materials	0% Magic Find	111% Magic Find
Butter	317	324
Blue	16	18
White	24	23
Non-Cooking Materials	0% Magic Find	111% Magic Find
Blue	224	231
White	259	254

Not only is the difference between the two runs absurdly small, the difference isn't statistically significant. I only obtained seven additional non-crafting blues with 111% magic find - a 3% difference between the two samples, which is well within the margin of error (approximately +/- 4%).

Again, the punchline: the quality and amount of loot from bags isn't affected by either magic find or the anti-bot farming code.

Update: dev post - MF only works on kills.

The Great Server Migration

Thought I'd share a visualization of how the SWTOR server migrations have shaped realm populations. Here I'm just taking a look at realm status (1 for light, 2 for medium) over time, by region and type. Short story: origin servers are ghost towns, destination servers are hopping (but that much was clear already).*

Slightly cooler story: even before the migrations, as far back as April, destination servers had higher populations than the other servers in their zone / type. There is one exception. Until just days before the migration, The Swiftsure averaged a higher population than The Bastion (which ultimately became the destination server for West Coast PvP servers). Makes me wonder why Bastion got destination status. Rounding errors between status and true population numbers? Server hardware?

I also went ahead and re-collected data on posting volume on the PvP forums. Activity - in both posts and views - is definitely down from earlier in the year. Curiously, while 1.2 saw a remarkable increase in activity on the pvp forums, the 1.3 release had almost no effect. It looks like a focus on the server migration drew posters away from the forums around the roll-out of that feature.**



*The gap in data from the start of April comes from my failure to backup a week's worth of data and then getting a drive failure.
**The white band around May 5th is a day of data I'm missing.

Voice of the Player: the PVP Forums

Mos Ila: Honorable Mention, Hives of Scum and Villainy

The PVP forums may be the most wretched hive of scum and villainy in the galaxy, but there are always a few holdout posts that I find helpful, or even inspiring.

What about the din, though?  Can something be learned from the countless threads of complaints, anecdotes turned into truths, and general hatefest?

I ran a program to download the approximately 32,000 threads posted to the PVP forums, taking the first page only.

Most threads are only active for a day before they no longer get replies (75%).  After a week of activity, only 5% of threads remain active. The median post gets around 5 replies and 240 views. There are a handful of notable outliers - including one post with around 330k views (a dev thread from Jan 18 that gave an update to Ilum's pvp).

Just from this activity, its immediately clear that 1.2 brought a dramatic increase in forum post volume. With the general renewed excitement over SWTOR with 1.2, this is a good indication that the volume of posts matches interest in the game. Posting is something that complements play time, not opposes it.

Class Warfare: calling down the nerfbat
What about the ever present cries for nerfs on the forums? Is the rise-to-power of sent/marauder reflected in forum posts? The stunlocking dominance of ops?

For this, I'm not going to do anything super fancy, just a simple search through the first 10 posts in each thread for mentions of each class (accounting for things like spelling mistakes, or clear references to a class via a popular spec, like 'concealment' matching to operative, or 'galactic overlord of awesomeness' to sorcerer.) I'll also combine the Empire and Republic advanced class mirrors, and reference them by their Empire names.

Looking at the percentage of threads that mention an advanced class, I'm actually astounded just how well that worked. The fights between operatives and sorcs, followed by the overwhelming numbers of sorcerers, until the rise of marauders with 1.2... its all reflected in the volume of mentions of each class on the PVP forums. In contrast: powertechs, snipers, and juggs are all but invisible.  Assassins have made a respectable rise, one that I'd expect to continue, given their impressive on-demand cooldowns and burst.

From this, I'd say forum chatter is a reasonable way to capture community awareness of issues.

Topical Applications
Next up: I sampled some of the other issues that players raised for discussion on the forums, looking mostly at pvp zones, exploits, and some quality of life issues. Again, for each category, I'm looking at word counts, accounting for things like misspellings and including related words.

The first plot nicely captures the rising attention given to Ilum, and its eventual irrelevance. Alderaan - the least innovative of the warzones - also gets the least attention.

Regarding bugs and exploits, mentions of bots and hacks are relatively infrequent - and steadily decreasing over time. Hopefully this is a reflection of the developers getting a better handle on the situation. Lag and bugs, however, remain persistent issues.

I'm glad to see that despite all the attention given to 'unsubbing' threads... they're rare. Like, people spend more time talking about powertechs rare. Premades and stunning, though - they're frequent issues.

But forum goers are weird!
For ages, developers have noted that players who post to the forums are 'different' and don't really represent MMO players in general. I think that makes intuitive sense.

But that only goes so far. Again I'll look at class balance. But this time, I'll divide posters into four groups, based on the number of posts each forum-goer made in my sample.

Post Groups	1 post	2-10 posts	11-100 posts	101 or more
Individuals	10,802	13,009	3,220	218
Posts Made	10,802	51,010	88,336	40,286
% of posts	5.7%	26.8%	43.9%	21.2%

Even though individuals who made 101+ posts are less than 1% of the population, they make over 20% of the posts in my data.

But those who post often are no different from those who post infrequently. Whether looking at those who make one post, or more than 101, the trend is still there. Each group is currently discussing marauders most frequently, was talking about sorcs before 1.2, and largely ignores the powertechs. It is definitely the case that the trend is smaller in the one-post group, but it is still there. 

Class balance concerns are relatively even across each group.

Forums: Windows into MMOs.

I'm pretty impressed with just how rich a source of information the forums can be (particularly given that this took... all of a couple hours to run, once I'd downloaded the data). 

But this was just what struck me while playing with the data. If you've got a pvp-forums related question, hit it up in the comments (or email) and I'll see what I can find.

Beating a sliced horse

Slicing is back to normal!

The most recent patch notes (for 1.2.2) note:

Premium Slicing lockboxes now yield the correct amount of credits.

But there is no way I'm just going to take that note at face value. And neither are the forums - several threads are dedicated to discussing whether the patch corrected the issue. To help sort out the issue, I collected data on another 400 grade 5 and 6 missions since 1.2.2, and reran my earlier analysis.

As shown in the first plot, the average earnings of all missions have reverted to their pre-1.2 levels. Premium (green) lockboxes now properly earn more than their white counterparts:

Note: this doesn't mean that every single time a lockbox comes back, it'll have a profit. Most of the grade 5 rich missions I run are losses - they're only profitable because crits return blue boxes which are extremely lucrative, and more than make up the loss. In general, any particular slicing mission can be a loss: several missions in a row can lose credits. Its only in the long run that slicing earns credits.

[EDIT:  realized that some folks are curious about 340 mission discoveries post 1-2.  I haven't seen much, if anything, in the way of differences from my earlier post on the topic.]

Now hopefully they leave slicing missions alone for a while...

That's no thread...

This week's feedback request is definitely a poll in disguise:

"What planet in the Star Wars universe do you want to see in Star Wars: The Old Republic that is not currently in the game?" (source)

Hmf... I want to see the breakdown of responses too!

I collected all the words used on the first 67 pages in response to the thread, and searched them for the just over 3600 planetary entries in the Star Wars universe listed on Wookiepedia. This gave approximately 1700 recognizable suggestions for planets from 670 different posters.*

Voting early and often, huh?

Here are the votes, as either a count of planetary mentions or weighted by the number of planets listed by a poster (ie, mentioning two planets gives each half a vote, three planets gives each a third... etc). I've also dropped planets that got less than three mentions (another 71 planets).

Be sure to click to enlarge!

I love that Kashyyyk is at the top of the list - the vertical element could really add a ton of fun to SWTOR questing. I'd love to play with a charge or updated MGGS to move through the trees. A jetpack to ease the concerns of heights and pvp would be pretty fun, too.

Old KOTOR favorites (?) seem to make it to the top of the list - like Dantooine or the aquatic planet Manaan. I have to confess, I'm not a huge fan of aquatic combat in MMOs: that also disqualifies the highly-ranked Naboo from selection. Ditto with Mon Calamari... which might win the contest for the most comic name on the list.

I'm a bit shocked that Dagobah didn't make it higher than #11. Training sequence daily quests? Lift a drowned fighter from the muck. Now jog through the bog while carrying a muppet. The cave alone would be a phenomenal flashpoint / daily quest.

Though, I'm a bit surprised that the devs polled the audience on this one. The SWTOR universe has so much awesome in it, I'd figure they'd just run with whatever idea geeked them the most.

*Just a simple search with some corrections for basic spelling mistakes (Kashyyyk has 3 ys) / abbreviations. While its possible to map "wroshyr trees" to Kashyyyk / account for feedback that looks like "meesa not a fan of Gungans"... I'm mostly interested in the infamy of a planet.