Our only reliable source of VR adoption data has lost its peg to reality [UPDATED]

From UploadVR:

Recent anomalies in the Steam Hardware Survey’s VR section suggest it has become unreliable.

Companies like Meta, Valve, & HTC don’t reveal hardware sales figures. Historically the Steam Hardware Survey was the the most reliable indicator of PC VR’s adoption. The survey is offered to a random sample of Steam’s userbase each month. If you choose to accept, it uploads your PC specifications and peripherals. Before March 2020 the survey relied on headsets being connected via USB at the time of sampling, but Valve changed it to scan your SteamVR logs from the past month.

I had missed this change when it happened, and I suspect that a lot of other people were also unaware that the methodology had changed. This isn't the first time that Valve have made changes to the VR section of the Survey, though; the VR Survey results are the only ones which are presented with the detailed breakdown first, and only then states the overall value, "Steam users with VR Headsets."

One can only speculate about the motivation for this change in methodology, but there's one obvious potential impact: counting headsets by polling the USB interface for VR devices only counts people who have headsets plugged in when polled, while searching users' SteamVR logs would count everyone who'd used a headset, whether or not their owned a headset, which would almost certainly be a larger number. People with borrowed head-mounted displays would effectively allow that HMD to count twice, for both the owner and the borrower, as long as both had logged VR activity in the last month. Households with multiple steam accounts, but only one VR headset, potentially have the same issue. 

Given that Valve make a VR headset, and thus have a vested material interest in fostering an impression that VR headset ownership is more popular and widespread than could be justified by an actual count of the HMDs in circulation, this change in Survey methodology raises obvious issues of ethics and transparency on the part of Valve. Further complicating these issues is the fact that Valve are not a publicly traded entity, and not directly required to give only accurate information out publicly; their public statements wouldn't count as being directed at shareholders, and their fiduciary responsibilities wouldn't necessarily be violated by tweaking the Survey results to push a specific narrative to their own benefit.

Even setting these ethical issues aside, even if we assume that this change was purely an attempt to better assess the real level of VR adoption on their platform, there are serious issues with the accuracy of the data as well. 

Again, quoting UploadVR:

[T]he data for two months this year contained clearly anomalous figures for ‘Steam users with VR Headsets’. Since the introduction of the SteamVR log scanning method this figure stayed roughly stable around 2%. But May 2022’s data showed a large unexplained jump to 3.24%. June’s data returned to the normal range, but July’s shot back up to 6.67%.

The data just in for August has a sensible value for this figure – 2.60% – but this time it’s the headset share data that’s suspect. 

As well as showing how many Steam users have a headset, the survey also shows the per-headset share. The ‘Other’ section – unidentified headsets or those with less than 0.01% share – usually hovers between 1% and 2%. But in August’s data Other jumped a whopping 13.76%.

This increase seems to be made up from losses in almost every headset listed in the survey, including an 8% drop in Quest 2 users, the platform’s most popular headset. According to the survey data, the only headsets which experienced growth in September were the original Oculus Rift, Pico Neo 2 and the Oculus DK2 from 2014. 

Upload VR go on to state the obvious: "Suffice to say, this almost certainly wasn’t the case – there’s likely something very wrong with this data."

As of this writing, Upload VR had reached out to Valve for comment, but hadn't heard back. Barring some sort of detailed explanation from Valve, though, I think we have to assume that their VR headset numbers as reported in the Survey mean nothing, which means that the only tool we had for independently assessing the growth (or lack thereof) of VR's presence in the market is effectively gone.

Valve, you need to address this. At a minimum, we need to know:

1. Why was this change to data collection methodology was made in the first place? Why was the fact that the number being reported was effectively a completely different statistic not made clear to people relying on the Survey results?
   
2. Why is the SteamVR data you've collected since the change has been so problematic? Are you even able to diagnose the issue?
   
3. What, if anything, can be done or is being done to remediate the data set?
4. Once remediated, when we can expect you to validated those changes by also correcting the data sets for the preceding three months, and to re-release the survey results for those three months? (The current page for the Survey does not give access to results from prior months' Surveys, which is no longer good enough.)
5. If remediation of the current SteamVR log data can't be remediated, so you still also have the USB connected headset number? If so, can you revert to using those numbers instead, not just for August but also for July, June, and May?

One of the reasons that Valve's customers are so loyal is that Valve have largely played straight with them. Unlike the publicly-traded AAA players in the video game industry who will say whatever they can legally get away with in order to drive the hype cycle for their products, Valve has mostly been a hype-free operation, putting out public statements only when they actually have things of interest or value to say, and being direct and honest when doing so.

These unexplained and ineffective changes to their own Hardware and Software survey, though, one of the few public-facing sources of data about their service to which we have access, and on which Valve know we rely, serve to undermine that trust and goodwill. Of particular concern is the fact that these changes were made to the VR portion of the Survey, one of only two Survey sections in which Valve have a product listed. With their objectivity already compromised by their participation in this market segment, the main tool that Valve have available for maintaining trust in their Survey is transparency.

Valve are currently not doing a great job of transparency, either. 

Updated 09/07/2022:

Still no announcement from Valve, but it looks like the correction has happened:

 

Sadly, this was the piece that I least cared about. I don't care nearly as much what the number is, as I do about whether the number can be trusted.  Valve needs to do more work, here.

UPDATED 10/10/2022:

UploadVR actually posted this update a month ago, but it confirms what I'd suspected -- Valve made some changes to the analysis, and re-posted August's numbers. Quoting:

Last week we published an article outlining recent anomalies in the Steam Hardware Survey’s VR section, including large unexplained jumps in the key ‘Steam users with VR Headsets’ figure. We concluded that we’d no longer be using it as a reliable source for our analysis of the VR market.

We had reached out to Valve for an explanation on multiple occasions – hoping for a fix – to no avail. But now Valve has responded, issuing corrected data for the past three months and saying it has fixed “collection and analysis” issues affecting the accuracy & consistency [...]

Here’s the ‘Steam users with VR Headsets’ data before correction, as seen in our previous article:

Now here’s the corrected data we received for May, June, July, and August:

If Valve’s claim of more accurate results going forward holds true, we’ll resume using it for analysis of the VR market. We’ll watch the data closely in the coming months to verify this is the case.

To say that I'm less than completely satisfied with Valve's response here would be putting it mildly, but I guess all we can do for the moment is watch and see.