There are a few reasons.

Cost. Unless you already have a PS4 Pro (base PS4 does not do well with VR) or a really decent PC, the cost is insane. The cheapest option is the Oculus Quest, it's one device and everything is included but that's £400. If you want to get into PCVR you’re looking at about £1000 to get a fair setup.

Until recently the games have been… limited, not bad. Bone works, Walking Dead: Saints and Sinners, and Half-Life. Are the first wave of games that feel like both “proper” games and that they wouldn’t be as good without VR.

The hardware is constantly changing and hasn’t really been standardized yet, though we are getting there. Some headsets track the controllers via cameras, others use external sensors that you have to do some DIY with. This makes joining VR more confusing than it needs to be.

It’s isolating. I have three kids aged 4–8, it simply isn’t possible for me to play a VR game when I’m alone with them but I can easily play a “normal” game and still keep an eye on them. It’s difficult to share the experience with others in the room.

Space. You can’t really play it at a desk, you can’t really use it in a room with people in (unless you have enough space) and that’s just the stationary games, room-scale is basically impossible if you own a coffee table. I’ve taken to using my Oculus Quest in the garden (assuming the sun isn’t too bright) because my wife’s current layouts don’t really give me space (I used to have the whole ground floor mapped, that was cool).

Discomfort. Motion sickness is still an issue, it's gotten a lot better but some people just don't react well to it, imagine spending all that money and discovering it makes you ill? The headsets themselves aren’t comfortable for long play. Remember 3D TVs died because people didn’t want to wear light plastic glasses to play games or watch films, strapping a screen to your face has the same problem.

Not all gaming companies are supporting it, without console makers like MS and Nintendo supporting VR or the large publishers it makes VR look very niche. This creates a weird loop where these companies won't jump on VR because it's not popular enough, people won't jump on VR because not enough companies are supporting it, and around we go*. The fact that we have Call of Duty on mobile but not VR is a great example of this.

Things are getting better and thanks to recent improvements and big games VR sales are improving, hopefully, more improvements and games will help build momentum. *This does have an upside in that small companies are making names for themselves in the VR space where they don’t have to compete with the massive publishers.