Oops, I noindexed the Blog!
File under 'mistakes made and lessons learned'
Many months ago, I set up Umami to track views on my blog. It doesn't track users, just counts page hits and referring sources and the like. The results were encouraging: I do get a fair amount of traffic. Hello! I realized too that about 50% of it is from mobile devices. There goes my theory that I don't need to make it look good on mobile...
Last month I noticed a trend: I was getting many fewer clicks and almost no traffic was coming from Bing or Google (having a lot of .NET content I came up in Bing results a fair amount). This was odd but I attributed it to the holiday season. I was still getting plenty of traffic from Yandex and Baidu.
Well, it happens that the decrease in traffic coincided with my release of StaticBlogroll, a repo/template I made to encourage others to share a public blogroll. It's intended to be deployed with GitHub Pages, and I figured it would be good to include noindex, nofollow by default in the robots.txt for this site, lest others' articles might appear under my domain on Google. Unlikely, but I didn't want the chance.
Well, as it happens, GitHub likes to deploy Pages sites under your domain if you've already registered it. Thus, my blogroll deployed to ian.wold.guru/Blogroll. No problem by my book!
Well, as it also happens, Bing and Google will read every robots.txt file on your site. As it further happens, noindex nofollow in any robots.txt overrides everything else.
Oops.
Two learnings:
- That's why I had fewer hits, and
- Yandex and Baidu are (maybe predictably) not following best practices.
Luckily, Bing and Google have webmaster tools with which you can notify any updates to these files, so the problem should be corrected. I've got no clue if some of my previously-most-popular articles will regain their positions though. I'm particularly happy with my article on E2E encryption in Blazor WASM so I hope that one recovers. Finally, I've updated the default for StaticBlogroll and I've edited my (now only) robots.txt for this site to exclude /Blogroll.