Note: Originally published on http://janeandrobot.com

Controlling what content is blocked from being found in search engines is crucial for many websites. Fortunately, the major search engines and other well-behaved robots observe the Robots Exclusion Protocol (REP), which has evolved organically since the early 1990′s to provide a set of controls over what [...]