Spider in the Web soup

  • Web Spiders or Robots are WWW search engines that "crawl" across the Internet and index pages on Web servers, catalog that information and make it available to the Internet for searching.
  • A Robots.txt file is a special text file located in your Web server's root directory that contains restrictions for Web Spiders, telling them where they have permission to search.
  • Web Robots are not required to respect Robots.txt files.
  • To disallow all Web Spiders, specify the following in the Robots.txt file:
    • User-agent: *
    Disallow: /


    More: How to Write a Robots.txt File

    Comments

    Post a Comment

    Popular posts from this blog

    Things Near Me – Find & Learn About Landmarks Nearby

    Image
    Back in 2012, I was planning to build a location-aware app that would tell me about interesting sights & news-worthy facts about places that fall in the way of a train journey. I discovered that Web Dev guru Chris Heilmann ( @codepo8 ) had built a nifty app on a similar theme and shared the JavaScript code under the BSD license.  Things Around You used the browser's geolocation feature to find Wikipedia articles on nearby landmarks via the GeoNames API & list them with a brief description. For a Windows 8 app hackathon, I utilized the Geolocation detection idea to build a little app that I called GeoBuzz & posted it on the Windows 8 App Store. It's been on the back of my mind to extend it with more features. Now that GitHub Copilot is here, there was no need to procrastinate. I had a friendly chat with GitHub Copilot ( Gemini Flash model ). In under an hour, I customized @codepo8's original sample to include a map directly on the web page instead of linking t...
    Read more

    Google Mobilizer Bookmarklet - view just text of web pages

    Image
    [Update: 2024]  -  Google has discontinued the Web Light service and retired the Web Light user agent. Therefore, this bookmarklet doesn't work anymore. [Update: Jan 2018] - The original Google Mobilizer service page isn't available anymore & the term Mobilizer doesn't appear to be used by Google anymore. While the bookmarklet mentioned below still works, it redirects to the googleweblight.com domain & then transcoded . I'll further update this article later. You can use Google Mobilizer application, when you need a mobile-friendly version of webpage. If you are on a low-bandwith Internet connection, Google Mobilizer can be used on a PC or laptop to view just the text (and optionally images). Before you read further, you can see how this page looks with and without images after being processed by Google Mobilizer. While you're there take a good look at the URL if you want to understand the bookmarklet code that I'm going to talk about next. ...
    Read more

    This Week I Learned - Week #9 2025

    Image
    This Week I Learned -  * Message queues (Kafka, RabbitMQ, SQS) solve problems but also introduce new ones (ordering, deduplication, backpressure). Nothing is more abused than Redis as a primary database. Understand trade-offs .  *  Roboflow provides the means to label, train, and deploy computer vision solutions. It offers a Free Plan with 15 credits per month to try it and get started. * Currently 45+ services are available in the AWS Free Tier  to both existing and new AWS customers indefinitely unlike some others that are free for the first 12 months. * Skype Joins Windows Phone in Microsoft's Graveyard of Forgotten Innovations -Founded in 2003, Skype’s cheap audio and video calls quickly disrupted the landline industry in the early 2000s. Microsoft bought Skype in 2011 for $8.5 billion after outbidding Google and Facebook. Microsoft will retire the two-decade-old internet calling service in May 2025. * Dark Patterns & Dirty Tricks in Smartphone Apps - The Di...
    Read more