Spider in the Web soup

  • Web Spiders or Robots are WWW search engines that "crawl" across the Internet and index pages on Web servers, catalog that information and make it available to the Internet for searching.
  • A Robots.txt file is a special text file located in your Web server's root directory that contains restrictions for Web Spiders, telling them where they have permission to search.
  • Web Robots are not required to respect Robots.txt files.
  • To disallow all Web Spiders, specify the following in the Robots.txt file:
    • User-agent: *
    Disallow: /


    More: How to Write a Robots.txt File

    Comments

    Post a Comment

    Popular posts from this blog

    From Stiff Chatbot to Savage Sidekick: The UI/UX Glow-Up Story

    Image
    It's interesting how the UI/UX of AI assistants has been evolving over time.  The interaction options with each individual Chat and a Conversation thread have gradually expanded over time.  ChatGPT Gemini Grok User Input Add photos & files Recent files Create image Deep research Web search OpenAI Platform Upload files Add from Drive Photos Notebooks Guided learning Create image Canvas Create Music Guided Learning Deep research More tools Upload a file Recent Skills Chat Copy  Good response Bad response Add to project sources (~8/26) Share  Try Again View sources Branch in new Chat  Read Aloud Report Message Good response  Bad response  Copy response Double check response  Listen  Export to Docs Draft in Gmail Export to Replit  Report Legal issue Start Thread (~2/26) Copy response Create share link  Love it  this Needs improvement  Regenerate...
    Read more

    HOW TO Turn Your Mobile Browser into a Barcode Scanner for Web Apps

    Image
    Smartphone barcode scanners typically require installing a dedicated app. But what if your web app could access the camera and scan codes directly — no app required, just HTML and JavaScript? I tried Minhaz's nifty HTML5 QR Code & Barcode scanner and I'm bowled over by its simplicity. With hundreds of GitHub stars and forks , it's clear I'm not the only one impressed! I adapted Minhaz's sample code into a minimalist web page that scans a product barcode and redirects to Open Food Facts to display its nutrition information. Here's the code : You can also try it out live. Related:  *  Scanapp   *  HOW TO scan a barcode with an Android smartphone camera from a web page
    Read more

    HOW TO Read Linux ext4 Files on Windows with 7-Zip (No Drivers Needed)

    Image
    File systems are the organizational systems that decide how data is stored, named, located, and protected on a disk, SSD, USB drive, or partition. They handle directories, file metadata (permissions, timestamps, ownership), free space tracking, and more. Windows side Windows primarily uses: NTFS — the modern default. It supports journaling (for crash recovery), large files and volumes, fine-grained permissions, encryption, and compression. FAT32 and exFAT — simpler, widely used on removable media (USB sticks, SD cards) because they are highly cross-compatible with other operating systems. Windows has no native support for the main Linux file systems. When you connect a drive formatted with ext2/3/4, Windows typically sees it as “RAW” or unknown and may offer to format it (which would destroy the data). Linux side Most Linux distributions default to the ext family: ext2 — the older, non-journaling version. ext3 — added journaling for better reliability. ext4 — the current standard. It a...
    Read more