Spider in the Web soup

  • Web Spiders or Robots are WWW search engines that "crawl" across the Internet and index pages on Web servers, catalog that information and make it available to the Internet for searching.
  • A Robots.txt file is a special text file located in your Web server's root directory that contains restrictions for Web Spiders, telling them where they have permission to search.
  • Web Robots are not required to respect Robots.txt files.
  • To disallow all Web Spiders, specify the following in the Robots.txt file:
    • User-agent: *
    Disallow: /


    More: How to Write a Robots.txt File

    Comments

    Post a Comment

    Popular posts from this blog

    Uncle Bob vs. Grady Booch: Rethinking Code Reviews in the Age of AI

    Image
    In response to a question about the feasibility of effective code reviews for large (e.g., 500-line) AI-generated PRs like those from Claude, especially when reviewers lack deep codebase familiarity in new projects or fast-paced environments, Uncle Bob Martin and Grady Booch have contrasting views Uncle Bob Martin advocates metrics-based oversight (test coverage, complexity, dependencies) and higher-level management over line-by-line AI code review, while Grady Booch stresses manual verification for vulnerabilities, dead code, and performance factors. Uncle Bob Martin : " I don’t review code written by agents . I measure things like test coverage, dependency structure, cyclomatic complexity, module sizes, mutation testing, etc.  Much can be inferred about the quality of the code from those metrics. The code itself I leave to the AI.  Humans are slow at code. To get productivity we humans need to disengage from code and manage from a higher level." Grady Booch : "Unlike B...
    Read more

    10 Tips to Avoid Claude Usage Limits

    Image
    Summary of an article on X by kaize - 1. Edit your prompt. Don't send a follow-up because Token cost per message = all previous messages + your new one. 2. Start a fresh chat every 15–20 messages: When a chat gets long → ask Claude to summarize everything → copy it → new chat → paste as first message. 3. Batch your questions into one message for fewer context reloads (that cost tokens). 4. Upload recurring files to Projects - Cached project content doesn't eat into your usage. 5. Set up Memory & User Preferences - Save your role, communication style, and settings once. Claude will automatically apply them to every new chat. 6. Turn off features (like Web search, Connectors, "Explore" mode, "Search and Tools", "Advanced Thinking") you're not actively using. 7. Use Haiku for simple tasks 8. Spread your work across the day as Claude uses a rolling 5-hour window.  9. Work during off-peak hours. Anything outside of peak hours: 5:00 AM to 11:00 ...
    Read more

    This Week I Learned - Week 14 2026

    Image
    This Week I Learned -  * Since A.I. coding tools from Anthropic, OpenAI, Cursor and other companies took off last year, one result has now become apparent: code overload . - NYT * GitHub platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear — Kyle Daigle, COO, GitHub * At tech companies like Meta and Shopify, managers have started to factor A.I. use into performance reviews, rewarding workers who make heavy use of A.I. tools and chastening those who don’t. It has created an expensive new status game, known as “tokenmaxxing,” among A.I.-obsessed workers who are desperate to prove how productive they are. - NYT * OpenAI's agentic coding tool, Codex, had tripled its weekly active users since the start of the year. Overall Codex use, measured in tokens, has increased fivefold. Google's A.I. models processed more than 1.3 quadrillion tokens a month in 2025. * AI companies char...
    Read more