Screen scrape with jQuery, AJAX, JSONP & YQL

Since reading this excellent article about scraping content from a Wikipedia page using Yahoo! Query Language (YQL) as a proxy for cross-domain Ajax, I'm hooked to YQL. YQL helps in circumventing the same-origin policy that prevents a script loaded from one domain from getting or manipulating properties of a document from another domain.  YQL has been around for about 2 years now & last year Yahoo introduced the capability to execute the tables of data built through YQL using JavaScript.

Ajax, jQuery, JSONP (JSON with Padding) & YQL make a heady combination - check Christian Heilmann's code samples.

Some facts about YQL from around the Web (work in progress) -
* YQL is a hosted web service that can scrape HTML for you. It also runs the HTML through HTML Tidy and caches it for you.
* It only returns the body content of the HTML - so no styling (other than inline styles) will get through.
* ...it treats the info on the web as a virtual table that developers can manipulate in a standardized way, regardless of the API that data came from.
* YQL understands and supports data sources like RSS, Atom, JSON, XML, CSV, HTML, Flickr, Yahoo! Finance, Weather, and so on.
* ...makes client-side mashups possible without using server-side proxies.
* Usage Limits:
Per application limit (identified by your Access Key): 100,000 calls per day
Per IP limits: /v1/public/*: 1,000 calls per hour; /v1/yql/*: 10,000 calls per hour

Also see:
HOW TO prevent screen scraping 
Google Spreadsheets functions for scraping external data
77

Comments

Post a Comment

Popular posts from this blog

HOW TO download files hosted in Azure Web App in bulk

Image
Here are the steps to download all the site files hosted in an Azure Web App service in bulk for local modifications or cloning to another hosting location - 1. Log in to the Azure portal 2. Navigate to your App Service 3. Go to the "Advanced Tools" section  4. Click on "Go" link to launch the Kudu console  5. In the Kudu console, navigate to the "Debug console" and then select "CMD" Once you are in the command prompt, navigate to the "Site" directory and then to the "wwwroot" directory. Your Web App should be located in a subdirectory of "wwwroot" 6. Click on the Download icon on the left of the label "wwwroot" to download all the files in the Web App as a zipped file. Alternatively, if your web app URL is https:// <yourappname> .azurewebsites.net/ insert the word scm after the subdomain name thus - https:// <yourappname> . scm .azurewebsites.net/ This will lead you directly to the Kudu con...
Read more

30+ GitHub Products & Key Ecosystem Features You Should Know in 2026

Image
GitHub has grown far beyond a place to host code — today it's an entire ecosystem covering CI/CD, AI-assisted development, security, package management, and team collaboration. Whether you're new to the platform or just want a single reference point, here's an alphabetical list of GitHub's major products and features, each linked to its official page - GitHub Actions GitHub Advanced Security GitHub Apps GitHub CLI GitHub Code Review GitHub Code Security GitHub Codespaces GitHub Container Registry GitHub Copilot GitHub Copilot Business GitHub Copilot CLI GitHub Copilot Enterprise GitHub Copilot Extensions GitHub Dependabot GitHub Desktop GitHub Discussions GitHub Education GitHub Enterprise Cloud GitHub Enterprise Server GitHub Gist GitHub Importer GitHub Innovation Graph GitHub Issues GitHub Marketplace GitHub Mobile GitHub Models  (retiring July 30, 2026) GitHub Packages GitHub Pages ...
Read more

HOW TO add a header or footer to a dynamically generated Word document

Image
There was a question recently on my CodeProject article "Dynamically generate a MS Word document using HTML & CSS" . The article describes how to generate a Word document programmatically without using any components, by exploiting the formatting features exposed through Office XML and CSS. The questioner wanted to know how to add a custom header and footer and show something like Page X of Y (total pages). Based on what I have tried so far, showing a header and footer using Office XML in MS Word is not as easy as it is in Excel . If you want to keep the code simple & don't need any great fireworks i.e. you will just settle for page numbers in the header or footer then it's a matter of adding a few lines to the original source code. The code can be viewed at this Github Gist Update (1-Sep-2010): To add a custom header & footer, check this new post To add a footer that shows Page number at the bottom right, here are the steps - 1) Add these classes...
Read more