HOW TO Download Open Food Facts Data for Any Country Using Google Colab and Push It to GitHub

Open Food Facts has data on 4.6+ million food products worldwide. But if you only need data for one country, downloading the full 7.64 GB file (size at the time of writing) hosted on Hugging Face is overkill. Hugging Face acts as a CDN for large ML datasets — it not only stores the full Parquet file but also handles versioning, caching, and fast global downloads.

This guide co-written with Meta AI shows you how to use Google Colab to filter the dataset for just one country, add direct product links, and save the result to a public GitHub repo. Takes < 30 minutes total based on your familiarity with these tools & file size. No software installs needed.

What you’ll end up with: A CSV like india_products_18392.csv with 16 columns, including a URL for each product. The same steps work for France, Canada, USA, UK, or any other country in the database.

What You Need Before Starting

  1. A Google account to use Colab
  2. A GitHub account + an empty public repo where the CSV will live
  3. A GitHub Personal Access Token so Colab can push files for you

Step 1: Create Your GitHub Token

  1. Colab needs permission to upload files to your repo. You give it that with a token.
  2. Go to GitHub → click your profile pic → Settings
  3. Scroll to the bottom of the left menu → Developer settings
  4. Personal access tokens → Tokens (classic) → Generate new token (classic)
  5. Name it Colab OFF and set expiration to 90 days
  6. Under “Scopes”, check repo only. Uncheck everything else.
  7. Click Generate token and copy the code that starts with ghp_. Save it somewhere. You won’t see it again.

Step 2: Set Up Your Colab Notebook

Go to colab.research.google.com → New notebook

Copy-paste this entire code block into the first cell:

Python

# Install deps

!pip install -q huggingface_hub

# Full pipeline: Download OFF data, filter by country, add URL column, push to GitHub

import duckdb

import time

import os

import getpass

from huggingface_hub import hf_hub_download

# ========== CHANGE THESE 5 VALUES ==========

GITHUB_USER = "your-github-username" # your GitHub username

GITHUB_REPO = "your-repo-name" # empty repo you created

GIT_EMAIL = "your_email@example.com" # your email

GIT_NAME = "Your Name" # your name

COUNTRY_TAG = "en:india" # change this for other countries

# ===========================================

start = time.time()


# 1. Download the 7.64GB Open Food Facts file from Hugging Face

print("Step 1/4: Downloading Open Food Facts database...")

print("This is 3.2GB and takes 3-8 minutes. It'll resume if interrupted.")

local_path = hf_hub_download(

    repo_id="openfoodfacts/product-database",

    filename="food.parquet",

    repo_type="dataset",

    local_dir="/content/hf_cache",

    resume_download=True

)

print(f"Downloaded: {os.path.getsize(local_path) / 1e9:.2f} GB")


# 2. Filter for your country and build the table

print(f"\nStep 2/4: Filtering for {COUNTRY_TAG}...")

con = duckdb.connect()

con.execute(f"""

CREATE TEMP TABLE country_data AS

SELECT

    code,

    'https://world.openfoodfacts.org/product/' || code AS url,

    list_extract(product_name, 1).text AS product_name,

    brands,

    categories,

    list_extract(list_filter(nutriments, n -> n.name = 'energy-kcal'), 1)."100g" AS "energy-kcal_100g",

    list_extract(list_filter(nutriments, n -> n.name = 'carbohydrates'), 1)."100g" AS carbohydrates_100g,

    list_extract(list_filter(nutriments, n -> n.name = 'sugars'), 1)."100g" AS sugars_100g,

    list_extract(list_filter(nutriments, n -> n.name = 'fiber'), 1)."100g" AS fiber_100g,

    list_extract(list_filter(nutriments, n -> n.name = 'proteins'), 1)."100g" AS proteins_100g,

    list_extract(list_filter(nutriments, n -> n.name = 'fat'), 1)."100g" AS fat_100g,

    list_extract(list_filter(nutriments, n -> n.name = 'saturated-fat'), 1)."100g" AS "saturated-fat_100g",

    list_extract(list_filter(nutriments, n -> n.name = 'salt'), 1)."100g" AS salt_100g,

    nova_group,

    nutriscore_grade,

    to_timestamp(last_modified_t) AS last_modified_datetime

FROM read_parquet('{local_path}')

WHERE list_contains(countries_tags, '{COUNTRY_TAG}')

  AND code IS NOT NULL AND code!= ''

  AND len(product_name) > 0

""")


# 3. Count rows and save CSV with count in the filename

filecount = con.execute("SELECT COUNT(*) FROM country_data").fetchone()[0]

country_name = COUNTRY_TAG.replace("en:", "")

filename = f"{country_name}_products_{filecount}.csv"

local_csv_path = f"/content/{filename}"

print(f"Found {filecount:,} products for {country_name}")

print(f"\nStep 3/4: Writing {filename}...")

con.execute(f"COPY country_data TO '{local_csv_path}' (HEADER, DELIMITER ',')")

# 4. Push to GitHub

print(f"\nStep 4/4: Pushing to GitHub...")

GITHUB_TOKEN = getpass.getpass("Paste your GitHub Personal Access Token: ")

!git config --global user.email "{GIT_EMAIL}"

!git config --global user.name "{GIT_NAME}"

!git clone https://{GITHUB_USER}:{GITHUB_TOKEN}@github.com/{GITHUB_USER}/{GITHUB_REPO}.git /content/repo

!cp {local_csv_path} /content/repo/

%cd /content/repo

!git add {filename}

!git commit -m "Update {country_name} products: {filecount} rows - $(date +%Y-%m-%d)"

!git push origin main

print(f"\n✅ Done in {round((time.time()-start)/60, 1)} min")

print(f"File: https://github.com/{GITHUB_USER}/{GITHUB_REPO}/blob/main/{filename}")

print(f"Raw CSV: https://raw.githubusercontent.com/{GITHUB_USER}/{GITHUB_REPO}/main/{filename}")

Step 3: Change It For Your Country

Before you run the cell, edit these 5 lines at the top:

Python

GITHUB_USER = "your-github-username" # put your username

GITHUB_REPO = "off-data" # name of your empty repo

GIT_EMAIL = "you@example.com" # your email

GIT_NAME = "Your Name" # your name

COUNTRY_TAG = "en:india" # change this line if you need data for a different country

For country-specific sites like https://in.openfoodfacts.org/product/, change the URL line to:

'https://in.openfoodfacts.org/product/' || code AS url,

Step 4: Run It

  • Click the play button next to the code cell
  • When prompted, paste your GitHub token and hit Enter
  • Wait ∼5-10 minutes. You’ll see progress for each step.
  • When it’s done you’ll get 2 links. The “Raw CSV” link is a direct download anyone can use.
Note: If your file count is >150k, the CSV will be ~30-40MB. GitHub is fine with that, but it'll take 10-20 sec to push.


What’s Happening Under the Hood

  • Download: The script grabs the official Open Food Facts 7.64GB (on 7-July-2026) Parquet file from Hugging Face.  Using hf_hub_download gives us two key benefits: 1) the download resumes automatically if Colab disconnects, so you don’t restart a pull, and 2) HF’s infrastructure avoids the strict rate limits you’d hit querying the file over HTTP directly. You don’t need an account or token for public datasets, though adding one boosts your bandwidth if you hit 429 errors. Essentially, Hugging Face is just the reliable warehouse we fetch the raw data from before DuckDB filters it. 
  • Filter: Uses DuckDB, which can query that huge file without loading it all into memory. It keeps only products tagged with your country. When Colab downloads the file, it saves it to /content/hf_cache on the notebook’s temporary VM disk. This is ephemeral storage that Google provides with every Colab session. It has nothing to do with your Google Drive, Google Cloud, or any other subscribed service you pay for. The VM and its disk are wiped when your runtime disconnects or times out, so you’re not charged for storage.
  • Transform: Pulls out 15 useful fields: barcode, name, brand, nutrients, NOVA score, etc. Adds a url column so you can click straight to the product page.
  • Export: Saves a CSV with the product count in the filename, like india_products_18392.csv
  • Upload: Pushes that CSV to your GitHub repo so it’s public and versioned.

Troubleshooting

  • HTTP 429 error - The code already handles this. If it still fails, add a free Hugging Face token.
  • File too large for GitHub - GitHub’s limit is 100MB. Most country files are 5-40MB. If yours is bigger, switch to Parquet or use Git LFS.
  • Wrong country / 0 rows - Check your COUNTRY_TAG. It must match Open Food Facts exactly. Use en: prefix.
  • Token rejected - Make sure you checked repo scope when creating the token. Generate a new one if needed.

Why This Approach Works

  • No local setup: Colab gives you a free VM with 12GB RAM. Enough to handle the 3.2GB file.
  • No rate limits: Downloads the file once locally. DuckDB then runs offline, so no more API calls.
  • Reproducible: Anyone can run your notebook, change COUNTRY_TAG, and get their own dataset.
  • Public + versioned: GitHub stores every run. You can track how the data changes over time.

You can use this process for any country or schedule it monthly using GitHub Actions. 

Comments

Post a Comment