Open Food Facts Data Analysis

As a self-taught nutrition enthusiast, I have been contributing to the crowdsourced India database of Open Food Facts - "the Wikipedia of food", by adding and editing products. The decentralized open source project has data on more than 3 million products (as of April 2024) from countries around the world.

Open Food Facts (OFF) data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL). 

I've been cleaning up & compiling the data by food categories to extract insights.

There are multiple ways to get the data:

  • CSV or Excel format using the advanced search feature
  • MongoDB daily export - more than 30GB uncompressed!
  • JSONL daily export
  • CSV daily export
  • API

As I was interested in select product features and macronutrient details, I decided to use the OFF API. This is how the process for compiling data for each food category currently goes:

1. I construct the following URL with the fields I require to fetch the nutrition data as JSON -

https://in.openfoodfacts.org/cgi/search.pl?search_terms=Breakfast Cereals&fields=code,brands,quantity,product_name,energy-kcal_100g,carbohydrates_100g,added-sugars_100g,fat_100g,calcium_100g,cholesterol_100g,saturated-fat_100g,proteins_100g,salt_100g,sugars_100g,fiber_100g,nutriscore_grade,nova_group&json=1&sort_by=last_modified_t&page_size=500

2. I then use the Data.Page Online JSON to CSV conversion tool to convert the JSON and provide me with a downloadable CSV file. Alternatively, you can use this Python script to convert the JSON output to a CSV file. AI assistants can also help in the direct conversion if the size of the output is small.

3. I edit the CSV file using Excel or Google Sheets to clean the data - add minor missing details, review anomalies and fix inconsistencies (sometimes sodium  values may be mistakenly specified in grams instead of mg), simultaneously edit the data on the OFF website and rearrange the columns so that there is some order. As the data on OFF is crowdsourced, checking for outliers (by filtering & looking for extreme values in the ranges) helps in weeding out anomalies.

4. I upload the CSV data to Github following a folder structure matching the Food Safety and Standards Authority of India (FSSAI) categorization for packaged food products. The CSV file is then available in a neat tabular format through a publicly accessible URL that looks like this - https://github.com/mvark/indiafoodstats/blob/main/1.%20Dairy%20products%20and%20analogues/Milk.csv

5. To better appreciate, visualize and analyze the product data, I use the  Flat Viewer tool by just prefixing "flat" to the CSV file on Github, thus - https://flatgithub.com/mvark/indiafoodstats/blob/main/1.%20Dairy%20products%20and%20analogues/Milk.csv

As I browse, sort and filter I jot down my notes about the patterns and number ranges that emerge from viewing the histograms generated by the tool.

Histograms generated by the Flat Viewer tool
Histograms generated by the Flat Viewer tool

Comments

Post a Comment

Popular posts from this blog

GitHub Copilot Q&A - 1

Image
Key takeaways from the talk in Q&A format - What is GitHub Copilot and how does it work? GitHub Copilot is a suite of tools designed for code completion. It works by analysing the code and comments you type in your IDE and sending that context to GitHub. Copilot then uses OpenAI's language model (in December 2024, it is GPT 4o by default and you can also choose the preview versions of o1-mini & o1-preview if you like) to understand your intent and generate suggestions for code completion. You can then accept, reject, or modify these suggestions. What context does GitHub Copilot consider when making suggestions? Copilot considers the entirety of the file you are currently working on, not just the code from your cursor position upwards. It also analyses the content of other open tabs in your IDE, starting with those closest to the current tab. This helps Copilot focus on the most relevant information. Is it necessary to meticulously manage open tabs for Copilot to work effect...
Read more

Maven Crash Course - Learn Power Query, Power Pivot & DAX in 15 Minutes

Image
In the Maven Crash Course " Learn Power Query, Power Pivot & DAX in 15 Minutes ", Chris Dutton walks through a hypothetical business case while showcasing the power of modern Excel tools like Power Query, Power Pivot and DAX. Power Query is used to extract, transform, and load data from external sources like SQL databases, PDF and CSV files. A relational model is created to join the tables without writing a single formula. Exploratory analysis is done using Power Pivot and Data Analysis Expressions (DAX). Data Analysis Expressions is the language that we can use to define new calculated columns and measures to enhance our Data Model instead of the traditional cell formulas that we'd typically use if our data was stored in worksheets. Pivot Charts and slicers are used to design a simple interactive report that the sales managers can use to analyze revenue trends and product performance.
Read more

"Data Prep & Exploratory Data Analysis" course by Maven Analytics

Image
The Maven Analytics course on " Data Prep & Exploratory Data Analysis " by Alice Zhao is a beginner-friendly introduction to machine learning with Python. It covers the key steps of the data science workflow, including: Gathering Data Scoping a Project Modeling Data Sharing Insights Cleaning Data Exploring Data Modeling Data  Sharing Insights The presentation deck which is available as a download crisply summarizes the essential details and is a great reference to keep going back to. The course is well-suited for aspiring data scientists looking to get hands-on experience with the data prep and EDA stages of the machine learning workflow using Python and Jupyter Notebook. 
Read more