Open Food Facts Data Analysis

As a self-taught nutrition enthusiast, I have been contributing to the crowdsourced India database of Open Food Facts - "the Wikipedia of food", by adding and editing products. The decentralized open source project has data on more than 3 million products (as of April 2024) from countries around the world.

Open Food Facts (OFF) data is released as Open Data: it can be reused freely by anyone, under the Open Database License (ODBL). 

I've been cleaning up & compiling the data by food categories to extract insights.

There are multiple ways to get the data:

  • CSV or Excel format using the advanced search feature
  • MongoDB daily export - more than 30GB uncompressed!
  • JSONL daily export
  • CSV daily export
  • API

As I was interested in select product features and macronutrient details, I decided to use the OFF API. This is how the process for compiling data for each food category currently goes:

1. I construct the following URL with the fields I require to fetch the nutrition data as JSON -

https://in.openfoodfacts.org/cgi/search.pl?search_terms=Breakfast Cereals&fields=code,brands,quantity,product_name,energy-kcal_100g,carbohydrates_100g,added-sugars_100g,fat_100g,calcium_100g,cholesterol_100g,saturated-fat_100g,proteins_100g,salt_100g,sugars_100g,fiber_100g,nutriscore_grade,nova_group&json=1&sort_by=last_modified_t&page_size=500

2. I then use the Data.Page Online JSON to CSV conversion tool to convert the JSON and provide me with a downloadable CSV file. Alternatively, you can use this Python script to convert the JSON output to a CSV file. AI assistants can also help in the direct conversion if the size of the output is small.

3. I edit the CSV file using Excel or Google Sheets to clean the data - add minor missing details, review anomalies and fix inconsistencies (sometimes sodium  values may be mistakenly specified in grams instead of mg), simultaneously edit the data on the OFF website and rearrange the columns so that there is some order. As the data on OFF is crowdsourced, checking for outliers (by filtering & looking for extreme values in the ranges) helps in weeding out anomalies.

4. I upload the CSV data to Github following a folder structure matching the Food Safety and Standards Authority of India (FSSAI) categorization for packaged food products. The CSV file is then available in a neat tabular format through a publicly accessible URL that looks like this - https://github.com/mvark/indiafoodstats/blob/main/1.%20Dairy%20products%20and%20analogues/Milk.csv

5. To better appreciate, visualize and analyze the product data, I use the  Flat Viewer tool by just prefixing "flat" to the CSV file on Github, thus - https://flatgithub.com/mvark/indiafoodstats/blob/main/1.%20Dairy%20products%20and%20analogues/Milk.csv

As I browse, sort and filter I jot down my notes about the patterns and number ranges that emerge from viewing the histograms generated by the tool.

Histograms generated by the Flat Viewer tool
Histograms generated by the Flat Viewer tool

Comments

Post a Comment

Popular posts from this blog

Maven Crash Course - Learn Power Query, Power Pivot & DAX in 15 Minutes

Image
In the Maven Crash Course " Learn Power Query, Power Pivot & DAX in 15 Minutes ", Chris Dutton walks through a hypothetical business case while showcasing the power of modern Excel tools like Power Query, Power Pivot and DAX. Power Query is used to extract, transform, and load data from external sources like SQL databases, PDF and CSV files. A relational model is created to join the tables without writing a single formula. Exploratory analysis is done using Power Pivot and Data Analysis Expressions (DAX). Data Analysis Expressions is the language that we can use to define new calculated columns and measures to enhance our Data Model instead of the traditional cell formulas that we'd typically use if our data was stored in worksheets. Pivot Charts and slicers are used to design a simple interactive report that the sales managers can use to analyze revenue trends and product performance.
Read more

"Data Prep & Exploratory Data Analysis" course by Maven Analytics

Image
The Maven Analytics course on " Data Prep & Exploratory Data Analysis " by Alice Zhao is a beginner-friendly introduction to machine learning with Python. It covers the key steps of the data science workflow, including: Gathering Data Scoping a Project Modeling Data Sharing Insights Cleaning Data Exploring Data Modeling Data  Sharing Insights The presentation deck which is available as a download crisply summarizes the essential details and is a great reference to keep going back to. The course is well-suited for aspiring data scientists looking to get hands-on experience with the data prep and EDA stages of the machine learning workflow using Python and Jupyter Notebook. 
Read more

Oracle Cloud Infrastructure 2024 Generative AI Professional Course & Certification Exam (1Z0-1127-24)

Image
I was happy to learn about the Oracle Cloud Infrastructure 2024 Generative AI Professional Course with video tutorials and the Certification Exam (1Z0-1127-24) from their newsletter. I found it extremely useful, timely, and relevant. The course, with about 6 hours of videos, provided me with a comprehensive view of what it takes to build a Generative AI application. Although some parts reference OCI services, much of the course remains vendor neutral. Rohit Rahi and his team at Oracle have done a great job of structuring the beginner-friendly yet comprehensive course so well, covering all the essentials across four modules. The curriculum strikes an excellent balance between depth and accessibility.  While Gen AI output is typically non-deterministic, I was surprised to learn that they can be made completely deterministic under certain conditions. Specifically, in the OCI Gen AI service, setting the temperature parameter to 0 produces consistent, deterministic output for a g...
Read more