Match real labels: E-numbers, aliases and Open Food Facts import; crop scans to the frame

.github/workflows/ci.yml

@@ -34,4 +34,5 @@ jobs:
           cache: npm
           cache-dependency-path: frontend/package-lock.json
       - run: npm ci
+      - run: npm test
       - run: npx tsc --noEmit

.gitignore

@@ -37,6 +37,7 @@ instance/
 
 # Node (in case any exist at root level)
 node_modules/
+frontend/.test-dist/
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*

README.md

@@ -2,7 +2,7 @@
 
 Point your phone at an ingredients label and AddiGuard tells you which food additives it found and how risky they are.
 
-The Expo app takes a photo and posts it to a Flask API. The API runs OCR on the image (EasyOCR), fuzzy-matches the text against a database of additives, scores each hit, and returns a Red / Yellow / Green traffic light per additive plus an overall rating for the label.
+The Expo app crops a photo to the on-screen ingredients frame and posts it to a Flask API. The API runs OCR on the image (EasyOCR), matches E-numbers, names and aliases against a database of additives, and returns a Red / Yellow / Green rating for curated additives or Unrated when risk information is missing, plus an overall rating for the label.
 
 ## Project structure
 
@@ -18,11 +18,16 @@ cd backend
 python -m venv .venv && source .venv/bin/activate
 pip install -r requirements.txt     # CPU-only torch; ~1 GB
 python seed.py                      # (re)creates the SQLite DB with sample additives
+python import_additives.py          # upserts the bundled Open Food Facts taxonomy, offline
 FLASK_DEBUG=1 python run.py         # serves on http://0.0.0.0:5000
 ```
 
 The OCR models (~100 MB) are downloaded on the first `/api/scan` request, so expect that call to be slow once.
 
+The additive schema includes a unique, nullable `e_number` (for example `E250`, `E150a` or `E101(i)`) and a JSON list of lowercase aliases. `seed.py` destructively rebuilds the database, so run it to adopt the new schema on an existing development database; there is no migration system yet. Back up any data you need first. The importer can then be run repeatedly without duplicating additives. It preserves the five seeded additives' existing factors and does not assign risk factors to newly imported entries. The bundled snapshot imports 643 concrete E-number entries, of which 638 are Unrated after seeding. Generic taxonomy categories and wildcard codes are excluded.
+
+The unmodified Open Food Facts snapshot and its source, checksum and licensing information are in `backend/data/`; importing and testing do not require network access. To use an updated raw taxonomy, run `python import_additives.py /path/to/additives.txt`. The importer invalidates the current process's alias index; restart other running API processes after import, particularly when updating existing aliases without adding rows.
+
 Configuration is via environment variables:
 
 | Variable       | Default                  | Purpose                                  |
@@ -45,6 +50,8 @@ pytest
 ```bash
 cd frontend
 npm ci
+npm test                          # pure crop geometry tests, no device required
+npx tsc --noEmit
 npx expo start
 ```
 
@@ -56,6 +63,8 @@ EXPO_PUBLIC_API_URL=http://192.168.1.20:5000/api
 
 In development builds a **Mock Mode** switch on the camera screen returns canned results so the UI can be exercised without a backend.
 
+The camera measures the preview and scan frame, maps the frame through the preview's centered aspect-ratio fill to captured photo pixels, and crops before upload. The cropped image is downscaled to at most 1600 pixels on its long side and encoded as JPEG at quality 0.8. Only the framed region is sent to OCR. Mock Mode bypasses image processing. Scan results live in a React context rather than URL parameters; opening the results route without an in-memory scan redirects to the camera.
+
 ## API
 
 `POST /api/scan` with a `multipart/form-data` body containing an `image` file.
@@ -69,16 +78,19 @@ In development builds a **Mock Mode** switch on the camera screen returns canned
   "results": [
     {
       "name": "Sodium Nitrite",
+      "matched_text": "e250",
       "risk_score": 0.83,
       "traffic_light": "Red",
-      "details": { "id": 1, "name": "Sodium Nitrite", "toxicity_level": 9, "...": "..." }
+      "details": { "id": 1, "name": "Sodium Nitrite", "e_number": "E250", "aliases": ["sodium nitrite"], "toxicity_level": 9, "...": "..." }
     }
   ]
 }
 ```
 
 Errors return `{ "error": "..." }` with a 4xx/5xx status.
 
+Each result includes `matched_text`, the normalized OCR phrase or code that triggered detection. Names and aliases match on phrase boundaries, with conservative fuzzy matching for longer aliases to tolerate OCR mistakes; short aliases such as `MSG` only match exactly. Explicit E-number variants such as `E 250` and `E-250` are recognized, while bare numbers require an additive-class context such as `preservative (250)` or `flavour enhancer (621)`. Multiple hits for the same additive produce one result.
+
 ### Risk score
 
 Each additive has four 1–10 factors. The score is a weighted sum normalised to 0–1:
@@ -89,6 +101,8 @@ risk = toxicity·0.4 + exposure·0.3 + sensitivity·0.2 + cumulative·0.1
 
 `> 0.7` is Red, `0.4–0.7` is Yellow, `< 0.4` is Green. The overall label rating is the highest-scoring additive found. The backend is the only place these thresholds live; the app just renders the `traffic_light` values it receives.
 
+If any risk factor is null, that additive has `risk_score: null` and `traffic_light: "Unrated"`. Unrated detections are shown with a grey information badge and excluded from the overall score; they are not a claim of safety. If every detection is Unrated, the overall score is null and the overall traffic light is Unrated. An empty result retains `overall_risk_score: 0` and `overall_traffic_light: "Green"` for compatibility; no detection is not proof of a safe label.
+
 ## Status
 
-Early MVP. The seed data has five additives with placeholder risk factors and health notes; none of it is medical advice. Matching is by additive name only (no E-numbers or synonyms yet), so real labels will be missed until the additive dataset is expanded.
+Early MVP. The seed data has five additives with placeholder risk factors and health notes; none of it is medical advice. Open Food Facts expands detection coverage, not the curated risk database. OCR and approximate matching can still miss ingredients or return false positives, and the preview-to-photo mapping still needs validation on physical iOS and Android devices.

backend/app/models.py

@@ -11,13 +11,15 @@ class Additive(db.Model):
 
     id = db.Column(db.Integer, primary_key=True)
     name = db.Column(db.String(255), unique=True, nullable=False)
+    e_number = db.Column(db.String(16), unique=True, nullable=True)
+    aliases = db.Column(db.JSON, nullable=False, default=list)
     description = db.Column(db.Text, nullable=True)
 
     # Risk factors (1-10 scale)
-    toxicity_level = db.Column(db.Integer, default=1) # toxicity_score
-    exposure_level = db.Column(db.Integer, default=1) # exposure_risk
-    sensitivity_level = db.Column(db.Integer, default=1) # sensitivity
-    cumulative_level = db.Column(db.Integer, default=1) # cumulative_effect
+    toxicity_level = db.Column(db.Integer, nullable=True, default=None) # toxicity_score
+    exposure_level = db.Column(db.Integer, nullable=True, default=None) # exposure_risk
+    sensitivity_level = db.Column(db.Integer, nullable=True, default=None) # sensitivity
+    cumulative_level = db.Column(db.Integer, nullable=True, default=None) # cumulative_effect
 
     health_risk = db.Column(db.String(255), nullable=True) # Text description
     usage_limit = db.Column(db.String(255), nullable=True)
@@ -26,6 +28,8 @@ def to_dict(self):
         return {
             'id': self.id,
             'name': self.name,
+            'e_number': self.e_number,
+            'aliases': self.aliases or [],
             'description': self.description,
             'toxicity_level': self.toxicity_level,
             'exposure_level': self.exposure_level,

backend/app/routes.py

@@ -27,7 +27,8 @@ def scan_image():
         current_app.logger.exception("Image analysis failed")
         return jsonify({"error": "Image analysis failed"}), 500
 
-    overall_risk_score = max((r["risk_score"] for r in results), default=0)
+    rated_scores = [r["risk_score"] for r in results if r["risk_score"] is not None]
+    overall_risk_score = max(rated_scores) if rated_scores else (None if results else 0)
     return jsonify({
         "status": "success",
         "additives_found": len(results),

backend/app/services/scan_service.py

@@ -1,8 +1,89 @@
+import re
 import threading
-
-from thefuzz import fuzz, utils
-
-from app.models import Additive
+from collections import defaultdict
+
+from flask import current_app
+from sqlalchemy import func, select
+from thefuzz import fuzz
+
+from app.models import Additive, db
+
+
+_CODE_BODY = r"\d{3,4}[a-z]?(?:\s*\(\s*[ivx]+\s*\))?"
+_E_NUMBER = re.compile(rf"\be\s*[-–—]?\s*({_CODE_BODY})(?!\w)", re.IGNORECASE)
+_CLASS_CODES = re.compile(
+    r"\b(?:preservatives?|flavou?r enhancers?|colou?rs?|emulsifiers?|antioxidants?|"
+    r"sweeteners?|stabili[sz]ers?|thickeners?|acidity regulators?|raising agents?)"
+    rf"\s*[:(\[]?\s*({_CODE_BODY}(?:\s*[,;/]\s*{_CODE_BODY})*)(?!\w)",
+    re.IGNORECASE,
+)
+
+
+def normalize_e_number(value):
+    if not value:
+        return None
+    match = _E_NUMBER.fullmatch(str(value).strip())
+    return "E" + re.sub(r"\s+", "", match.group(1)).lower() if match else None
+
+
+def normalize_phrase(value):
+    text = str(value).lower()
+    text = re.sub(r"\b[a-z0]+\b", lambda m: m[0].replace("0", "o") if re.search(r"[a-z]", m[0]) else m[0], text)
+    return " ".join(re.findall(r"[^\W_]+", text))
+
+
+def match_additives(extracted_text, index):
+    raw = " ".join(extracted_text)
+    aliases, codes, fuzzy, widths = index
+    found = {}
+    for match in _E_NUMBER.finditer(raw):
+        key = codes.get(normalize_e_number(match[0]))
+        if key is not None:
+            found.setdefault(key, normalize_phrase(match[0]))
+    for match in _CLASS_CODES.finditer(raw):
+        for code in re.findall(_CODE_BODY, match[1], re.IGNORECASE):
+            key = codes.get(normalize_e_number(f"E{code}"))
+            if key is not None:
+                found.setdefault(key, code.lower())
+
+    tokens = normalize_phrase(_E_NUMBER.sub(" ", raw)).split()
+    occupied = set()
+    for width in widths:
+        for start in range(len(tokens) - width + 1):
+            span = set(range(start, start + width))
+            if occupied & span:
+                continue
+            phrase = " ".join(tokens[start:start + width])
+            if phrase in aliases:
+                for key in aliases[phrase]:
+                    found.setdefault(key, phrase)
+                occupied.update(span)
+
+    for width in widths:
+        for start in range(len(tokens) - width + 1):
+            span = set(range(start, start + width))
+            if occupied & span:
+                continue
+            phrase = " ".join(tokens[start:start + width])
+            # A lone word gets a stricter bar: one substitution in a word under twelve letters
+            # scores at most 90, which is exactly where everyday label words collide with
+            # additive names ("carbonated" vs "carbonates"). Multi-word phrases carry enough
+            # context to keep the looser threshold for OCR slips.
+            threshold = 91 if width == 1 else 88
+            best_score, best_keys = threshold - 1, set()
+            for alias in fuzzy.get((width, phrase[0]), []):
+                if abs(len(alias) - len(phrase)) > max(1, len(alias) // 8):
+                    continue
+                score = fuzz.ratio(alias, phrase)
+                if score > best_score:
+                    best_score, best_keys = score, set(aliases[alias])
+                elif score == best_score:
+                    best_keys.update(aliases[alias])
+            if best_score >= threshold and len(best_keys) == 1:
+                found.setdefault(next(iter(best_keys)), phrase)
+                occupied.update(span)
+
+    return found
 
 
 class ScanService:
@@ -23,73 +104,77 @@ def reader(self):
         return self._reader
 
     def extract_text(self, image_bytes):
-        """
-        Extract text from image bytes using EasyOCR.
-        """
         return self.reader.readtext(image_bytes, detail=0)
 
     def calculate_risk_score(self, additive):
-        """
-        Risk Formula:
-        total_risk = (toxicity_score * 0.4) + (exposure_risk * 0.3) + (sensitivity * 0.2) + (cumulative_effect * 0.1)
-
-        Input levels are 1-10, normalized to 0.1-1.0 so the result lands in the 0-1 range.
-        """
-        t = additive.toxicity_level / 10.0
-        e = additive.exposure_level / 10.0
-        s = additive.sensitivity_level / 10.0
-        c = additive.cumulative_level / 10.0
-
-        total_risk = (t * 0.4) + (e * 0.3) + (s * 0.2) + (c * 0.1)
-        return round(total_risk, 2)
+        factors = (additive.toxicity_level, additive.exposure_level,
+                   additive.sensitivity_level, additive.cumulative_level)
+        if any(value is None for value in factors):
+            return None
+        return round(sum(value * weight / 10 for value, weight in zip(factors, (0.4, 0.3, 0.2, 0.1))), 2)
 
     def determine_traffic_light(self, risk_score):
-        """
-        If > 0.7 -> High Risk (Red)
-        0.4-0.7 -> Medium (Yellow)
-        < 0.4 -> Low (Green)
-        """
+        if risk_score is None:
+            return "Unrated"
         if risk_score > 0.7:
             return "Red"
-        elif risk_score >= 0.4:
+        if risk_score >= 0.4:
             return "Yellow"
-        else:
-            return "Green"
+        return "Green"
+
+    @classmethod
+    def refresh_alias_index(cls):
+        current_app.extensions.pop("additive_alias_index", None)
+
+    @staticmethod
+    def _build_index(additives):
+        aliases = defaultdict(set)
+        codes = {}
+        for key, additive in additives.items():
+            if additive.e_number:
+                codes[normalize_e_number(additive.e_number)] = key
+            for value in [additive.name, *(additive.aliases or [])]:
+                phrase = normalize_phrase(value)
+                if not phrase or not re.search(r"[^\W\d_]", phrase) or normalize_e_number(value):
+                    continue
+                aliases[phrase].add(key)
+        fuzzy = defaultdict(list)
+        widths = set()
+        for phrase in aliases:
+            width = len(phrase.split())
+            widths.add(width)
+            if len(phrase.replace(" ", "")) >= 6:
+                fuzzy[(width, phrase[0])].append(phrase)
+        return aliases, codes, fuzzy, sorted(widths, reverse=True)
 
     def analyze_image(self, image_bytes, additives_cache=None):
-        """
-        1. Extract text
-        2. Match against DB
-        3. Calculate Risk
-        """
         extracted_text = self.extract_text(image_bytes)
-
-        # Get all additives from DB (optimize by caching or specific query later)
-        # For MVP, fetching all names is fine if list is small.
-        if additives_cache is None:
-            all_additives = Additive.query.all()
+        if additives_cache is not None:
+            additives = dict(enumerate(additives_cache))
+            index = self._build_index(additives)
         else:
-            all_additives = additives_cache
-
-        found_additives = []
-
-        # Strategy: Iterate through DB names and check if they appear in the extracted text (fuzzy).
-        # full_process lowercases and turns punctuation into spaces so "vitamin-c" still matches "Vitamin C".
-        full_text = utils.full_process(" ".join(extracted_text))
+            key = tuple(db.session.execute(select(func.count(Additive.id), func.max(Additive.id))).one())
+            cached = current_app.extensions.get("additive_alias_index")
+            if cached is None or cached[0] != key:
+                additives = {a.id: a for a in Additive.query.all()}
+                index = self._build_index(additives)
+                current_app.extensions["additive_alias_index"] = (key, index)
+            else:
+                index = cached[1]
 
-        for additive in all_additives:
-            # partial_ratio matches substring
-            match_score = fuzz.partial_ratio(utils.full_process(additive.name), full_text)
+        found = match_additives(extracted_text, index)
 
-            if match_score > 85: # Threshold for match
-                risk_score = self.calculate_risk_score(additive)
-                traffic_light = self.determine_traffic_light(risk_score)
-
-                found_additives.append({
-                    "name": additive.name,
-                    "risk_score": risk_score,
-                    "traffic_light": traffic_light,
-                    "details": additive.to_dict()
-                })
-
-        return found_additives
+        if additives_cache is None:
+            additives = {a.id: a for a in Additive.query.filter(Additive.id.in_(found)).all()} if found else {}
+        results = []
+        for key in sorted(found):
+            additive = additives[key]
+            risk_score = self.calculate_risk_score(additive)
+            results.append({
+                "name": additive.name,
+                "matched_text": found[key],
+                "risk_score": risk_score,
+                "traffic_light": self.determine_traffic_light(risk_score),
+                "details": additive.to_dict(),
+            })
+        return results

backend/data/PROVENANCE.md

@@ -0,0 +1,34 @@
+# Open Food Facts additive taxonomy snapshot
+
+`additives.txt` is an unmodified raw taxonomy from Open Food Facts and its
+contributors, downloaded successfully on 2026-09-09 from:
+
+https://raw.githubusercontent.com/openfoodfacts/openfoodfacts-server/main/taxonomies/additives.txt
+
+Size: 1,114,164 bytes.
+
+SHA-256: `e30d5ad1316f8ecdaa3710e7b1632437c206843d0f9ff28b2068cd2dfe5445a5`
+
+Source project: https://openfoodfacts.org and
+https://github.com/openfoodfacts/openfoodfacts-server.
+
+Open Food Facts publishes its database under the Open Database License (ODbL)
+1.0 and individual database contents under the Database Contents License:
+https://world.openfoodfacts.org/data and
+https://world.openfoodfacts.org/terms-of-use.
+Attribution and applicable share-alike obligations must be retained on reuse.
+
+The importer reads English entries with concrete E-numbers, including letter
+and Roman-numeral subtypes. It skips wildcard codes and generic taxonomy
+categories without an E-number. Escaped commas in synonyms are preserved.
+Entries without an English descriptive name use their E-number as the name.
+Names shared by distinct E-numbers are disambiguated with the code in the
+database, while the original name remains an alias.
+
+The snapshot supplies detection vocabulary only. It does not supply AddiGuard
+risk factors; the five placeholder ratings remain solely in `seed.py`.
+All newly imported additives have null risk factors.
+
+To refresh, download the raw URL to `additives.txt`, update this provenance and
+checksum, run importer and label tests, then restart the running API after
+importing so that other processes discard their cached alias indexes.