FarmOS

QuackHacks 2026 — Precision agriculture platform powered by Gemini AI and real-world farm data.

FarmOS gives farmers AI-powered, real-time visibility into crop health, weed threats, and soil conditions — rendered inside a photorealistic 3D reconstruction of their actual farm.

We went in person to real farms to capture on-site data. Using 3D Gaussian Splatting, we simulate drone flight paths through the 3D environment and extract frames. Those frames go through a two-step Gemini pipeline: first to detect bare soil regions, then to find weeds growing within those soil crops. By combining the saved 3D drone coordinates with the pixel locations Gemini flags, we place weed markers at exact real-world positions inside the 3D viewer — letting farmers digitally traverse their fields before ever stepping outside.

---

Features

3D Farm Viewer

A real-world farm environment captured on-site and reconstructed as a 3D Gaussian Splat (.ply point cloud). Rendered via a custom WebGL viewer in the browser. Supports two camera modes:

• Top-down — default dashboard view
• Perspective / free-roam — interactive 3D exploration via /3d route

Live Farm Dashboard

The main interface (/) is a HUD overlay on the 3D viewer with three panels:

• Agent Log + Problems — real-time feed of AI agent activity, anomaly alerts with severity badges, and detected issues (weeds, pests, sprinkler faults)
• Ambient Conditions — live temperature, humidity, UV index, and solar radiation pulled from Firebase
• Soil Health Summary + Report Preview — condensed scores and a link to the full AI report

Weed Identification & Treatment Dispatch

Detected weeds (e.g., Pigweed, Crabgrass) surface as cards with botanical details — characteristics, seasonal behavior, crop impact. From the card, you can:

1. Choose a treatment method (5+ options including precision herbicide, manual removal, etc.)
2. Dispatch an autonomous farm robot to execute the treatment in-field

Two-Step Weed Detection Pipeline

The core detection runs via scripts/detect.py using Gemini 3.0 Flash:

Drone frames (simulated flight path through Gaussian Splat)
      │
      ▼
Step 1: Full frame → Gemini → bounding boxes of bare soil regions
      │
      ▼
Step 2: Each soil crop → Gemini → bounding boxes of weeds within soil
      │
      ▼
Step 3: Weed pixel coords + saved 3D camera position/quaternion/FOV
        → ray-cast to ground plane → real-world XYZ marker position
      │
      ▼
anomalies.json → loaded by frontend → markers placed in 3D viewer

Each frame goes through saturation enhancement first, then Gemini identifies soil strips (brown/tan tilled rows), then each soil crop is re-analyzed for green leafy growth on the brown soil. The weed center pixel is unprojected through the camera's saved 3D transform to produce a world-space position, with a calibrated offset applied for left-angle frames.

AI Farm Analysis Report (/report)

A full markdown report generated by Gemini 2.5 Flash using live Firebase data as context:

• Overall farm health score (0–100)
• Per-category scores: Moisture, Nutrients, Temperature, Weed Control
• Anomaly count, soil health index, and actionable recommendations
• Cached locally; regenerable on demand with a single click

Soil Sensor Dashboard (/soil)

An interactive field-level heatmap across 40 IoT sensors on an 8×5 grid (~800m × 1000m coverage):

• 5 layers: Moisture %, Nitrogen (ppm), Phosphorus (ppm), Potassium (ppm), pH
• IDW interpolation — smooth per-pixel field estimates from sparse sensor points
• Hover tooltips — individual sensor readings with GPS coordinates
• Sidebar with 7-day weather forecast and ambient conditions

Natural Language Agents

Three separate agent interfaces powered by Google ADK (Agents Development Kit):

Agent	Route	Capability
Research Agent	/api/agent/research	Live Google Search + Gemini synthesis for any agricultural question
Operations Agent	/api/ops-agent	Query inventory, field records, equipment status, soil sensors, market prices, weather
Robot Agent	/api/robot	Command autonomous farm robot — navigate, pull weeds, precision spray

Flight Path Calibration

An optional in-app calibration mode lets you define a drone flight path directly on the 3D top-down view by picking waypoints for left and right crop rows. The resulting path is saved as flight-path.json and persists across sessions.

---

Tech Stack

Layer	Technology
Frontend	React 18 + TypeScript, Vite, React Router 7
Styling	Tailwind CSS 4
3D Rendering	Three.js, @react-three/fiber, @react-three/drei
Gaussian Splatting	@mkkellogg/gaussian-splats-3d (WebGL)
Charts	Recharts
Markdown	react-markdown + remark-gfm
Icons	Lucide React
Database	Firebase Firestore (real-time sensor + event data)
Backend	FastAPI (Python), Uvicorn
AI Models	Google Gemini 3.0 Flash (weed detection pipeline), Gemini 2.5 Flash (reports), Gemini 2.0 Flash (agents)
Agent Framework	Google ADK (Agents Development Kit)
Image Processing	OpenCV, Pillow, NumPy
3D File Handling	plyfile (PLY point cloud format)
HTTP Client	httpx (async)
Config	python-dotenv

---

Project Structure

farm/
├── frontend/
│   ├── public/
│   │   ├── scene.ply              # 3D Gaussian splat point cloud
│   │   ├── scene_clean.ply        # Pruned splat (filtered by opacity/scale)
│   │   └── flight-path.json       # Calibrated drone flight waypoints
│   └── src/
│       ├── App.tsx                # Routing (/ /3d /soil /report /review)
│       ├── Overlay.tsx            # Main dashboard HUD, calibration UI
│       ├── ThreeDViewer.tsx       # Interactive 3D camera mode
│       ├── Report.tsx             # AI report page (markdown + score cards)
│       ├── SoilDashboard.tsx      # Soil heatmap full-page view
│       ├── WeedMarkers.tsx        # Weed overlays on splat viewer
│       ├── ProblemMarkers.tsx     # Anomaly markers + drone animation
│       ├── components/
│       │   ├── NavBar.tsx
│       │   ├── GaussianSplatScene.tsx
│       │   ├── SoilHeatmap.tsx    # IDW heatmap canvas (40-point grid)
│       │   ├── SoilSensorOverlay.tsx
│       │   ├── DroneScanner.tsx
│       │   └── PathEditor.tsx
│       └── lib/
│           ├── firebase.ts        # Firebase SDK init
│           └── db.ts              # Typed Firestore query layer
├── backend/
│   ├── main.py                    # FastAPI app + route mounting
│   ├── gemini_logic.py            # Dual-agent analysis + report generation
│   ├── heatmap_pipeline.py        # Top-down image sharpness/focal extraction
│   └── agents/
│       ├── farm_agent.py          # Research agent (Google Search + Gemini)
│       ├── farm_ops_agent.py      # Operations agent (inventory, equipment, etc.)
│       └── farm_robot_agent.py    # Robot command agent
└── scripts/
    ├── daily_poll.py              # Daily Firebase poll + soil health computation
    ├── analyze.py                 # Photo batch analysis utility
    ├── detect.py                  # Weed/pest detection
    ├── cluster.py                 # Anomaly clustering
    └── capture.js                 # Drone frame capture manifest generation

---

Firebase Collections

Collection	Contents
soilSensors	40 sensor docs — lat/lng, moisture, NPK, pH
agentLogs	Real-time agent activity feed
problems	Detected anomalies (weed, pest, sprinkler faults)
weeds	Weed identification database with botanical details
anomalies	Historical anomaly records with drone action taken
soilHealth	Daily computed health index (moisture, nutrient, pH, weed pressure scores)
config/ambientConditions	Live temp, humidity, UV, solar radiation
config/forecast	7-day weather forecast

---

Getting Started

Prerequisites

• Node.js >= 18
• Python >= 3.10
• Google Gemini API key (from Google AI Studio)
• Firebase project with Firestore enabled

Backend

cd backend

Create a .env file:

GEMINI_API_KEY=your_key_here

# Optional: connect to real farm APIs (mock fallbacks used if unset)
FARM_INVENTORY_API_URL=https://erp.myfarm.com/api
FARM_FIELD_API_URL=https://fms.myfarm.com/api
FARM_EQUIPMENT_API_URL=https://telematics.myfarm.com
FARM_SENSOR_API_URL=https://iot.myfarm.com/api
FARM_DRONE_API_URL=https://drones.myfarm.com/api
FARM_MARKET_API_URL=https://market-data.agri.com/api
FARM_WEATHER_API_URL=https://wx.myfarm.com/api
FARM_API_KEY=your_farm_api_key

python -m venv venv
venv\Scripts\activate      # Windows
# source venv/bin/activate  # macOS/Linux
pip install -r requirements.txt
uvicorn main:app --reload

API available at http://localhost:8000.

Frontend

Create frontend/.env.local:

VITE_FIREBASE_API_KEY=...
VITE_FIREBASE_AUTH_DOMAIN=...
VITE_FIREBASE_PROJECT_ID=...
VITE_FIREBASE_STORAGE_BUCKET=...
VITE_FIREBASE_MESSAGING_SENDER_ID=...
VITE_FIREBASE_APP_ID=...

cd frontend
npm install
npm run dev

App available at http://localhost:5173.

---

API Endpoints

Method	Path	Description
GET	/api/health	Health check
POST	/api/analyze	Upload 1–20 farm photos for dual-agent AI analysis
GET	/api/report	Generate (or return cached) AI farm health report
POST	/api/save-flight-path	Save drone flight path calibration
GET	/api/manifest	Retrieve drone frame capture manifest
POST	/api/cleanup-splat	Prune PLY point cloud by opacity/scale thresholds
POST	/api/process-topdown	Extract in-focus strip from top-down image
POST	/api/agent/research	Natural language farm research (live Google Search)
POST	/api/ops-agent/query	Query farm operations data
POST	/api/robot/command	Send command to autonomous farm robot

---

Built at QuackHacks 2026