Feat: Add Real-time Object Detection with MediaPipe - #108
Open
kkbin505 wants to merge 1 commit into
Open
Conversation
- Replace main SurfaceView with TextureView to support capturing frame Bitmaps via getBitmap(). - Add ObjectDetectorHelper to encapsulate MediaPipe Object Detector Task API with CPU/GPU delegates. - Create OverlayView for scaling and rendering bounding boxes and classification labels onto standard video stream. - Integrate background detection thread execution loop (~10 FPS) in VideoActivity with lifecycle management. - Set up automated downloads for EfficientDet-Lite0 and EfficientDet-Lite2 model assets in app/build.gradle. - Add user menu toggles for object detection settings, persisting config values via SharedPreferences. - Update README.md with detailed instructions for the object detection feature. Reference: https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/object_detection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Inspired by MarioFPV, I added a object detection function to OpenIPC. Instead of adding this function to the airunit, I added this funtion to the receiver end (PixelPilot) with Google MediaPipe Tasks Vision:
https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/object_detection
Users can now enable detection overlay to visualize bounding boxes, classification labels, and confidence scores directly on top of the live video stream.
Key Changes
1. Build & Dependencies
mediapipe(0.10.29) andgradleDownload(5.5.0) dependencies.downloadModelFile0,downloadModelFile1) to automatically fetchefficientdet-lite0.tfliteandefficientdet-lite2.tflitemodels duringpreBuild.noCompress 'tflite'to optimize model loading performance.2. UI & Rendering Setup
mainVideofromSurfaceViewtoTextureViewto support runtime frame grabbing viagetBitmap(). AddedOverlayViewabove it.configureTextureView()to adapterTextureViewlifecycle events (such as surface creation and destruction) for native decoding.3. Detection Core & Integration
onResume/onPause) to ensure safe thread termination.Manual Verification