RGBD-Based Produce Recognition System with Dynamic Depth-Aware Cropping

Authors: 
Tim Crockett, Tuyen Bui

Abstract:
This disclosure proposes using an RGBD camera to provide both color and depth data, enabling dynamic depth-based masking and improved recognition accuracy.

Background:
TGCS Produce Recognition currently uses an RGB camera to capture images, which are processed by firmware and passed into an inference engine for classification. RGB-only imaging suffers from limitations that reduce recognition accuracy, including distance sensitivity, background clutter, and difficulty recognizing bagged produce. An RGB camera produces an N x M array of pixels, each containing red, green, and blue intensity values. These images are processed by neural networks for classification.
However, RGB-only systems suffer from issues such as distance sensitivity, background interference, and reliance on fixed positioning for cropping.

Description:
The system uses an RGBD camera to capture both RGB images and depth maps.
Depth data is used to create dynamic segmentation masks that isolate objects within the operating region.

Depth-aware segmentation allows accurate separation of produce from background clutter.
Dynamic cropping improves recognition consistency regardless of object placement.
Combining RGB and depth data enhances neural network classification performance.
Depth data also improves recognition of bagged produce by distinguishing contours.
 

Usages:

1. Self-checkout produce recognition
2. Cashier-assisted checkout lanes
3. Smart produce scales (automatic PLU entry)
4. Bagged produce recognition
5. Conveyor belt checkout systems
6. Loss prevention / shrink reduction
7. Backroom sorting and inventory handling
8. Customer-facing kiosks for produce identification
9. Multi-item detection and counting
10. General object isolation in cluttered retail environments

Claims:

1. Multimodal Imaging for Recognition
   A produce recognition system that utilizes an RGBD camera to capture both color (RGB) and depth (D) data simultaneously for each frame.
2. Depth-Augmented Inference Pipeline
   Integration of depth information into the inference pipeline to enhance object recognition accuracy compared to RGB-only systems.
3. Dynamic Depth-Based Segmentation
   A method of using depth maps to segment objects in real time by distinguishing foreground items from background elements based on distance.
4. Adaptive Cropping Using Depth Data
   Generation of a dynamic cropping mask derived from depth thresholds to isolate objects within a defined operating region without requiring fixed camera calibration.
5. Robust Recognition Across Distance Variation
   Improved recognition performance regardless of object distance from the camera by leveraging depth information to normalize spatial variability.
6. Background Noise Reduction
   Suppression of interference from adjacent objects (e.g., items on a conveyor belt) through depth-based filtering and masking.
7. Improved Recognition of Bagged Produce
   Use of depth contour data to differentiate produce from enclosing materials (such as plastic bags), improving classification accuracy.
8. Reduced Dependence on Operator Interaction
   Minimization of manual repositioning requirements by enabling consistent recognition across different positions on the scanning surface.
9. Environment-Independent Operation
   Elimination of dependence on fixed spatial relationships between camera and scanning surface by dynamically identifying the region of interest using depth.
10. Enhanced Neural Network Input Representation
    Fusion of RGB data with depth-derived features (e.g., masks or contour maps) as input to a neural network trained for multimodal recognition.
11. Real-Time Processing Capability
    Implementation of the above techniques in firmware and/or embedded systems capable of processing RGBD data streams in real time.

TGCS Reference 4051