I'm working on a project where I need to overlay a 3D model onto a detected object in real-time using computer vision. Specifically, I want to detect the presence of a glass in a camera feed and overlay a 3D model of the glass onto it. Additionally, I want to use the overlaid 3D model to identify missing components in the real glass, such as handles.
I've explored using techniques like YOLO for object detection and Unity3D for rendering the 3D model, but I'm unsure of how to proceed with the overlay and component identification process.
Could someone please provide guidance on how to achieve this? Specifically, I'm looking for suggestions on:
How to accurately overlay a 3D model onto a detected object in real-time.
Techniques or algorithms for identifying missing components in the real object using the overlaid 3D model.
Any libraries, frameworks, or tools that could assist with implementing this functionality.
Any help or suggestions would be greatly appreciated. Thank you!
Object Detection with YOLO: I have successfully implemented object detection using YOLO to detect the presence of a glass in the camera feed. This step provides me with the bounding box coordinates of the detected glass.
Rendering 3D Model with Unity3D: I have created a 3D model of the glass using Unity3D and have been able to render it onto the camera feed. However, I'm struggling with accurately overlaying the 3D model onto the detected glass in real-time.
Expectations:
Accurate Overlay: I was expecting that by utilizing the bounding box coordinates obtained from object detection, I could easily overlay the 3D model onto the detected glass in real-time. However, I'm facing challenges in ensuring that the 3D model aligns correctly with the detected glass and maintains its position and orientation as the camera or the glass moves.
Component Identification: Additionally, I was hoping to leverage the overlaid 3D model to identify missing components in the real glass, such as handles. I'm unsure of the best approach or algorithms to achieve this component identification task.
Overall, I'm seeking guidance on how to improve the accuracy of the overlay and implement component identification using the overlaid 3D model.
I have experience with this field.
I have ever done Shoes virtual try-on project and that was the mobile application project. Of course the back-end was computer vision technique.
For this project I used 3D object key points detection technique. All of shoes models should have 3D key points information and after detecting the 3D keypoints of shoes that peope is wearing, and then matching those key points.
The result was successful. The issue of this application was on running time. It takes little bit long time for detecting, matching and wearing steps. It takes about 4~7 seconds.
Hope my answer would be helpful for you.
Thank you