A Hybrid Deep Learning Model For 3D Scene Alignment And Reconstruction With 3D U-Net Network

Naveen Kumar N; Giribabu Kande; B Prabhakar Rao

doi:10.29284/y3xe3q38

Authors

Naveen Kumar N Research Scholar, Department of ECE, JNTUK, Kakinada, 533003, Andra Pradesh, India
Giribabu Kande Professor, ECE Department, VVIT, Nambur,522509, Andra Pradesh, India
B Prabhakar Rao Professor, ECE Department, JNTUK, Kakinada, 533003, Andra Pradesh, India

DOI:

https://doi.org/10.29284/y3xe3q38

Keywords:

U-Net, Encoder, Decoder, 3D Reconstruction, IoU, ShapeNet

Abstract

Reconstructing 3D geometry from multi-view 2D RGB imagery has emerged as a key research focus due to its wide applicability in areas such as virtual reality, robotic perception, and human–computer interaction. This work presents a hybrid deep learning framework that integrates 3D U-Net with transformer-based attention mechanisms. The architecture consists of a visual autoencoder and a dedicated 3D refinement network. Feature representations extracted from unordered image sets are aggregated through a self-attention module to produce a rich global feature map, which is decoded into an initial volumetric 3D representation and subsequently refined for improved structural accuracy. The proposed model supports 3D reconstruction from RGB inputs captured from arbitrary viewpoints. Experimental validation on ShapeNet and PASCAL datasets demonstrates that the proposed method consistently surpasses existing state-of-the-art approaches, achieving superior mean Intersection over Union (IoU) performance.

References

1. R. Hartley and A. Zisserman, Multiple view geometry in computer vision. Cambridge university press, 2003.

2. A. Laurentini, “The visual hull concept for silhouette-based image understanding,” IEEE TPAMI, vol. 16, no. 2, pp. 150–162, 1994.

3. C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. D. Reid, and J. J. Leonard."Past, Present, and Future of Simultaneous Localization and Mapping: Towards the Robust-Perception Age.", IEEE Transactions on Robotics, 32(6):1309–1332, 2016.

4. O. Ozyesil, V. Voroninski, R. Basri, and A. Singer."A Survey of Structure from Motion.", Acta Numerica, 26:305–364, 2017.

5. Li Yi , Lin Shao , Manolis Savva , Haibin Huang , Yang Zhou , Qirui Wang ,Benjamin Graham , Martin Engelcke , Roman Klokov , Victor Lempitsky , Yuan Gan ." arge Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55". ECCV 2017

6. Y. Xiang, R. Mottaghi, and S. Savarese." Beyond PASCAL:A benchmark for 3D object detection in the wild. In WACV 2014".

7. Abhishek Kar, Christian Häne, Jitendra Malik." Learning a Multi-View Stereo Machine"., NIPS 2017.

8. Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, Jia-Bin Huang."DeepMVS: Learning Multi-view Stereopsis"., CVPR 2018.

9. Thibault Groueix, Matthew Fisher, Vladimir G Kim,Bryan C Russell, and Mathieu Aubry. A papier-mach ˆ e approach to learning 3d surface generation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 216–224, 2018.

10. Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Eur. Conf. Comput. Vis.,2018.

11. Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Int. Conf. Comput.Vis., pages 9964–9973, 2019.

12. Edward J Smith, Scott Fujimoto, Adriana Romero, and David Meger. Geometrics: Exploiting geometric structure for graph-encoded objects. arXiv preprint arXiv:1901.11461, 2019.

13. Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Int. Conf. Comput. Vis., pages 9785–9795, 2019.

14. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1912–1920, 2015.

15. Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Eur. Conf. Comput. Vis., pages 628–644. Springer, 2016.

16. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.

17. Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Adv. Neural Inform. Process. Syst., pages 82–90, 2016.

18. David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1955–1964, 2018.

19. Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. arXiv preprint arXiv:1706.07036, 2017.

20. Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Int. Conf. Comput. Vis., pages 4541–4550, 2019.

21. Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Trans. Graph., 36(4):1–14, 2017.

22. Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10344–10353, 2019.

23. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4460–4470, 2019.

24. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. arXiv preprint arXiv:1901.05103, 2019.

25. Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711, 2019.

26. Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Int. Conf. Comput. Vis., pages 2304–2314, 2019.