Dual Pixel 简介
双像素是成像系统的感光元器件中单帧同时生成的图像：通过双像素可以实现：深度估计、图像去模糊去雨去雾恢复、图像重建
成像原理来源如上，也有遮罩等方式的pd生成，如图双像素视图可以看到光圈的不同一半，这提供了一个深度提示。然而，由于基本的模糊性，如果相机的焦距（或光圈大小或焦距）发生变化，不同的场景可能会产生相同的双像素图像。在（a）中，具有焦距g1的相机在距离Z1处成像聚焦的蓝色点和离焦的橙色点。通过光圈左半部分折射的光（深蓝色和橙色光线）到达每个双像素的右半部分，反之亦然。这导致了一个双像素图像，其中失焦橙色点被d像素（a，“DP数据”）位移，被b像素模糊（a，”图像”）。在（b）中，不同的焦距和场景深度集产生相同的双像素和RGB图像。然而，如文中所示，该场景通过逆深度上的仿射变换与（a）中的场景相关。消费者迅速采用。双像素相机似乎代表了更雄心勃勃的光场相机和传统相机之间的一种有前景的折衷方案——DP相机牺牲了可忽略的空间分辨率来采样光场中的两个角度，而真正的单眼相机只采样一个角度，Lytro Illum等光场相机以牺牲显著的空间分辨率为代价采样196个角度。因此，它们在消费类相机和内窥镜等空间受限的应用中得到了更广泛的采用[6]。
消费类硬件的最新发展可能为深度估计的新方法提供机会。最近，通过使用密集的双像素（DP）传感器（图2），可以使用一台相机同时捕获两张类似于具有微小基线的立体对的图像（图1）。虽然这项技术最初是为相机自动对焦而开发的，但双像素图像也可以用来从单个相机中恢复密集的深度图，从而消除了对额外硬件、校准或同步的任何需求。例如，Wadhwa等人[55]使用经典的立体技术（块匹配和边缘感知平滑）从DP数据中恢复深度。但如图1所示，传统立体技术可以生成的深度图的质量是有限的，因为DP图像中视差和焦点之间的相互作用可能会导致经典立体匹配技术失败。现有的基于单眼学习的技术在这方面也表现不佳
接下来就介绍双像素相关的sota论文和代码，长期更新教学
Table of contents
- DualPixel
- Dateset
Dual Pixel
| Year | Pub | Paper | App | Repo |
|---|---|---|---|---|
| 2018 | SIGGRAPH | Synthetic Depth-of-Field with a Single-Camera Mobile Phone | Depth / Segmenation / Synthetic DoF | |
| 2019 | CVPR | Reflection Removal Using a Dual-Pixel Sensor | Reflection Removal | Code |
| 2019 | ICCV | Learning Single Camera Depth Estimation using Dual-Pixels | Depth | Code & Dataset |
| 2020 | ICCP | Modeling Defocus-Disparity in Dual-Pixel Sensors | Depth | Code & Dataset |
| 2020 | CVPR | Learning to Autofocus | Autofocus | Dataset |
| 2020 | ECCV | Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels | Disparity (w/Stereo) | |
| 2020 | ECCV | Defocus Deblurring Using Dual-Pixel Data | Deblur | Code & Dataset |
| 2021 | CVPR | Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration | Depth / Deblur | Code & Dataset |
| 2021 | CVPRW | NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results | Deblur | |
| 2021 | CVPRW | ATTSF Attention! Stay Focus! | Deblur | Code |
| 2021 | ICCV | Defocus Map Estimation and Deblurring From a Single Dual-Pixel Image | Deblur / Defocus Map | Code & Dataset |
| 2021 | ICCV | Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data | Deblur | Code & Dataset |
| 2021 | IEEE | World Largest Mobile Image Sensor with All Directional Phase Detection Auto Focus Function | Depth | |
| 2021 | ICTC | Disparity probability volume guided defocus deblurring using dual pixel data | Deblur | |
| 2021 | Journal of Electronic Imaging | Defocus deblurring: a designed deep model based on CNN | Deblur | |
| 2021 | IEEE | All-Directional Dual Pixel Auto Focus Technology in CMOS Image Sensors | Circuits | |
| 2022 | WACV | Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning | Deblur | Code & Dataset |
| 2022 | ISSCC | A 1/1.57-inch 50Mpixel CMOS Image Sensor With 1.0μm All-Directional Dual Pixel by 0.5μm-Pitch Full-Depth Deep-Trench Isolation Technology | Circuits | |
| 2022 | ECCV | Facial Depth and Normal Estimation using Single Dual-Pixel Camera | Depth / Surface Normal / Anti-spoofing / Relighting | Code & Dataset |
| 2023 | CVPR | K3DN: Disparity-Aware Kernel Estimation for Dual-Pixel Defocus Deblurring | Deblur | |
| 2023 | CVPR | Spatio-Focal Bidirectional Disparity Estimation From a Dual-Pixel Image | Depth / Disparity | Code |
| 2023 | ICCP | Learning to Synthesize Photorealistic Dual-pixel Images from RGBD frames | Simulator | Code & Dataset |
| 2023 | ICCV | Exploring Positional Characteristics of Dual-Pixel Data for Camera Autofocus | Autofocus |
这其中
| 2019 | ICCV | Learning Single Camera Depth Estimation using Dual-Pixels | Depth |
网络的代码被复现在：DualPixelFace/src at main · MinJunKang/DualPixelFace · GitHub ；GitHub - vyi/PdCapture: [google-research/dual-pixels](Forked from https://github.com/google-research/google-research/tree/master/dual_pixels)
GitHub - RugvedKatole/Learning-Single-Camera-Depth-Estimation-using-Dual-Pixels: This Repo is an implementation of paper titled "Learning Single Camera Depth Estimation using Dual-Pixels"
Dataset
| Year | Pub | Paper | Detail |
|---|---|---|---|
| 2019 | ICCV | Learning Single Camera Depth Estimation using Dual-Pixels | Train:2506, Test:684, Res:1512x2016(DP), 16bit png, DP Raw / Depth |
| 2020 | ICCP | Modeling Defocus-Disparity in Dual-Pixel Sensors | Num:100, Res:5180x2940, RGB 8bit jpg / 16bit tif Depth, DP LR / Depth |
| 2020 | ECCV | Defocus Deblurring Using Dual-Pixel Data | Num:500, Res:1680x1120, 16bit, Used for NTIRE 2021 Challenge (CVPRW) |
| 2021 | CVPR | Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration | DP Simulator from NYUD Dataset |
| 2021 | ICCV | Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data | DP Simulator form SYNTHIA-SF dataset |
| 2023 | ICCP | Learning to Synthesize Photorealistic Dual-pixel Images from RGBD frames | Num:5130, Res:1680x1120, RGB(DP LRC) 8bit png / Depth 16bit png |
小结
欢迎AI深度估计、AI重建、AI恢复图像：图像去雾去雨去模糊降噪方向的同学一起交流，cv君正在打造最大最前沿的AI社区，让你在浓厚的学习氛围中进步，让你在飞速发展的时代中砥砺前行；免费进群交流请私信cv君（免费）