自己怎么做logo_建设企业网站报价_百度问问首页_滨州seo招聘

ABSTRACT

Using three-dimensional structured light to measure high-reflective objects accurately is a big challenge. Some overexposed and blurred areas in the structured light fringe patterns lead to the loss of semantic and texture information on the object surface, further leading to 3D model reconstruction errors. To solve this problem, a lightweight novel network, Deformable Convolutional and Multi-scale Convolutional Network (DcMcNet), is proposed for repairing highly reflectively distorted regions in sinusoidal fringe patterns and the Gray code binary fringe patterns. DcMcNet uses deformable convolution and multi-scale convolution to realize the effective utilization of multiscale global features in repairing highly reflective distorted regions and can adapt well to the different shapes of these regions. In addition, virtual software is used to acquire large-scale and high-fidelity datasets and a diverse loss function is constructed to better optimize DcMcNet. Depth reconstruction experiments of sinusoidal fringe patterns and point cloud reconstruction experiments of Gray code binary fringe patterns show that our method can repair highly reflective distortion regions with high accuracy, and for aeronautical blades, the MAE of the depth map is reduced from 0.1608 to 0.0675, and the point cloud coverage is improved from 76.64% to 98.95%.

使用三维结构光精确测量高反射物体是一个很大的挑战。结构光条纹图案中的一些过度曝光和模糊区域导致物体表面的语义和纹理信息丢失，进一步导致3D模型重建错误。为了解决这个问题，提出了一种轻量级新型网络——可变形卷积和多尺度卷积网络（DcMcNet），用于修复正弦条纹图案和格雷码二进制条纹图案中的高反射畸变区域。 DcMcNet利用可变形卷积和多尺度卷积实现了多尺度全局特征在修复高反射畸变区域中的有效利用，并且能够很好地适应这些区域的不同形状。此外，利用虚拟软件获取大规模高保真数据集，构建多样化的损失函数以更好地优化DcMcNet。正弦条纹图的深度重建实验和格雷码二值条纹图的点云重建实验表明，我们的方法可以高精度修复高反射畸变区域，对于航空叶片，深度图的MAE从0.1608降低到0.0675，点云覆盖率从76.64%提高到98.95%。

1.Introduction

…
Without a doubt, when using SL to measurehigh-reflective objects, it is necessary to capture high-quality and non-highly reflective sinusoidal and Gray code binary fringe patterns. The most commonly used traditional method is to spray white powder on the target object, but this method is both time-consuming and cumbersome. Some hardwareassisted methods, such as adding polarizers to the optical path [7–9], are also commonly used, but the control of these hardware devices is complicated. Later, multiple exposure methods [10] and adaptive fringe projection techniques [11–12] emerged, in which fringe patterns with different brightness can be obtained by adjusting the camera or pro jector parameters, and then distortionless fringe patterns can be ob tained by image fusion. However, traditional methods require additional hardware assistance or multiple redundant shots, which are less efficient and cannot be used for rapid detection.

Based on the references investigated above, it is worth mentioning that the current fringe restoration networks are designed specifically for sinusoidal fringe patterns, and there is no network designed for Gray code binary fringe patterns. Therefore, in this paper, a neural network for restoring highly reflective regions in sinusoidal fringe patterns and Gray code binary fringe patterns is proposed. UNet [21] is a commonly used network structure in the field of image restoration [22], but it has limited accuracy in restoring SL fringe patterns due to the fixed geometries and scales of its convolution kernels. Therefore, Deformable Convolution (DC) [23] and Multi-scale Convolution (MC) [24] are introduced to replace the ordinary convolution in UNet, and a novel network called DcMcNet is constructed which is more suitable for restoring the highly reflective fringe patterns in SL. DC can adapt the convolutional kernel shape to better handle the various shapes of highly reflective regions caused by inhomogeneous high reflectance, and MC can use convolution with different scales to balance the attention to local details and overall structure when restoring highly reflective regions. Restoration of highly reflective fringe patterns by DcMcNet can greatly improve the quality of the input for SL measurements on highly reflective objects. In addition, this paper proposes a method for generating large-scale and high-quality datasets through the virtual software 3ds Max and designs diverse loss functions.

基于上述研究的文献，值得一提的是，当前的条纹恢复网络是专门针对正弦条纹图案设计的，并且没有针对格雷码二进制条纹图案设计的网络。因此，本文提出了一种用于恢复正弦条纹图案和格雷码二进制条纹图案中高反射区域的神经网络。 UNet[21]是图像恢复领域常用的网络结构[22]，但由于其卷积核的几何形状和尺度固定，它在恢复SL条纹图案方面的精度有限。因此，引入变形卷积（DC）[23]和多尺度卷积（MC）[24]来代替UNet中的普通卷积，并构建了一种称为DcMcNet的新型网络，该网络更适合恢复高反射条纹图案在 SL 中。 DC可以调整卷积核形状以更好地处理由于不均匀的高反射率而导致的高反射区域的各种形状，而MC可以使用不同尺度的卷积来平衡恢复高反射区域时对局部细节和整体结构的关注。通过 DcMcNet 恢复高反射条纹图案可以极大地提高高反射物体上 SL 测量的输入质量。此外，本文提出了一种通过虚拟软件3ds Max生成大规模高质量数据集的方法，并设计了多种损失函数。

To better organize this paper, the SL reconstruction process and the principles of high-reflection fringe patterns have been introduced in Section 2. The DcMcNet network architecture and the loss functions have been provided in Section 3. Based on that, the dataset generation method has been described in detail in Section 4. Further, experimental results and discussion have been presented in Section 5.

为了更好地组织本文，第 2 节介绍了 SL 重建过程和高反射条纹图案的原理。第 3 节提供了 DcMcNet 网络架构和损失函数。基于此，数据集生成方法第 4 节详细描述了这一点。此外，第 5 节介绍了实验结果和讨论。

2.Basic principle

2.1. The principle of 3D model reconstruction by using structured light

To build a SL system, as shown in Fig. 1 (a), a projectoris required to illuminate the object using pre-coded fringe patterns, while a camera simultaneously captures the images. The inherent encoding information in the captured patterns serves as a temporary texture of the object, which can be converted into a 3D depth map or point cloud. In recent years, various algorithms have appeared for SL reconstruction, and different algorithms need to collect different types of fringe patterns. The following two algorithms are introduced as examples. (1) As shown in Fig. 1 (b), whensinusoidal fringe patterns are projected on the object, phase calculations are required because the phase value in the sinusoidal function is essential for accurate depth reconstruction. The commonly used algorithm is to calculate the wrapped phase by N-step phase shifting and phase unwrapping by M frequency heterodyne [25]. Therefore, a total of N timesM fringe patterns with different phase shifts and frequencies need to be projected and collected. (2) As shown in Fig. 1 ( c), 3D reconstruction can also be performed using a sufficient number of Gray code binary fringe patterns [26]. This approach involves encoding each pixel in the image with Gray code and then utilizing a stereo-matching strategy similar to binocular reconstruction to establish a matching relationship between camera pixels and projector pixels. In the final 3D reconstruction of these two methods, the phase height method is used to directly convert the calculated phase into a 3D depth map, or the matchingrelationship between the projector and the camera is used to calculate the 3D point cloud by triangulation method [6], as shown in Fig. 1 (d).

为了构建 SL 系统，如图 1 (a) 所示，需要投影仪使用预编码的条纹图案照亮物体，同时相机捕获图像。捕获的图案中的固有编码信息充当对象的临时纹理，可以将其转换为 3D 深度图或点云。近年来，出现了各种用于SL重建的算法，并且不同的算法需要收集不同类型的条纹图案。下面以两种算法为例进行介绍。 (1)如图1(b)所示，当正弦条纹图案投影在物体上时，需要进行相位计算，因为正弦函数中的相位值对于准确的深度重建至关重要。常用的算法是通过N步相移计算包裹相位，通过M频外差计算相位展开[25]。因此，总共需要投影和采集N×M个具有不同相移和频率的条纹图案。 (2)如图1( c)所示，也可以使用足够数量的格雷码二进制条纹图案进行3D重建[26]。该方法涉及使用格雷码对图像中的每个像素进行编码，然后利用类似于双目重建的立体匹配策略来建立相机像素和投影仪像素之间的匹配关系。这两种方法最终的3D重建中，都是利用相位高度法直接将计算出的相位转换为3D深度图，或者利用投影仪和相机之间的匹配关系通过三角测量法计算3D点云[6]如图1(d)所示。

2.1.1. The principle of the N-step phase-shifting algorithm

The N-step phase-shifting algorithm, known for its high precision, insensitivity to ambient light, and per-pixel phase measurement, has been widely applied in phase calculation. The N-step phase-shifting fringe patterns captured by the camera can be represented as:

N步相移算法以其精度高、对环境光不敏感、逐像素相位测量等优点在相位计算中得到了广泛的应用。相机捕获的N步相移条纹图案可以表示为：

Where $I_i$ represents i-step phase-shifting fringe pattern, $A$ and $B$ represent the background and modulated light intensity, and $\phi$ represents the phase value. ${\delta }_i=2\pi (i-1)/N$ represents the phase shift, where $N$ is the number of phase shift steps. The final calculated phase is as follows, and the derivation process can be seen in reference [26].

其中$I_i$表示$i$步相移条纹图案，$A$和$B$表示背景和调制光强度，$\phi$表示相位值。 ${\delta }_i=2\pi (i-1)/N$ 表示相移，其中 $N$ 是相移步数。最终计算的相位如下，推导过程参见参考文献[26]。

2.1.2. The principle of Gray code algorithm

Gray code is a binary coding method used in fringe patterns, where black regions are encoded as 0, and white regions are encoded as 1. The n-order Gray code binary fringe pattern can be divided into $2^n$ regions, each of which corresponds to a unique Gray code value. Through operations such as binarization, bitwise XOR, and numerical expansion, the unique identifier for each region can be calculated. Taking a 4-order Gray code binary fringe pattern as an example, as shown in Fig. 2, the image can be divided into 16 regions, each encoded with gray codes ranging from 0000 to 1111, which can be transformed into unique identifiers from 1 to 16 through calculations.

格雷码是一种二进制编码方法，用于条纹图案中，黑色区域编码为 0，白色区域编码为 1。n 阶灰度编码二进制条纹图案可分为 $2^n$ 个区域，每个区域对应一个唯一的灰度编码值。通过二值化、比特 XOR 和数字扩展等操作，可以计算出每个区域的唯一标识符。以图 2 所示的 4 阶格雷码二进制边缘模式为例，图像可被划分为 16 个区域，每个区域的格雷码编码范围为 0000 至 1111，通过计算可将其转换为 1 至 16 的唯一标识符。

The low-order Gray code algorithm [27] is usually used for phase unwrapping, providing unique identifiers to regions corresponding to different fringe orders of the wrapped phase. Conversely, high-order Gray code algorithms [26], using a sufficient number of Gray code binary fringe patterns, assign a unique identifier to each pixel. For example, in an image with a size of 512 × 512, using sufficient Gray code binary fringe patterns with 9 horizontal and 9 vertical directions ensures that each pixel has a unique code and identifier. This pixel-wise identifier can be used directly to establish a triangulation relationship between the projector and the camera,similar to binocular stereo vision, facilitating the acquisition of 3D point cloud information about the object.

低阶格雷码算法[27]通常用于相位展开，为与包裹相位的不同条纹阶对应的区域提供唯一标识符。相反，高阶格雷码算法[26]使用足够数量的格雷码二进制条纹图案，为每个像素分配唯一的标识符。例如，在尺寸为512×512的图像中，使用足够的9个水平和9个垂直方向的格雷码二值条纹图案可以确保每个像素具有唯一的编码和标识符。这种逐像素识别器可以直接用于在投影仪和相机之间建立类似于双目立体视觉的三角测量关系，有助于获取物体的3D点云信息。

2.2. Highly reflective fringe distortion

To analyze the damage caused by high reflection to fringe coding information, a pixel column is randomly selected on the distorted and the ideal fringes to pass through the highly reflective regions (green dashed line), and its gray value is drawn as a curve, as shown in Fig. 3. For Gray code binary fringe patterns, encoding is achieved by setting a brightness threshold, where pixels with intensities below this threshold are encoded as 0, and thoseabove it are encoded as 1. However, regions with severe specular reflection can cause a jump from the dark pixel coded 0 to the bright pixel coded 1, as indicated by the green points in the curve graph in Fig. 3 (a).

为了分析高反射对条纹编码信息造成的损害，在畸变条纹和理想条纹上随机选择一个像素列穿过高反射区域（绿色虚线），将其灰度值绘制为曲线，为如图3所示。对于格雷码二值条纹图案，通过设置亮度阈值来实现编码，强度低于该阈值的像素编码为0，高于该阈值的像素编码为1。但是，镜面反射严重的区域可以导致从编码0的暗像素跳到编码1的亮像素，如图3(a)中曲线图中的绿点所示。

For the sinusoidal fringe pattern, it is observed that two cases would lead to distortion. (1) Fringe Saturation. As shown in Fig. 3 (b), when specularly reflected pixels are in the bright region of the sinusoidal fringe pattern, the light intensity values at these positions exceed 255 and are truncated by the 8-bit camera. This leads to a saturation of the fringes, manifesting as the “peak platform effect” in the sinusoidal curve [28]. (2) Fringe Blurring. As shown in Fig. 3 ( c), when specularly reflected pixels are in the dark region of the sinusoidal fringe pattern, the brightness values of the dark region increase, resulting in a decrease in the contrast between bright and dark areas. In a word, both the Gray code binary fringe pattern and the sinusoidal fringe pattern will be distorted due to the high reflection, resulting in 3D reconstruction errors. Therefore, it is necessary to perform image restoration to obtain ideal fringe patterns with the correct encoded information.

对于正弦条纹图案，观察到两种情况会导致失真。 (1) 条纹饱和度。如图3（b）所示，当镜面反射像素位于正弦条纹图案的明亮区域时，这些位置的光强度值超过255并被8位相机截断。这导致条纹饱和，表现为正弦曲线中的“峰值平台效应”[28]。 (2) 边缘模糊。如图3（c）所示，当镜面反射像素位于正弦条纹图案的暗区时，暗区的亮度值增加，导致亮区和暗区之间的对比度降低。总之，格雷码二值条纹图和正弦条纹图都会因高反射而失真，导致3D重建误差。因此，需要进行图像恢复以获得具有正确编码信息的理想条纹图案。

3.Network building

3.1. Network structure

The architecture of the DcMcNet is shown in Fig. 4. The fringe pattern with highly reflective distorted regions is taken as input, and the ideal fringe pattern is taken as output. The role of DcMcNet is to repair the distorted and blurry areas in the fringe pattern and replenish the missing coding information. DcMcNet is an encoder-decoder structure [21], and the highly reflective fringe pattern is first passed through a 3 × 3 deformable convolution (DC) [22] for feature extraction. DC is shown in Fig. 5, where red points represent points in the convolution kernel. The use of DC allowsthe convolutional kernel’s fixed shape to be flexibly adjusted based on the shape of the highly reflective distorted regions in the image, demonstrating better adaptability to non-uniform highly reflective conditions than ordinary convolution. The implementation process is to adaptively change the shape of the convolution kernel in DC by addinga 2D offsetto the samplingposition of the feature map. The offsets are learned from the feature maps through additional convolutional layers. The offset map has the same spatial resolution as the input feature but has twice the number of channels as the input because each channel includes offset maps along the height and width directions. The 2D offset maps are added to the input features, and then ordinary convolution is used for feature extraction, which implements the deformation of the size, shape, and direction of the convolution kernel.

DcMcNet的架构如图4所示。将具有高反射失真区域的条纹图案作为输入，将理想的条纹图案作为输出。 DcMcNet的作用是修复条纹图案中畸变和模糊的区域，补充缺失的编码信息。 DcMcNet是一种编码器-解码器结构[21]，高反射条纹图案首先通过3×3可变形卷积（DC）[22]进行特征提取。 DC如图5所示，其中红点代表卷积核中的点。 DC的使用使得卷积核的固定形状可以根据图像中高反射畸变区域的形状进行灵活调整，表现出比普通卷积更好的对非均匀高反射条件的适应性。实现过程是通过在特征图的采样位置添加2D偏移来自适应改变DC中卷积核的形状。偏移量是通过附加的卷积层从特征图中学习的。偏移图具有与输入要素相同的空间分辨率，但通道数是输入的两倍，因为每个通道都包括沿高度和宽度方向的偏移图。将2D偏移图添加到输入特征中，然后使用普通卷积进行特征提取，实现了卷积核的大小、形状和方向的变形。

The feature maps are then passed to the five encoder layers of the encoder, each of which consists of a convolution block (3 × 3 convolution + Batch Normalization + Rectified Linear Unit) and a 2 × 2 maximum pooling layer. For each encoder layer passed, the resolution of the feature map is halved and the number of channels is doubled. The encoder is employed to extract scene context information with global structure and high-level semantic features. The output feature map of the encoder passes through a multi-scale convolution block (MC) [23], which uses convolutional kernels of different scales (utilizing 1 × 1, 3 × 3, and 7 × 7 kernels in this case) to gather information from different receptive fields. The introduction of the MC has two purposes. Firstly, when repairing highly reflective images, it is necessary to pay attention to both local texture details and the overall context of the image. Multiscale convolution kernel helps balance the attention to local details and the overall structure. Secondly, in the highly reflective fringe pattern, the fringe distortion and blurry regions may have different scales, as shown in Fig. 6. A Multi-scaleconvolution kernel can make the network more adaptable and able to process regions of different sizes, thus improving the robustness of image restoration.

然后将特征图传递到编码器的五个编码器层，每个编码器层由一个卷积块（3×3卷积+批量归一化+整流线性单元）和一个2×2最大池化层组成。对于通过的每个编码器层，特征图的分辨率减半，通道数加倍。编码器用于提取具有全局结构和高级语义特征的场景上下文信息。编码器的输出特征图通过多尺度卷积块（MC）[23]，该块使用不同尺度的卷积核（在本例中使用 1 × 1、3 × 3 和 7 × 7 核）来收集来自不同感受野的信息。 MC的引入有两个目的。首先，修复高反射图像时，既要关注局部纹理细节，又要关注图像的整体脉络。多尺度卷积核有助于平衡对局部细节和整体结构的关注。其次，在高反射条纹图案中，条纹畸变和模糊区域可能具有不同的尺度，如图6所示。多尺度卷积核可以使网络更具适应性并能够处理不同尺寸的区域，从而提高图像恢复的鲁棒性。

The output of the multi-scale convolution block is used as the input of the decoder, which uses the multi-scale context information to estimate the ideal fringe pattern without high reflection. The decoder comprises five layers, with each layer adopting a multi-branch structure composed of three 4 × 4 transposed convolutions, which are used to restore the scale of the feature map to its original size layer by layer. A multi-scale convolution is also added after the third decoder layer to achieve multi-scale receptive fields. To fuse the features of the encoder into the decoder, skip connections are introduced to compensate for the information loss caused by resampling.

多尺度卷积块的输出用作解码器的输入，解码器使用多尺度上下文信息来估计没有高反射的理想条纹图案。解码器由五层组成，每层采用由三个4×4转置卷积组成的多分支结构，用于逐层将特征图的尺度恢复到原始大小。在第三个解码器层之后还添加了多尺度卷积以实现多尺度感受野。为了将编码器的特征融合到解码器中，引入了跳跃连接来补偿重采样造成的信息损失。

3.2. Loss function building

Previous methods primarily utilized per-pixel loss functions, which are computationally straightforward but often lead to blurry predictions due to their inability to capture the comprehensive characteristics of the image distribution. To suppress the blurring artifacts and provide more visually appealing results, a loss function is designed that takes into account statistical similarity and human perception, including Structural Similarity (SSIM) [29], Mean Absolute Error (MAE), and Maximum Mean Discrepancy (MMD) [30], as shown in Eq. (3).

以前的方法主要利用每像素损失函数，这些函数计算简单，但由于无法捕获图像分布的综合特征，常常导致预测模糊。为了抑制模糊伪影并提供更具视觉吸引力的结果，设计了一种损失函数，考虑了统计相似性和人类感知，包括结构相似性（SSIM）[29]、平均绝对误差（MAE）和最大平均差异（MMD） ）[30]，如式（1）所示。 （3）。

In this loss function, component weights are selected based on validation results. Since the MAE and MMD weights have little effect on the experimental results, they are directly set to 1. Whereas a weight of 0.8 for SSIM reduces the MSE on the test dataset by approximately 30 % compared to other weights, so a weight of 0.8 is assigned to SSIM.

在此损失函数中，根据验证结果选择分量权重。由于MAE和MMD权重对实验结果影响不大，所以直接设置为1。而SSIM的权重为0.8，与其他权重相比，测试数据集上的MSE降低了约30%，因此权重为0.8分配给 SSIM。

3.2.1. Structural similarity (SSIM) loss

Structural Similarity (SSIM) is a method for measuring the similarity between two images. SSIM takes into account the structural information of the imageand can betterreflect the perception of image quality by the human eye. To calculate SSIM, the following function is established, which considers three components: brightness similarity, contrast similarity, and structure similarity.

结构相似度（SSIM）是一种测量两幅图像之间相似度的方法。 SSIM考虑了图像的结构信息，能够更好地反映人眼对图像质量的感知。为了计算SSIM，建立了以下函数，该函数考虑三个组成部分：亮度相似度、对比度相似度和结构相似度。

where $x$ and $y$ represent the the ground truths of the fringe patterns and the fringe patterns restored by DcMcNet respectively, ${\mu }_x$ and ${\mu }_y$ are their pixel means, ${\sigma }_x$ and ${\sigma }_y$ are their pixel standard deviations, and ${\sigma }_{xy}$ is their pixel covariance. $c_1$ and $c_2$ are smoothing parameters used to stabilize the division in the denominator.
其中$x$和$y$分别表示条纹图案和DcMcNet恢复的条纹图案的gt，${\mu }_x$和${\mu }_y$是它们的像素均值，${\sigma }_x$和${\sigma }_y$是它们的像素标准差，${\sigma }_{xy}$ 是它们的像素协方差。 $c_1$ 和 $c_2$ 是用于稳定分母除法的平滑参数。

3.2.2. Maximum Mean Discrepancy (MMD) loss

Maximum Mean Discrepancy (MMD) measures the difference between two probability distributions by comparing the representations of the samples from these two distributions in the feature space, which is represented as follows.

最大平均差异（MMD）通过比较这两个分布的样本在特征空间中的表示来衡量两个概率分布之间的差异，其表示如下。

where $x$ and $y$ are two probability distributions, which represent the ground truths and the restored fringe patterns respectively. $x_i$ and $y_i$ are samples taken from these two probability distributions, which represent the pixels in the fringe patterns. $k(g)$ is the kernel function, and the Gaussian kernel (RBF kernel) is chosen in this paper. There are three terms in Eq. (5), the first termrepresents the similarity between samples from distribution $x$, the second term captures the difference between samples from distribution $x$ and $y$, and the third term represents the similarity between samples from distribution $y$. The MMD loss function is used to measure the difference between two distributions. By adjusting the parameters of the kernel function and optimizing this loss function, the maximum mean discrepancy between the two distributions can be minimized, aiming to make them more similar.

其中$x$和$y$是两个概率分布，分别代表真实情况和恢复的条纹图案。 $x_i$ 和 $y_i$ 是从这两个概率分布中获取的样本，代表条纹图案中的像素。 $k(g)$为核函数，本文选择高斯核（RBF核）。等式中有三项。 (5)，第一项表示分布 $x$ 中的样本之间的相似度，第二项表示分布 $x$ 和 $y$ 中的样本之间的差异，第三项表示分布 $y$ 中的样本之间的相似度。 MMD损失函数用于衡量两个分布之间的差异。通过调整核函数​​的参数并优化该损失函数，可以最小化两个分布之间的最大均值差异，旨在使它们更加相似。

4.Dataset preparation

4.1. Virtual dataset

To train DcMcNet, large-scale and high-quality datasets need to be built. To make our network suitable for different SL reconstruction algorithms, the dataset constructed in this paperincludes sinusoidal fringe patterns and Gray code binary fringe patterns. The virtual software 3ds Max is used to build a digital twin of the real SL 3D reconstruction system, simulating the placement of projectors, cameras, and objects in the real world, as shown in Fig. 7 (a) (b). 300 models are selected from the model library available online as the measured objects, and the reflection characteristics are adjusted by changing the surface material. As shown in Fig. 7 ( c), two sets of imagesare captured for each measured object. The surface material of the first set of images is metallic silver paint, and the collected images are highly reflective fringe patterns. The other set of images is made of a standard material, capturing the ideal fringe patterns without high reflection. A virtual dataset has since been constructed with the highly reflective fringe pattern as the input to the DcMcNet and the ideal fringe pattern as the ground truth for optimizing the output of the DcMcNet.

为了训练 DcMcNet，需要构建大规模、高质量的数据集。为了使我们的网络适用于不同的 SL 重建算法，本文构建的数据集包括正弦条纹图案和格雷码二进制条纹图案。虚拟软件3ds Max用于构建真实SL 3D重建系统的数字孪生，模拟现实世界中投影仪、相机和物体的放置，如图7(a)(b)所示。从在线模型库中选择300个模型作为测量对象，通过改变表面材质来调整反射特性。如图7（c）所示，为每个被测物体捕获两组图像。第一组图像的表面材质为金属银漆，采集到的图像为高反光条纹图案。另一组图像由标准材料制成，捕捉到理想的条纹图案，没有高反射。此后，我们构建了一个虚拟数据集，其中高反射条纹图案作为 DcMcNet 的输入，理想条纹图案作为优化 DcMcNet 输出的GT。

For sinusoidal fringe patterns, due to the complexity of surface coding, the parameters of fringe patterns in each model are adjusted to make the constructed dataset have diversity. As shown in Fig. 7 (d), the projected fringe patterns are encoded using sinusoidal functions with periods of 28, 32, and 36 pixels. Due to variations in hardware focal length and distance during data acquisition, the periods of cameracaptured fringe patterns are diverse. This ensures that the trained network can generalize to different fringe periods. It is worth mentioning that due to limited data volume and computational complexity, fringe patterns with extremely large or small periods are not included in our dataset,as they are of poorvisual quality and rarely used in SL reconstruction algorithms. The specific range of fringe periods suitable for DcMcNet is indicated in Section 5. As shown in Fig. 7 (e), changing the brightness of the fringe pattern by adjusting the exposure time of the camera makes the prediction accuracy of our network independent of the ambient light intensity or other factors that affect the brightness of the fringe. For Gray code binary fringe patterns, because they consist of only black and white, there is no need to adjust or enhance the parameters.

对于正弦条纹图案，由于表面编码的复杂性，需要调整每个模型中条纹图案的参数，以使构建的数据集具有多样性。如图 7 (d) 所示，投影条纹图案使用周期为 28、32 和 36 个像素的正弦函数进行编码。由于数据采集过程中硬件焦距和距离的变化，相机捕获的条纹图案的周期是不同的。这确保了训练后的网络可以推广到不同的边缘周期。值得一提的是，由于有限的数据量和计算复杂性，具有极大或极小周期的条纹图案不包含在我们的数据集中，因为它们的视觉质量较差并且很少在SL重建算法中使用。适合DcMcNet的条纹周期的具体范围在第5节中指出。如图7（e）所示，通过调整相机的曝光时间来改变条纹图案的亮度，使得我们的网络的预测精度独立于环境光强度或其他影响条纹亮度的因素。对于格雷码二值条纹图案，由于它们仅由黑色和白色组成，因此不需要调整或增强参数。

4.2. Real dataset

To enable subsequent evaluation of the DcMcNet ’s performance in real-world scenarios, an SL system,as shown in Fig. 8 (a), is constructed, consisting of a projector (DLP LightCrafter 4500, resolution 912 x 1140) and a camera (resolution 2048 x 2448). This system is used to capture fringe patterns from various objects, such as aeronautical blades, automobile parts, and metal standard blocks, forming a real dataset. This dataset includes two types of fringe patterns: sinusoidal fringe patterns and Gray codes binary fringe patterns. Each object has two sets of images, one set contains the highly reflective fringe patterns captured directly on the highly reflective surface, which serves as the input to the network (as shown in Fig. 8 (b)). The other set of images contains the non-highly reflective fringe patterns captured on the diffusely reflective surface after spraying white powder, which serves as the ideal truthvalued fringe patterns (as shown in Fig. 8( c)).

为了后续评估 DcMcNet 在实际场景中的性能，构建了如图 8（a）所示的 SL 系统，由投影仪（DLP LightCrafter 4500，分辨率 912 x 1140）和相机组成（分辨率 2048 x 2448）。该系统用于捕获航空叶片、汽车零部件、金属标准块等各种物体的条纹图案，形成真实的数据集。该数据集包括两种类型的条纹图案：正弦条纹图案和格雷码二进制条纹图案。每个物体都有两组图像，一组包含直接在高反射表面上捕获的高反射条纹图案，作为网络的输入（如图8（b）所示）。另一组图像包含喷涂白色粉末后在漫反射表面上捕获的非高反射条纹图案，这是理想的真值条纹图案（如图8（c）所示）。

5.Experimentation and analysis

Our network was developed based on Pytorch and trained on a server equipped with an NVIDIA GeForce 3090 GPU. The training process involved 300 epochs and took approximately 4.5 h, with a batch size of 64 and a learning rate set to 0.0001. The virtual dataset discussed in Section 4 was used for network learning and contained a total of 3000 data scenarios, with 70 % allocatedfor training, 20 % for validation,and 10 % for testing. In this section, the performance of DcMcNet was evaluated in the following aspects. Firstly, ablation experiments were performed to demonstrate the effectiveness of multi-scale convolution and deformable convolution in our network. Secondly, the restoration effectiveness of fringe patterns with different periods was verified, determining the suitable range of fringe periods for DcMcNet. Then, depth reconstruction experiments were performed on sinusoidal fringe patterns, because the high reflection distortion of fringe patterns can lead to defects and fluctuations in the reconstructed depth maps, and thus the quality of the depth map was used to evaluate the restoration effectiveness of our network on sinusoidal fringe patterns. The depth reconstruction experiment further verified the performance of our method by comparing DcMcNet with other networks [19,20]. Finally, point cloud reconstruction experiments were performed on Gray code binary fringe patterns, and the presence of outliers in the reconstructed point clouds was used to assess the quality of the Graycode binary fringe patterns repaired by our network.

我们的网络是基于 Pytorch 开发的，并在配备 NVIDIA GeForce 3090 GPU 的服务器上进行训练。训练过程涉及 300 个 epoch，大约需要 4.5 小时，批量大小为 64，学习率设置为 0.0001。第 4 节中讨论的虚拟数据集用于网络学习，总共包含 3000 个数据场景，其中 70% 分配用于训练，20% 用于验证，10% 用于测试。本节从以下几个方面评估了DcMcNet的性能。首先，进行消融实验来证明我们网络中多尺度卷积和可变形卷积的有效性。其次，验证了不同周期条纹图案的恢复效果，确定了DcMcNet合适的条纹周期范围。然后，对正弦条纹图案进行深度重建实验，因为条纹图案的高反射失真会导致重建深度图的缺陷和波动，因此深度图的质量被用来评估我们网络的恢复效果正弦条纹图案。深度重建实验通过将 DcMcNet 与其他网络进行比较，进一步验证了我们方法的性能[19,20]。最后，对格雷码二进制条纹图案进行点云重建实验，并使用重建点云中异常值来评估我们的网络修复的格雷码二进制条纹图案的质量。

5.1. Ablation experiment

To verify the effectiveness of multi-scale convolution (MC) and deformable convolution (DC) in DcMcNet, subnets 1 to 3 were designed to represent the removal of both MC and DC, onlyMC, and only DC from DcMcNet. The MAE and SSIM of the training, validation, testing, and real datasets for DcMcNet and subnets 1 to 3 are shown in Table 1. It can be seen that the MAEsof DcMcNet reach the $10^{-4}$ level (0.00072), which is one order of magnitude smaller than the MAEs of subnets 1 and 2, and slightly lower than those of subnet 3. Moreover, the SSIMs of DcMcNet are much closer to 1 thanthose of subnets 1 to 3. The quantitativeresults illustrate that MC greatly improves the quality of the restored fringe patterns, and DC has the same enhancement effect. For visual comparison, fringe patterns before and after restoration using DcMcNet and subnets 1 to 3 are shown in Fig. 9. It can be seen that the restoration of DcMcNet results in real fringe patterns without high reflection traces and virtual fringe patterns with complete shape. However, for subnets 1 to 3, there are still obvious traces of highly reflective anomalies in the restored real fringe patterns, and the restoredvirtual fringe patterns lose many details. In summary, MC and DC contribute to improving restoration accuracy in highly reflective regions and preserving more details.

为了验证DcMcNet中多尺度卷积（MC）和可变形卷积（DC）的有效性，子网1到3被设计为表示从DcMcNet中同时去除MC和DC、仅去除MC、仅去除DC。 DcMcNet 和子网 1 至 3 的训练、验证、测试和真实数据集的 MAE 和 SSIM 如表 1 所示。可以看出，DcMcNet 的 MAE 达到了 $10^{-4}$ 级别（0.00072），这是一个数量级小于子网 1 和 2 的 MAE，略低于子网 3 的 MAE。此外，DcMcNet 的 SSIM 也小得多比子网 1 到 3 更接近 1。定量结果表明 MC 极大地提高了恢复的条纹图案的质量，并且 DC 具有相同的增强效果。为了进行视觉比较，使用 DcMcNet 和子网 1 至 3 恢复前后的条纹图案如图 9 所示。可以看出，DcMcNet 的恢复结果得到了没有高反射痕迹的真实条纹图案和具有完整形状的虚拟条纹图案。然而，对于子网1到3，恢复的真实条纹图案中仍然存在明显的高反射异常痕迹，并且恢复的虚拟条纹图案丢失了许多细节。综上所述，MC和DC有助于提高高反射区域的恢复精度并保留更多细节。

5.2. Effects analysis of fringe period

To evaluate our network’s ability to restore highly reflective sinusoidal fringe patterns with varying periods, and to further define the range of fringe periods suitable for DcMcNet, an experiment was conducted on nine highly reflective fringe patterns with periods ranging from 8 to 104 pixels. The results are shown in Fig. 10, with their corresponding MSE labeled. From Fig. 10, it is evident that fringe patterns with periods between 30 and 80 pixels exhibit significantly superior results, with MSE values that are one to two orders of magnitude lower than patterns with other periods. This disparity is caused by fringe patterns with extremely small or large periods differing significantly from the periods in our training dataset, resulting in decreased performance of the trained network on these data.

为了评估我们的网络恢复不同周期的高反射正弦条纹图案的能力，并进一步定义适合 DcMcNet 的条纹周期范围，对周期范围从 8 到 104 像素的 9 个高反射条纹图案进行了实验。结果如图 10 所示，并标记了相应的 MSE。从图 10 可以明显看出，周期在 30 到 80 像素之间的条纹图案表现出明显优异的结果，其 MSE 值比其他周期的图案低一到两个数量级。这种差异是由周期极小或极大的边缘模式与我们的训练数据集中的周期显着不同引起的，导致训练网络在这些数据上的性能下降。

5.3. Verification of depth reconstruction effect for sinusoidal fringe patterns

In evaluating the effectiveness of our network for restoring highly reflective regions in the sinusoidal fringe patterns, the smoothness and accuracy of the reconstructed depth maps are crucial metrics. In this section, four-step phase-shifting, three-frequency heterodyne, and phase-height methods were used for depth reconstruction of the sinu soidal fringe patterns before and after restoration. To highlight the advantages of our network, comparative experiments were also conducted with contrast_net1 [19], and contrast_net2 [20].

在评估我们的网络恢复正弦条纹图案中高反射区域的有效性时，重建深度图的平滑度和准确性是至关重要的指标。本节采用四步相移、三频外差和相位高度方法对恢复前后的正弦曲线条纹图案进行深度重建。为了突出我们网络的优势，还与contrast_net1 [19]和contrast_net2 [20]进行了比较实验。
Aeronautical blades are complex-shaped aircraft engine parts, and their 3D shapes have an important impact on aircraft safety, therefore, in this section, the depth reconstruction results of the aeronautical blades in the real dataset were shown. The comparison of the fringe patterns, depth maps, and depth error maps before and after restoration using DcMcNet and the above comparison network are shown in Fig. 11. In addition, to illustrate the effectiveness of our method on the virtual test dataset, the fringe patterns, depth maps, and depth error maps of two randomly selected virtual models are shown in Fig. 12. It can be observed from Fig. 11 and Fig. 12 that DcMcNet successfully eliminates depth fluctuations and shape discontinuities caused by high reflection distortions, and generates smoother and more realistic depth maps compared to other comparison networks. In contrast, the depth maps reconstructed by the comparison networks still have distorted areas (marked by red circles). The depth map comparison results show that DcMcNet outperforms the other networks in repairing the distorted regions of the highly reflective sinusoidal fringe patterns.

航空叶片是形状复杂的飞机发动机零件，其3D形状对飞机安全具有重要影响，因此，本节展示了航空叶片在真实数据集中的深度重建结果。使用 DcMcNet 和上述比较网络恢复前后的条纹图案、深度图和深度误差图的比较如图 11 所示。此外，为了说明我们的方法在虚拟测试数据集上的有效性，图 12 显示了随机选取的两个虚拟模型的条纹图案、深度图和深度误差图。从图 11 和图 12 可以看出，DcMcNet 成功消除了由高反射畸变引起的深度波动和形状不连续性，并且与其他对比网络相比，生成了更平滑、更真实的深度图。相反，对比网络重建的深度图仍然存在畸变区域（用红色圆圈标记）。深度图比较结果表明，DcMcNet 在修复高反射正弦条纹图案的扭曲区域方面优于其他网络。

For quantitative evaluation, the MAE and SSIM of the depth maps were used as evaluation metrics, and the error values of DcMcNet, contrast_net1, and contrast_net2 are shown in Table 2. It can be seenthat our method has the smallest MAEs, followed by the two comparison networks whose MAEs are 2–3 times higher than those of our method, and the worst is the direct use of distorted fringe patterns whose MAEs are an order of magnitude higher than those of our method. Similarly, the SSIMs of our method are closer to 1 than those of the comparison networks. In conclusion, although the comparison networks can also improve the depth map accuracy, they are still inferior to our method, indicating that our method is more effective in repairing highly reflective distorted regions of the sinusoidalfringe patterns and provides more accurate input for SL depth reconstruction.

对于定量评估，使用深度图的MAE和SSIM作为评估指标，DcMcNet、contrast_net1和contrast_net2的误差值如表2所示。可以看出，我们的方法具有最小的MAE，其次是两个比较网络的 MAE 比我们的方法高 2-3 倍，最糟糕的是直接使用畸变的条纹图案，其 MAE 比我们的方法高一个数量级。同样，我们方法的 SSIM 比比较网络的 SSIM 更接近 1。总之，虽然比较网络也可以提高深度图精度，但它们仍然不如我们的方法，这表明我们的方法在修复正弦条纹图案的高反射畸变区域方面更有效，并为 SL 深度重建提供更准确的输入。

5.4. Verification of point cloud reconstruction effect for Gray code fringe patterns

The distorted regions of highly reflective Gray code fringe patterns can lead to outliers and irregular shapes in the reconstructed point clouds, so the quality of the point clouds was used to reflect the effectiveness of our network in restoring Gray code binary fringe patterns. In this section, 42-order Gray code (21 orders vertically and 21 orders horizontally) was used for point cloud reconstruction. Since there are currently no precedents for restoring Gray code binary fringe patterns using deep learning methods, DcMcNet was compared with traditional multi-exposure methods [11].

高反射格雷码条纹图案的畸变区域会导致重建点云中出现异常值和不规则形状，因此点云的质量被用来反映我们的网络在恢复格雷码二进制条纹图案方面的有效性。本节中，使用42阶格雷码（垂直21阶，水平21阶）进行点云重建。由于目前还没有使用深度学习方法恢复格雷码二进制条纹图案的先例，因此将DcMcNet与传统的多重曝光方法进行了比较[11]。

The results of point cloud reconstruction for three industrial parts with different reflectivity are shown in Fig. 13. Objects 1 and 2 are the front and back of an aeronautical blade, object 3 is a metal standard block, and object 4 is an automobile part. It can be observed that our method produces more complete point clouds with fewer missing regions than the four-exposure method. Moreover, the geometric shapes reconstructed by our method more closely resemble the ground truth, indicating that our method achieves better accuracy in point cloud reconstruction. The coverage rate $C$ [15], used for quantitative evaluation, was defined as:

三个不同反射率的工业零件的点云重建结果如图13所示。对象1和2是航空叶片的正面和背面，对象3是金属标准块，对象4是汽车零件。可以看出，与四次曝光方法相比，我们的方法产生了更完整的点云，丢失区域更少。此外，我们的方法重建的几何形状更接近真实情况，表明我们的方法在点云重建方面取得了更好的精度。用于定量评估的覆盖率$C$[15]定义为：

where $P_m$ represents the number of points in the measured point cloud, and $P_b$ represents the number of points in the benchmark point cloud, which was reconstructed from the fringe patterns captured after spraying white powder on the object.The point clouds reconstructed from the Gray code binary fringe patterns before and after enhancement using DcMcNet and the four-exposure method are shown in Table 3. It can be seen that compared to the point clouds reconstructed directly using the highly reflective fringe patterns, our method exhibits superior performance in repairing distorted areas, resulting in more complete point clouds (average coverage rate $C$: 98.72 % vs. 78.38 %). Although the four-exposure method can partially restore the lost featurescompared to the direct use of highly reflective fringe patterns, its average coverage rate $C$ is still lowerthan that of our proposedmethod (98.72 % vs. 88.08 %). In conclusion, our method can effectively restore the highly reflective distorted regions, thus generating more complete and high-quality point clouds.

其中$P_m$表示测量点云中的点数，$P_b$表示基准点云中的点数，基准点云是根据在物体上喷涂白色粉末后捕获的条纹图案重建的。重建的点云从使用DcMcNet和四次曝光方法增强前后的格雷码二进制条纹图案如表3所示。可以看出，与直接使用高反射条纹图案重建的点云相比，我们的方法在以下方面表现出了优越的性能：修复畸变区域，产生更完整的点云（平均覆盖率 $C$：98.72 % vs. 78.38 %）。虽然与直接使用高反射条纹图案相比，四次曝光方法可以部分恢复丢失的特征，但其平均覆盖率 $C$ 仍然低于我们提出的方法（98.72 % vs. 88.08 %）。总之，我们的方法可以有效地恢复高反射畸变区域，从而生成更完整和高质量的点云。

6.Conclusion

This paper proposed a lightweight novel network, DcMcNet, for restoring SL fringe patterns destroyed by high reflection to provide highquality inputs for subsequent 3D reconstruction. DcMcNet is the first network capable of restoring both highly reflective sinusoidal fringe patterns and the Gray code binary fringe patterns, which have completely different encoding modes. Moreover, in DcMcNet, deformable convolution can better handle the variously shaped distorted regions caused by inhomogeneous high reflectance, and multi-scale convolution can help to balance the attention to local details and the overall structure in the restoration process. To improve the network performance, a large-scale and high-fidelity dataset was also constructed using virtual software, and the diversity loss function was proposed. Through depth and point cloud reconstruction experiments, the effectiveness of DcMcNet in restoring highly reflective regions in sinusoidal and Gray code binary fringe patterns was verified, with depth MAE able to reach 0.0675 mm and point cloud coverage able to reach 98.95 %. In summary, our approach provides an effective solution to the challenge of accurately measuring high-reflective objects using 3D structured light. However, the current work is only the restoration of the highly reflective fringe pattern, and in the future, we will try the study obtaining 3D shapes directly from the highly reflective fringe pattern.

本文提出了一种轻量级新型网络 DcMcNet，用于恢复被高反射破坏的 SL 条纹图案，为后续 3D 重建提供高质量输入。 DcMcNet 是第一个能够同时恢复高反射正弦条纹图案和格雷码二进制条纹图案的网络，它们具有完全不同的编码模式。此外，在DcMcNet中，可变形卷积可以更好地处理由不均匀的高反射率引起的各种形状的畸变区域，而多尺度卷积可以帮助平衡恢复过程中对局部细节和整体结构的关注。为了提高网络性能，还利用虚拟软件构建了大规模、高保真数据集，并提出了多样性损失函数。通过深度和点云重建实验，验证了DcMcNet在恢复正弦和格雷码二值条纹图案中高反射区域的有效性，深度MAE能够达到0.0675 mm，点云覆盖率能够达到98.95%。总之，我们的方法为使用 3D 结构光精确测量高反射物体的挑战提供了有效的解决方案。然而，目前的工作只是高反射条纹图案的恢复，未来我们将尝试直接从高反射条纹图案获取3D形状的研究。

---