Procedure for using a depth map

The Enable checkbox in the DEPTH MAP REMAP tab is checked by default. When enabled, depth map data will be remapped to the bit depth required for steps that link to the Camera step. It is strongly recommended to always enable remapping because the unmapped data types provided by most 3D sensors are not valid for processing and analysis.

You can select a Z-range from the total depth of your image and specify how to deal with pixels outside of this range.

1. Select the Z-range. In the DEPTH MAP REMAP tab, manually adjust the sliders, or choose a Start Mode and/or End Mode from the dropdown list:

   • MinValue or MaxValue. Choose the minimum or maximum depth as an end point of the Z-range. Choose both MinValue and MaxValue to specify a Z-range that includes all depths in the depth map.

   • Custom. Adjust the sliders or enter a specific Start Value or End Value. Note that this mode is automatically selected if you use the sliders.

   • StdDev. Enter a value equivalent to the number of standard deviations from the mean depth, for which you want to start or end the Z-range. The default value is 1, which sets the start or end at 1 standard deviation below or above the mean depth value.

     The following illustration shows an example StdDev selection.

     

   Note that when the depth map is remapped, the scale along Z is adjusted so that each pixel maintains its real-world depth value in Z.

2. Enable or disable the Saturate Values checkbox. When enabled, depth values smaller than the specified Z-range minimum are set to that minimum. Similarly, depth values larger than the specified Z-range maximum are set to that maximum. Otherwise, pixels with a depth outside the selected Z-range are considered invalid.

You can see a pixel's Z-value in the display status bar, located below the depth map display. The bar shows the Z-value, as well as the X- and Y-coordinates, of the pixel under the mouse cursor, and updates as the cursor moves. For a calibrated depth map, the X- and Y-coordinates are in calibrated units; to show them in pixel units, right-click on the image to display the context menu and uncheck Calibrated Coordinates. When you do so, the Z-value switches to show intensity (gray) values.

Gaps, or invalid values, are missing data points in the depth map, where the 3D sensor could not pick up information, due to occlusion, reflections, or areas out of the camera's field of view.

Typically, 3D sensors set invalid pixels to the maximum or some other distinct value, depending on the sensor's specifications. Alternatively, some 3D sensors identify invalid pixels using a separate mask or map, such as a confidence map.

In Matrox Design Assistant, you can fill gaps. For some applications, you can reasonably approximate gap values with an interpolation of neighboring depth values. In other situations, such as when the gap spans an abrupt change in depth, or the gap is very wide, you can replace the invalid value with a constant value, such as the minimum/maximum depth value, or you can choose to propagate one of the gap's boundary values across the gap.

Gaps are filled in a 2-step process: first along X, then along Y (or vice-versa). For example, if you choose to fill first along the X-direction, all gaps along rows in the depth map are evaluated and filled before any columns are examined (see below for evaluation criteria). Therefore, gaps are examined not as an X-by-Y area to be filled, but on a per-row (or column) basis.

Note that gaps whose boundaries touch the border of the image are not filled. That is, a gap qualifies for potential filling only if it exists between 2 non-invalid pixels. You can, however, replace the pixel values of gaps that touch the border with the Replace Remaining Invalid Values setting (also discussed below).

To fill gaps, perform the following:

1. Choose the DEPTH MAP FILL GAP tab in the Camera step's Configuration pane.

2. Enable the Enable Gap Filling checkbox.

3. Set the size of the Maximum Gap To Fill. Enter a maximum gap size for the When Filling Along X and When Filling Along Y inputs, where Infinite specifies no maximum size. If your depth map is calibrated in X and Y, use calibrated units for these inputs.

4. Set the Fill Mode, which determines gap replacement pixel values. Set Interpolate to one of the following:

   • Always. Specifies to fill all gaps with a linear interpolation between the 2 depth values on either side of a gap. (This difference in depth is known as the Z-difference).

   • WhenZDifferenceLessThan. Specifies to interpolate only when a gap's Z-difference is below the specified threshold. Enter a threshold below which a gap's pixels are filled with interpolated values. If a gap's Z-difference is above this threshold, gaps are filled according to the Gap Boundary Propagation setting. Possible settings are:

     • GapMinZ. Specifies to propagate the minimum gap boundary value across the gap.

     • GapMaxZ. Specifies to propagate the maximum gap boundary value across the gap.

     • DoNotPropagate. Specifies not to fill gaps.

   • Never. Specifies not to interpolate. Gaps are filled according to the Gap Boundary Propagation setting, as described above.

   The illustration below shows how gaps are filled when Fill Mode is set to the following:

   Interpolate = WhenZDifferenceLessThan.

   Z-difference threshold value = 20.

   

   Above left, the gap's Z-difference is less than the threshold of 20; therefore, interpolation is applied.

   Above right, the gap's Z-difference is greater than 20; therefore, filling depends on the Gap Boundary Propagation setting, as illustrated in the 3 subsequent graphs.

5. Set the Fill Direction. Specify the Order in which to fill the missing data points: along rows first, then columns (XthenY), or vice versa (YthenX).

6. Optionally, enable the Replace Remaining Invalid Values checkbox to control all gaps, including gaps touching the image border and leftover gaps not filled under Enable Gap Filling. Set the Fill Value input to one of the following:

   • MinValue. Sets remaining values to the minimum depth value of the selected Z-range.

   • MaxValue. Sets remaining values to the maximum depth value of the selected Z-range.

   • Enter a specific value between the minimum and maximum depth of the depth map.

The following images show the same depth map with various gap filling settings.

Depth map	Settings	Notes
	No fill settings enabled.	Unfilled gaps are displayed in white (MILPseudo colormap).
	Enable Gap Filling = enabled Fill Mode: Interpolate = Always	All non-border gaps are filled with interpolated pixel values. In this case, and in the subsequent images, the following settings also apply: Maximum Gap To Fill: When Filling Along X / When Filling Along Y = Infinite. Fill Direction: Order = XthenY.
	Enable Gap Filling = enabled Fill Mode: Interpolate = WhenZDifferenceLessThan Z-difference threshold = 8 Gap Boundary Propagation = DoNotPropagate	Only gaps with a Z-difference less than 8 are filled with interpolated pixels. Remaining gaps are left untouched.
	Enable Gap Filling = enabled Fill Mode: Interpolate = WhenZDifferenceLessThan Z-difference threshold = 8 Gap Boundary Propagation = GapMinZ	Gaps with a Z-difference less than 8 are filled with interpolated pixels. Gaps with a Z-difference above 8 are filled with the minimum gap boundary value (GapMinZ).
	Enable Gap Filling = enabled Fill Mode: Interpolate = WhenZDifferenceLessThan Z-difference threshold = 8 Gap Boundary Propagation = DoNotPropagate Replace Remaining Invalid Values = enabled Fill Value = MinValue	When you enable Replace Remaining Invalid Values, any previously unfilled gaps are filled with the specified Fill Value. Note that this includes unfilled gaps that were specified with DoNotPropagate under Enable Gap Filling.
	Enable Gap Filling = not enabled Replace Remaining Invalid Values = enabled Fill Value = MinValue	All invalid pixels or gaps are replaced with the specified Fill Value.

You can display depth maps with one of several colormaps. By default, Matrox Design Assistant displays depth maps with the Jet colormap. To change the colormap, right-click on the display, and when the context menu appears, select Colormap and then the required colormap. You can also change the colormap from the Display element of the operator view layout. For more information on colormaps, see the Displaying images using a colormap section in the Display view reference chapter.

Note that, when you choose a depth map tab in the Camera step's Configuration pane, the display changes to depth map. When you choose the IMAGE REMAP tab, the display changes to the intensity image. Depth maps and intensity images have independent colormap settings.

You can right-click on the image to change the displayed image from the Display Image option, or you can click on the Camera step image toolbar button in the Project toolbar to choose which image to display from a dropdown list..

A depth map is often one component of several from a 3D sensor. For example, in addition to a depth map, you might have an intensity image and a confidence map. You can choose to display more than one image at a time, including the same component at different available bit depths. For information on displaying multiple images at once, see the Displaying multiple images subsection of the Quick Evaluate pane section in the Panes and editors reference chapter.

You can draw a line profile across a depth map. The resulting Line Profile graph shows depth values for the region underneath the drawn line.

To draw a line profile, click on the Trace a line profile () toolbar button in the Project toolbar, and then click on the display to draw a line. The accompanying Line Profile graph shows the depth at each pixel along the line. The X-axis represents the distance along the line in calibrated units and the Y-axis represents the depth. (For an uncalibrated depth map, the X-axis represents pixel units.) When a line profile crosses a region of invalid points, the line on the graph displays in red, indicating approximated depth values.

From a Line Profile graph, you can determine the exact width of specific gaps, or find the Z-difference of a gap; you can use these values to establish gap filling settings. Typically, values just above the exact measurements give successful results. For example, using the line profile below, the targeted gap's width is 0.85 units, so the Maximum Gap To Fill inputs are set to 0.9 units along X and 1.1 units along Y. The measured Z-difference is about 3.2 units, so the Z-difference threshold is set to 4.

Note the unfilled gap (labeled below). Given the gap filling settings used in this example, the gap remains unfilled because its width (2.8 units) is larger than the maximum gap and its Z-difference (4.8 units) is larger than the specified Z-difference threshold.

For more information on line profiles, see the Line Profile graph subsection of the Using the display graphs section in the Display view reference chapter.

A Histogram graph shows the number of pixels with a specific depth in a region of the depth map. To see the graph, click on the View histogram () toolbar button. The graph's X-axis represents depth values and the Y-axis represents the number of pixels in the image that have that depth.

You can place graph markers along the histogram to target specific depths. The accompanying table below the graph shows related pixel information, including the Count of pixels at each graph marker. Also available below the histogram is a table with statistical information about the entire region, such as the mean depth value.

For more information on histograms, including how to specify a region from which to create the Histogram graph, see the Histogram graph subsection of the Using the display graphs section in the Display view reference chapter.

Depth map remapping allows all processing and analysis steps to work with depth maps.

Note that the IntensityChecker step and the BlobAnalysis step ignore invalid values.

When binarizing a depth map using the ImageProcessing step or the BlobAnalysis step, use a world Z-value (depth value) to set the Threshold value.

When setting the Value and BackgroundValue inputs for a depth map using the BlobAnalysis step, you must use uncalibrated units. For example, for an 8-bit depth map, choose a value between 0 and 255 for these inputs. For more information, see the Creation of an output image subsection of the Procedure for using the BlobAnalysis step section in Chapter 3: BlobAnalysis step.

The ImageProcessing step's WindowLeveling operation uses world Z-values (depth values) for the Input range input, whereas the Output range input is expressed in uncalibrated units (for example, 0-255 for an 8-bit image). For more information, see the LutMap subsection of the ImageProcessing step operations section in Chapter 9: ImageProcessing step.

The ImageProcessing step's FillOrClear operation uses a world Z-value for the Fill Value input, if the input image is a depth map.