IDS peak ICV v1.6.0
Loading...
Searching...
No Matches
Textured Point Cloud

Table of Contents

A textured point cloud combines 3D geometry with image information acquired by a camera.

This is useful for applications such as:

  • filtering 3D points by intensity values, color values, segmentation labels, classification results,
  • analyzing 3D geometry together with image data,
  • visualizing a point cloud with texture information.

The 3D geometry is usually provided by a 3D camera as an XYZ image. Each valid pixel of an XYZ image represents one 3D point. The texture information may be acquired by another camera. Depending on the camera and the pixel format, the texture image may contain intensity values or color values.

To assign texture values to 3D points, each 3D point must be associated with the image value observed by the texture camera at the same physical position. This requires a geometric relationship between the 3D camera and the texture camera. If both cameras are calibrated with respect to the same workspace coordinate system, the 3D points measured by the 3D camera can be projected into the image space of the texture camera.

The projected XYZ image and the texture image then have matching image geometry and can be combined pixel by pixel.

Background

A 3D camera typically provides an XYZ image. An XYZ image is an image-like representation of a point cloud. Instead of storing intensity or color values, each pixel stores the 3D coordinates of the point measured at that pixel. The pixel position in the XYZ image belongs to the image space of the 3D camera. The stored XYZ value represents the measured 3D point.

A texture camera may acquire an image of the same scene. The image data can then be used to provide additional information for the 3D points. This image data is referred to as the texture.

However, the pixel coordinates of the 3D camera and the pixel coordinates of the texture camera usually do not match directly. The cameras may have different positions, orientations, lenses, image sizes, and different intrinsic camera parameters. Therefore, the XYZ image and the texture image cannot usually be combined by using identical pixel coordinates. Instead, the 3D points from the XYZ image must be projected into the view of the texture camera and thereby aligned to the grid of the texture image. The important idea is that this alignment does not alter the XYZ values; it only reassigns them to the corresponding pixel locations in the texture camera image.

For this alignment, each 3D point is transformed from the coordinate system of the source 3D camera into the coordinate system of the texture camera. This transformation is derived from the extrinsic parameters of both cameras.

After the point has been transformed into the coordinate system of the texture camera, the intrinsic parameters of the texture camera are used to project the point onto the image plane of the texture camera. The resulting pixel position is the position at which the texture camera would observe the 3D point.

Required Calibration Data

The following calibration data are required:

  • the extrinsic parameters of the source 3D camera,
  • the extrinsic parameters of the texture camera,
  • the intrinsic parameters of the texture camera.

In the API, the extrinsic parameters of the source 3D camera are provided as ExtrinsicParameters.

The intrinsic and extrinsic parameters of the texture camera are provided together as CalibrationParameters.

Both cameras must be calibrated with respect to the same workspace coordinate system. The extrinsic parameters of the source 3D camera and the extrinsic parameters contained in the calibration parameters of the texture camera must therefore refer to the same workspace coordinate system. If this requirement is not fulfilled, the transformation between the camera coordinate systems cannot be determined correctly. As a result, the projected XYZ image will not match the texture image.

Usage Instructions

Creating a textured point cloud consists of the following steps:

  • Acquire an XYZ image with the 3D camera.
  • Acquire a texture image with the texture camera.
  • Align the XYZ image to the pixel grid of the texture camera.
  • Create a textured point cloud from the aligned XYZ image and the texture image.

The important idea is that the texture image remains unchanged. Instead, the XYZ image is aligned with the pixel grid of the texture camera.

After this alignment, both images describe the same camera view and can be combined pixel by pixel.

  • C++
    // Create the XYZ texture alignment
    peak::icv::experimental::XYZTextureAlignment xyzTextureAlignment{ xyzExtrinsicParameters,
    rgbCalibrationParameters };
    //
    // Align the XYZ image to the pixel gird of the texture camera
    auto xyzImageAligned = xyzTextureAlignment.AlignXYZToTextureGrid(xyzImage);
    //
    // Create the textured poind cloud
    auto texturedPointCloud = peak::icv::PointCloudXYZRGB{ xyzImageAligned, rgbImage };
  • C#
    // Create the XYZ texture alignment
    var xyzTextureAlignment = new XYZTextureAlignment(xyzExtrinsicParameters, rgbCalibrationParameters);
    //
    // Align the XYZ image to the pixel gird of the texture camera
    using var xyzImageAligned = xyzProjection.AlignXYZToTextureGrid(xyzImage);
    //
    // Create the textured poind cloud
    using var texturedPointCloud = new PointCloudXYZRGB(xyzImageAligned, rgbImage);
  • Python
    # Create the XYZ texture alignment
    xyz_texture_alignment = XYZTextureAlignment.create_from_extrinsics_and_calibration_parameters(
    xyz_extrinsic_parameters, rgb_calibration_parameters)
    # Align the XYZ image to the pixel gird of the texture camera
    xyz_image_aligned = xyz_texture_alignment.align_xyz_to_texture_grid(xyz_image)
    # Create the textured poind cloud
    textured_point_cloud = PointCloud.create_from_xyz_image(xyz_image_aligned, rgb_image)

Individual steps

Create the XYZ Texture Alignment

Depending on the used interface language create an XYZTextureAlignment or 'xyz_texture_alignment' using the extrinsic parameters of the 3D camera and the calibration parameters of the texture camera.

The extrinsic parameters are used to determine the transformation from the coordinate system of the source 3D camera into the coordinate system of the texture camera.

The intrinsic parameters of the texture camera are used to project the transformed 3D points onto the image plane of the texture camera.

Align the XYZ Image with the pixel grid of the Texture Camera

Use AlignXYZToTextureGrid() or 'align_xyz_to_texture_grid()' (depending on the used interface language) to project the XYZ image into the view of the texture camera.

The returned image is again an XYZ image. It contains the original 3D point coordinates, but arranged in the image geometry of the texture camera. The XYZ coordinates themselves are not changed. Only their pixel positions are changed. A point that was originally stored at a pixel position in the 3D camera image is written to the pixel position at which the same 3D point would be observed by the texture camera.

The size of the returned XYZ image corresponds to the calibrated image size of the texture camera.

The image region of the returned XYZ image contains all pixels for which a projected 3D point exists.

Create the Textured Point Cloud

After the alignment step, the aligned XYZ image and the texture image refer to the same camera view. Each valid pixel in the aligned XYZ image therefore corresponds directly to the pixel at the same position in the texture image.

A textured point cloud can then be created by using one of the point cloud classes that combine XYZ data with additional image values:

  • PointCloudXYZRGB to associate RGB color values with each 3D point,
  • PointCloudXYZI to associate intensity values with each 3D point.

For every valid 3D point in the aligned XYZ image, the corresponding pixel value from the texture image is assigned to that 3D point.

Result

The resulting point cloud contains 3D geometry from the source 3D camera and texture information from the texture camera.

Each point in the point cloud is associated with the image value observed by the texture camera at the projected position of that 3D point.

The combined data supports two main processing use cases.

First, the texture image can be used to identify image regions that are relevant for further 3D processing. For example, regions may be selected based on intensity ranges, color classes, segmentation labels, classification results, or defect markers. The corresponding 3D points can then be extracted and used for further processing or inspection in 3D space.

Second, the texture image and the 3D points can be evaluated together to extract information that depends on both image data and 3D geometry. For example, applications can analyze image-based features at their corresponding 3D positions, combine classification results with geometric measurements, or evaluate defects with respect to their 3D shape, position, or orientation.

Because each 3D point is linked to a corresponding texture value, the result can also be used for visualization by displaying the point cloud with intensity values or color values from the texture image.