Crafting Immersive Virtual Spaces: A Guide to Interior Reconstruction with Vision API and Three.js

1 Essential Data for Application Development:
- 1.1 Camera Options:
- 1.2 Wall Model:
- 1.3 Segmentation Masks:
2 Exploring Key Components:
- 2.1 Aspect Ratio Management:
- 2.2 Setting Up the Camera:
- 2.3 Reconstructing Interior Surfaces:
3 Blending Results with Original Image:
- 3.1 Configuring EffectComposer:
- 3.2 Custom ShaderPass Configuration:
- 3.3 Updating ShaderPass Values:
- 3.4 Shader programs
  - 3.4.1 How Shaders Work:

In today's digital age, the boundaries between physical and virtual spaces are becoming increasingly blurred. From immersive virtual tours to augmented reality experiences, technology offers us new ways to interact with and explore our surroundings. One exciting application of this technology is interactive interior reconstruction, where virtual environments are created to simulate real-world spaces.

In this guide, we'll explore the fascinating world of interactive interior reconstruction and learn how to leverage the power of Three.js, a popular JavaScript library for 3D rendering, along with the Vision API to bring virtual spaces to life within a web environment. Whether you're an aspiring developer looking to dive into the world of 3D graphics or a seasoned pro seeking to expand your skill set, this guide will provide you with the tools and knowledge you need to create immersive and engaging applications.

Essential Data for Application Development:

To kickstart your development process, let's examine the key data required from the Vision API to build your application.

Camera Options:

The camera parameters provided by the Vision API are essential for achieving the correct perspective within your application. These parameters include:

Field of View (fov): The vertical field of view in degrees.

Pitch and Roll: Tilt angles of the camera in radians, crucial for orienting the view correctly.

Height: The height of the chamber above the floor level, providing additional context for camera positioning.

Wall Model:

The wall data comprises information about each wall in the room, including:

Points: Four 3D points in meters representing the corners of each wall.

Wall Normal: Vector defining the face of the wall, aiding in proper rendering and orientation.

Area and Width: Additional attributes providing insights into wall dimensions and properties.

Please note that the points are calculated using a right-handed Cartesian coordinate system. In this system, the positive x-axis points to the right, the positive y-axis points up, and the positive z-axis points towards the viewer (out of the screen).

Segmentation Masks:

Segmentation masks for walls, floors, and ceilings enable precise rendering and mixing of results in the shader program. These masks help determine what to display in the final rendering stage, enhancing realism and visual fidelity.

Exploring Key Components:

For building your web application, we recommend harnessing the power of the Three.js library. With extensive functionality tailored for web-based 3D rendering, Three.js simplifies the development process and offers a wide range of tools and features.

As you embark on developing your interactive application, it's vital to focus on key components that contribute to creating immersive experiences. Let's delve into these critical components in detail:

Aspect Ratio Management:

Maintaining the correct aspect ratio is essential for ensuring that your rendered scenes appear natural and visually accurate. This is particularly important when dealing with perspective projection and rendering to match the original photo's proportions.

WebGLRenderer: This component handles the rendering of your scenes using WebGL, providing efficient and hardware-accelerated graphics rendering capabilities.

PerspectiveCamera: Utilized for constructing perspective projection, the PerspectiveCamera simulates the way human eyes perceive the world, adding depth and realism to your scenes.

EffectComposer and Passes: EffectComposer enables the implementation of post-processing effects, enhancing visual quality and realism. Passes, combined within the EffectComposer, apply specific effects such as blurs, shaders, or color adjustments.

Upon resizing the screen or when the aspect ratio changes, it's imperative to update all relevant components to maintain consistency and preserve the intended perspective. Here's an example of how you can handle aspect ratio updates within your application:

resize(width: number, height: number) {

    // Update camera aspect ratio and projection matrix
    this.camera.aspect = width / height;
    this.camera.updateProjectionMatrix();
		
    // Resize renderer and composer to match new dimensions
    this.renderer.setSize(width, height);
    this.composer.setSize(width, height);
		
    // Update any additional passes, such as outline pass, if applicable
    this.outlinePass.setSize(width, height);
}

By synchronizing the aspect ratio across all components, you ensure that your application maintains visual consistency and accurately represents the original scene.

Setting Up the Camera:

Correctly configuring the camera is crucial for accurately rendering scenes and providing users with immersive views of the reconstructed environment. Here's how you can set up the camera using the camera parameters obtained from the Vision API response:

setupCamera(cameraParams: CameraParams, aspectRatio: number) {

  // Create a new PerspectiveCamera with the provided field of view (FoV) and aspect ratio
  this.camera = new PerspectiveCamera(cameraParams.fov, aspectRatio, 0.1, 1000);
	
  // Apply rotations to the camera using Quaternion
  const quaternionXYZ = new THREE.Quaternion();
  const initialEuler = new THREE.Euler(cameraParams.pitch || 0.0, 0.0, cameraParams.roll || 0.0, 'XYZ');
  quaternionXYZ.setFromEuler(initialEuler);
  this.camera.applyQuaternion(quaternionXYZ);
	
  // Update the camera's projection matrix after applying rotations
  this.camera.updateProjectionMatrix();
	
  // Ensure that the camera remains at the origin (0, 0, 0)
  // This position serves as the reference point for building the room model
  this.camera.position.set(0, 0, 0);
}

Explanation:

PerspectiveCamera Creation: Instantiate a new PerspectiveCamera object with the provided field of view (FoV) and aspect ratio. Setting near and far clipping planes to 0.1 and 1000, respectively, ensures that objects within this range are rendered.

Applying Rotations: Use Quaternion to apply rotations to the camera based on the received pitch and roll parameters. This ensures that the camera's orientation matches the captured environment.

Updating Projection Matrix: After applying rotations, update the camera's projection matrix to reflect the changes in orientation.

Maintaining Camera Position: Keep the camera at the origin point (0, 0, 0). This position is crucial as the room model will be constructed around the camera. By maintaining this reference point, you ensure that the bottom walls' points lie in the negative y dimension, and ceiling points lie in the positive y dimension.

Reconstructing Interior Surfaces:

To accurately place models within the room depicted in the image, it's essential to reconstruct basic interior surfaces such as walls, floor, and ceiling. The walls data obtained from the Vision API response provides valuable information for reconstructing these surfaces. Let's explore how to create wall models based on this data:

export class Wall extends Surface {
    declare public normal: Vector3;
    constructor(geometry: BufferGeometry, material: Material, attributes: WallDataAttributes) {
        super(geometry, material);
        this.normal = new Vector3(attributes.normal[0], attributes.normal[1], attributes.normal[2]);
        this.name = ProductSurface.WALL;
    }
}

// Indices for creating wall geometry
static geometryIndices = [0, 2, 1, 2, 3, 1];
static createWall(wallData: WallData): Wall {
    const points = wallData.attributes.points;
		
    // Define vertices for the wall geometry
    const vertices = new Float32Array([
        points[0][0], points[0][1], points[0][2],
        points[1][0], points[1][1], points[1][2],
        points[2][0], points[2][1], points[2][2],
        points[3][0], points[3][1], points[3][2]
    ]);
		
    // Define normals for the wall geometry
    const normal = new Float32Array([
        wallData.attributes.normal[0], wallData.attributes.normal[1], wallData.attributes.normal[2],
        wallData.attributes.normal[0], wallData.attributes.normal[1], wallData.attributes.normal[2],
        wallData.attributes.normal[0], wallData.attributes.normal[1], wallData.attributes.normal[2],
        wallData.attributes.normal[0], wallData.attributes.normal[1], wallData.attributes.normal[2]
    ]);
		
    // Create buffer geometry for the wall
    const geometry = new BufferGeometry();
    geometry.setIndex(Wall.geometryIndices);
    geometry.setAttribute('position', new BufferAttribute(vertices, 3));
    geometry.setAttribute('normal', new BufferAttribute(normal, 3));
		
    // Define material properties for the wall
    const material = new MeshPhysicalMaterial({ side: DoubleSide, transparent: true, opacity: 0.5, color: '#FFFFFF' });
		
    // Create wall object
    const wall = new Wall(geometry, material, wallData.attributes);
    return wall;
}

// Reconstruct walls using walls data from the Vision API response
const wallsData: WallData[] = sceneData.walls;
wallsData.forEach((wallData: WallData) =&gt; {
    this.scene.add(Wall.createWall(wallData));
});

Note the GeometryIndices array, and ensure you maintain the order of the indices as demonstrated in the example above. This ensures consistency in defining the geometry of walls and other surfaces, facilitating proper rendering and interaction.

Explanation:

Wall Class: The Wall class represents a wall surface in the room. It stores information about the wall's normal vector and other attributes.

Creating Wall Geometry: The createWall method constructs the geometry for a wall based on the provided wall data. It defines vertices and normals for the wall geometry and creates a buffer geometry object.

Defining Material: The wall is given a MeshPhysicalMaterial with properties such as color and opacity. Making walls transparent ensures that only models loaded into the application are shown, enhancing the overall visualization.

Reconstructing Walls: Using the wallsData obtained from the Vision API response, iterate through each wall data and create corresponding wall objects. These wall objects are then added to the scene for visualization.

By reconstructing interior surfaces such as walls using the provided data, you can accurately place models within the room and create a realistic virtual environment.

Blending Results with Original Image:

To seamlessly integrate rendering results with the original image of the interior, you can utilize shader programs within the EffectComposer. This allows for a smooth blending of virtual elements with the real-world environment captured in the image.

Configuring EffectComposer:

Configure the EffectComposer to include passes tailored to your application's needs. Consider the following setup with three passes:

RenderPass: This is the basic rendering pass responsible for rendering your scene.

ShaderPass (FXAAShader): Apply anti-aliasing techniques to enhance the visual quality of the rendered results.

Custom ShaderPass for Blending Results: Use a custom shader program to blend the rendering results with the original image.

Custom ShaderPass Configuration:

In the uniforms section of the shader pass configuration, declare the variables to be used in the shader program:

this.shaderPass = new ShaderPass({
    uniforms: {
        "tDiffuse": { value: null },
        "tMask": { value: null },
        "tBackground": { value: null }
    },
    vertexShader: vertexShader,
    fragmentShader: fragmentShader
});

tDiffuse: Rendering results from the RenderPass.

tMask: The segmentation mask obtained from the Vision API.

tBackground: Original image of the interior.

Updating ShaderPass Values:

You can dynamically update the values of the shader pass properties, such as tBackground and tMask, when necessary. For example, upon loading textures or changing the room in your application:

this.shaderPass.tBackground.value = bgTexture; 
this.shaderPass.tMask.value = maskTexture;

Adding Passes to EffectComposer Ensure the correct order of passes by adding them to the EffectComposer in sequence:

this.composer.addPass(renderPass); 
this.composer.addPass(this.fxaaPass); 
this.composer.addPass(this.shaderPass);

Rendering the Scene Launch the rendering process by calling the render method:

this.composer.render();

Bind the render method to the requestAnimationFrame event to continuously update the rendering.

Shader programs

Shaders are powerful programs used in computer graphics to manipulate and render images, making them an integral part of modern rendering pipelines. They are written in specialized programming languages, such as GLSL (OpenGL Shading Language) for WebGL applications, and they run directly on the GPU (Graphics Processing Unit).

Types of Shaders: There are two primary types of shaders:

Vertex Shader: This shader operates on each vertex (or point) of a 3D model and is responsible for transforming the vertex positions from 3D space to 2D screen space. It computes attributes such as color, texture coordinates, and normals for each vertex.

Fragment Shader (Pixel Shader): The fragment shader, on the other hand, processes each pixel of the rendered image. It determines the final color of each pixel by interpolating values from nearby vertices and applying lighting, texturing, and other effects.

How Shaders Work:

When rendering a scene, the GPU executes the vertex and fragment shaders for each vertex and pixel, respectively. This process generates the final image that is displayed on the screen. Here's a brief overview of how shaders work:

Vertex Processing:

Vertex shaders receive as input the geometry of 3D objects in the scene, along with transformation matrices.

They perform transformations on each vertex to position and orient it correctly in 3D space.

Additionally, vertex shaders can calculate attributes such as texture coordinates and normals.

Rasterization:

After vertex processing, the GPU rasterizes the primitives (triangles, lines, points) into fragments, or pixels, on the screen.

Each fragment corresponds to a pixel in the final image.

Fragment Processing:

Fragment shaders take the rasterized fragments as input and determine the color of each pixel.

They perform various operations, such as lighting calculations, texture mapping, and applying post-processing effects.

The final color of each pixel is computed based on the calculations performed in the fragment shader.

Using Shaders in Rendering Pipelines: Shaders play a crucial role in rendering pipelines by enabling developers to implement complex rendering techniques and achieve visually stunning graphics. They allow for real-time manipulation of geometry, lighting, and materials, resulting in immersive and interactive experiences for users.

Example Shader Usage: In the context of blending rendering results with the original image, shaders are utilized to apply custom image processing and compositing operations. By writing custom fragment shaders, you can blend textures, apply filters, and create visual effects, enhancing the realism and visual fidelity of rendered scenes.

Vertex Shader:

varying vec2 vUv;
varying vec2 vPosition;
void main() {
    vPosition = position.xy;
    vUv = uv;
    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}

Fragment Shader:

varying vec2 vPosition;
uniform sampler2D tDiffuse;
uniform sampler2D tMask;
uniform sampler2D tBackground;
void main() {
    vec4 background = texture2D(tBackground, vUv);
    vec4 texel = texture2D(tDiffuse, vUv);
    float wallMask = texture2D(tMask, vUv).b;
		
    // Apply mask to texel
    texel = texel * wallMask;
		
    // Conditionally blend texel with background
    if (texel.r &gt; 0.0 || texel.g &gt; 0.0 || texel.b &gt; 0.0) {
        background = vec4(0.0);
    }
    gl_FragColor = texel + background;
}

Make sure to follow the instructions to build an interactive application with Vision API and Three.js successfully.

Wizart Documentation