Guide for applying 2D texture labels onto a 3D object

Question

Guide for applying 2D texture labels onto a 3D object

I am working with a texture file applied to a complex 3D object in the format of an .OBJ file. This texture contains various labels, and I would like to generate additional objects for these distinctive labels at their corresponding locations on the 3D model, allowing for interactive functionality.

While researching a solution, I came across a thread that utilizes outdated methods such as Geometry: Three.js - calculate 3D coordinates from UV coordinates

I am unsure how to adapt this approach to the latest version of ThreeJS, or if there is a more effective method to accomplish my objective.

For context regarding the specific 3D object, it pertains to human muscle anatomy, with the texture serving as both labels and color shading for the muscles.

As a newcomer to ThreeJS and 3D modeling, I acknowledge that I may be missing some crucial details.

typescript three.js 3d-modelling 3d-model

Answer 1

Answer №1

It is unlikely that you can directly make points on your texture interactive, but have you explored the option of using CSS2DRenderer?

Check out some examples below:
css2d_label
molecules

To implement this, I suggest creating a button or label element for each point within the DOM, or reference an existing element with getElementById. Then, create a new CSS2DObject for each label/button, specify their positions in relation to your 3D object, and add them to your scene.

const labelOne = document.getElementById('labelOne');
const labelTwo = document.getElementById('labelTwo');

const labelRenderer = new CSS2DRenderer();
labelRenderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(labelRenderer.domElement);

const labelObjOne = new CSS2DObject(labelOne.nativeElement);
labelObjOne.position.set(0, 0, 0);
scene.add(labelObjOne);

const labelObjTwo = new CSS2DObject(labelTwo.nativeElement);
labelObjTwo.position.set(0, 1, 0);
scene.add(labelObjTwo);

This method may require you to remove any existing labels from your texture for a cleaner look.

Answer 2

It is unlikely that you can directly make points on your texture interactive, but have you explored the option of using CSS2DRenderer?

Check out some examples below:
css2d_label
molecules

To implement this, I suggest creating a button or label element for each point within the DOM, or reference an existing element with getElementById. Then, create a new CSS2DObject for each label/button, specify their positions in relation to your 3D object, and add them to your scene.

const labelOne = document.getElementById('labelOne');
const labelTwo = document.getElementById('labelTwo');

const labelRenderer = new CSS2DRenderer();
labelRenderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(labelRenderer.domElement);

const labelObjOne = new CSS2DObject(labelOne.nativeElement);
labelObjOne.position.set(0, 0, 0);
scene.add(labelObjOne);

const labelObjTwo = new CSS2DObject(labelTwo.nativeElement);
labelObjTwo.position.set(0, 1, 0);
scene.add(labelObjTwo);

This method may require you to remove any existing labels from your texture for a cleaner look.

Answer 3

Answer №2

After conducting extensive research, I finally discovered the solution I had been seeking.

To view my code, which unfortunately is too lengthy to display here, please visit https://github.com/tedik123/Obj-Extraction

The crucial first step was realizing that in my case with the .obj file, everything was stored in parallel UVs, vertices (faces), and normals (applicable to any ThreeJS object). The challenge lay in the fact that the UVs and faces were organized as 1-dimensional arrays. Additional effort was required to structure them into triplets for indexing by face index, where each index retrieves all the vertices, UVs, and normals comprising one face - each face consisting of 3 vertices, 3 normals, and 3 UVs. Preprocessing in a manner where a face index yielded all relevant data streamlined most of the process. For further understanding, refer to this informative link:

Subsequently, I transitioned to Python for its user-friendliness in addressing a one-time task: extracting all pixels constituting a single label on the texture. Since each label bore a distinct color in my case, identifying and saving them via dictionary mapping proved efficient - associating the label as the key and the pixel composition as the value.

The third pivotal realization was that UVs form triangles! Each face represents a triangle, featuring a corresponding textured triangle. Fortunately, UVs are confined to 2D space while faces inhabit 3D space, simplifying spatial alignment given our preprocessed UVs, faces, and normals indexed by face. This arrangement obviates the need for manual mapping.

Further steps involved dissecting the UV triangles* by converting them to integer-based pixel coordinates and identifying points within each triangle's perimeter, subsequently storing these in a hash map corresponding to their respective face indices. Although time-consuming initially, this optimization significantly accelerates subsequent searches. Running this process once allows for future expedited executions, especially if saved to a file.

I utilized the following code references to aid me in deciphering triangles: Determining all discrete points inside a triangle and Get all points of a straight line in python

By cross-referencing our label pixels against the aforementioned dictionary to determine their corresponding faces, we can then import this information back into JavaScript. Creating a new Buffer Geometry object entails adding the vertices composing the desired label face(s) before generating the mesh (ensuring color setting for visibility) along with adjustments for position and scale relative to the original object. This process yields a 3D object overlaying the designated label area.

*Alternatively, one could iterate through each pixel label to ascertain its presence within a UVs triangle, albeit at O(n^2) complexity - potentially impractical for large textures or complex objects with numerous vertices. While more precise, such an approach sacrifices speed efficiency compared to my method operating at O(n) with marginal accuracy trade-offs.

In upcoming revisions, I intend to append concise functioning code snippets to this response; although presently, substantial cleanup remains imperative!

Answer 4