I'm using LibGDX and Bullet to create a screen for my Android app for painting on a textured 3D models. I'm trying to get the coordinate on the texture from the ray hit on the 3D model. But I'm completely clueless how to do this, there's no documentation regarding this, ChatGPT didn't helped also))
I'm already able to display the 3D model (g3db) with its texture (png) and perform the raycast on it. But how I can now find out what the coordinate on texture the user pointed to with raycast?
if (Gdx.input.isTouched) {
callback.closestHitFraction = 1f
callback.collisionObject = null
val ray = cam.getPickRay(Gdx.input.x.toFloat(), Gdx.input.y.toFloat())
rayFromWorld.set(ray.origin)
rayToWorld.set(ray.direction).scl(5f).add(ray.origin)
bulletPhysicsSystem?.raycast(rayFromWorld, rayToWorld, callback)
if (callback.hasHit()) {
val collisionObject = callback.collisionObject
if (collisionObject is btRigidBody) {
println("Hit detected!")
}
}
}
Update
I've thrown away the Bullet usage for this task. Now I'm trying to use the barycentric coordinates of the intersection point to find out the UV coordinates and finally convert them to the texture coordinates.
The flow I came up with:
- Obtaining all triangles from the model
- Finding the ray cast hit intersection triangle and point
- Conversion of the intersection point and triangle to barycentric coordinates
- Conversion of barycentric coordinates to UV values
- Conversion of UV values to the actual texture coordinates.
- Attempt to draw on the obtained texture coordinates to see if it works.
I'm able to obtain sometimes the right coordinates but it's completely unreliable, drawing on the front face completely brings drawing points to the back face, and in general even on other faces the texture coordinates I'm getting are wrong somehow depending on the model rotation. video
I'm completely out of any variants of what is going wrong here
if (Gdx.input.isTouched) {
val ray = cam.getPickRay(Gdx.input.x.toFloat(), Gdx.input.y.toFloat())
val meshPart = instance!!.model.meshParts[0]
val numVertices = meshPart.mesh.numVertices
val vertexSize = meshPart.mesh.vertexSize / 4
val vertices = FloatArray(numVertices * vertexSize)
meshPart.mesh.getVertices(0, vertices.size, vertices)
val indices = ShortArray(meshPart.mesh.numIndices)
meshPart.mesh.getIndices(indices)
println("Vertices: ${vertices.asList()}")
println("Vertex size: $vertexSize")
println("Num vertices: $numVertices")
println("Indices: ${indices.asList()}")
val triangles = mutableListOf<ModelTriangle>()
for (i in indices.indices step 3) {
val stride = meshPart.mesh.vertexSize / 4
val vertexIndex1 = indices[i].toInt() * stride
val vertexIndex2 = indices[i + 1].toInt() * stride
val vertexIndex3 = indices[i + 2].toInt() * stride
val vertex1 = Vector3(
vertices[vertexIndex1],
vertices[vertexIndex1 + 1],
vertices[vertexIndex1 + 2]
)
val vertex2 = Vector3(
vertices[vertexIndex2],
vertices[vertexIndex2 + 1],
vertices[vertexIndex2 + 2]
)
val vertex3 = Vector3(
vertices[vertexIndex3],
vertices[vertexIndex3 + 1],
vertices[vertexIndex3 + 2]
)
val vertexAttributes = meshPart.mesh.vertexAttributes
val uvOffset =
vertexAttributes.findByUsage(VertexAttributes.Usage.TextureCoordinates).offset / 4
val uv1 = Vector2(
vertices[vertexIndex1 + uvOffset],
vertices[vertexIndex1 + uvOffset + 1]
)
val uv2 = Vector2(
vertices[vertexIndex2 + uvOffset],
vertices[vertexIndex2 + uvOffset + 1]
)
val uv3 = Vector2(
vertices[vertexIndex3 + uvOffset],
vertices[vertexIndex3 + uvOffset + 1]
)
val uvs = arrayOf(uv1, uv2, uv3)
val triangle = ModelTriangle(vertex1, vertex2, vertex3, uvs)
triangles.add(triangle)
}
println("Triangles: $triangles")
println("Triangles size: ${triangles.size}")
val intersectionPoint = Vector3()
val intersectionTriangle = triangles.find {
Intersector.intersectRayTriangle(
ray,
it.v1,
it.v2,
it.v3,
intersectionPoint
)
}
println("Intersection point: $intersectionPoint")
println("Is intersection found: ${intersectionTriangle != null}")
intersectionTriangle?.let { triangle ->
val instanceTexture =
instance!!.materials.first().get(TextureAttribute.Diffuse) as TextureAttribute
val uv = getIntersectionUV(intersectionPoint, triangle, triangle.uvs)
val textureX = (uv.x * instanceTexture.textureDescription.texture.width).toInt()
val textureY = (uv.y * instanceTexture.textureDescription.texture.height).toInt()
println("Texture coordinates: x: $textureX, y: $textureY")
val texturePixmap = textureToPixmap(instanceTexture.textureDescription.texture)
drawPointOnPixmap(texturePixmap, textureX, textureY, Color.RED)
val newTexture = Texture(texturePixmap)
instance!!.materials.first().set(TextureAttribute.createDiffuse(newTexture))
}
}
ModelTriangle class:
data class ModelTriangle(
val v1: Vector3,
val v2: Vector3,
val v3: Vector3,
val uvs: Array<Vector2>
) {
// equals and hashCode implementations automatically generated by AndroidStudio
}
getIntersectionUV:
private fun getIntersectionUV(
intersectionPoint: Vector3,
triangle: ModelTriangle,
uvs: Array<Vector2>
): Vector2 {
val barycentricCoords = toBarycoord3D(
intersectionPoint, triangle.v1, triangle.v2, triangle.v3
)
val isInsideTheTriangle = GeometryUtils.barycoordInsideTriangle(Vector2(barycentricCoords.x, barycentricCoords.y))
println("Is inside the triangle: $isInsideTheTriangle")
val uv1 = uvs[0]
val uv2 = uvs[1]
val uv3 = uvs[2]
return Vector2(
barycentricCoords.z * uv1.x + barycentricCoords.x * uv2.x + barycentricCoords.y * uv3.x,
barycentricCoords.z * uv1.y + barycentricCoords.x * uv2.y + barycentricCoords.y * uv3.y
)
}
toBarycoord3D:
private fun toBarycoord3D(
point: Vector3,
a: Vector3,
b: Vector3,
c: Vector3,
): Vector3 {
val v0 = Vector3().set(b).sub(a)
val v1 = Vector3().set(c).sub(a)
val v2 = Vector3().set(point).sub(a)
val d00 = v0.dot(v0)
val d01 = v0.dot(v1)
val d11 = v1.dot(v1)
val d20 = v2.dot(v0)
val d21 = v2.dot(v1)
val denom = d00 * d11 - d01 * d01
val barycentricOut = Vector3()
barycentricOut.x = (d11 * d20 - d01 * d21) / denom
barycentricOut.y = (d00 * d21 - d01 * d20) / denom
barycentricOut.z = 1.0f - barycentricOut.x - barycentricOut.y
return barycentricOut
}