Manual retargeting - How to compute target pose bone positions correctly?

12 Views Asked by At

I am retargeting a cat object (target) from another object's (source) bone positions in Unity/C# environment.

  • What I know: Source rest bone positions, target rest bone positions, source pose bone positions
  • What I don't know: Source model, source rotations
  • What I want: target pose bone positions (no rotations) retargeted from the source pose bone positions

My Current Approach:
I compute the local rest and pose directions for each bone in the source (from immediate parent to each bone), and then apply the rotation from the source rest to the source pose in the target model. I'm scaling the bone lengths based on a ratio calculated from the rest poses of the source and target.

Current Output:
The spine positions and the relative positions of leg bones seem correct. However, the first leg bone's relative position from the body/spine is incorrect, causing all child leg bone positions to be incorrect as well.

Possible Issue:
I think it is incorrect to calculate and apply rotation in this way. With different rest bone orientations of the two objects, the rotation becomes incorrect. In the spine and the bones in the leg chain, the relative direction from the parent is similar in the source and the target. But the direction from the spine4 (the spine bone that connects the front legs) to the first bones of the front legs are opposite in the two models. As I apply the rotation from source rest dir -> source pose dir to the target rest dir, it yields a wrong result. I thought I compute relative directions from the immediate parent so orientation won't be an issue, but I'm struggling for days. Please help :(


Target pose calculation code
For each frame, I fix the root position first to be identical to that of the source, and then I iterate down the hierarchy to compute next bone's position.

// 1. Initial Setup - Rest Pose Positions
List<Vector3> sourceRestPositions = new List<Vector3> { /* the bone positions at source rest pose */ };
List<Vector3> targetRestPositions = new List<Vector3> { /* the bone positions at target rest pose */ };
    
// Source & Target - Calculate relative lengths and unit directions 
Vector3[] sourceRestDirections = new Vector3[numJoints];
Vector3[] targetRestDirections = new Vector3[numJoints];
float[] restLengthRatios = new float[numJoints];
for (int i = 0; i < numJoints; i++)
{
    if (bones[i] == null) continue; // Skip if no corresponding bone in the target 
        
    // Source
    var sourceParentRestPosition = sourceRestPositions[(int)GetParentBone(i)];
    var sourceLength = Vector3.Distance(sourceParentRestPosition, sourceRestPositions[i]);
    sourceRestDirections[i] = (sourceRestPositions[i] - sourceParentRestPosition).normalized;
        
    // Target
    var targetParentRestPosition = targetRestPositions[(int)GetParentBone(i)];
    var targetLength = Vector3.Distance(targetParentRestPosition, targetRestPositions[i]);
    targetRestDirections[i] = (targetRestPositions[i] - targetParentRestPosition).normalized;
        
    // Length ratio
    restLengthRatios[i] = targetLength / sourceLength;
}
    
// 2. Retargeting
// Calculate retargeted joint positions
_targetJointSequence = new Vector3[sourceJointSequence.Length][];
for (int f = 0; f < sourceJointSequence.Length; f++)
{
    // Fix root position
    _targetJointSequence[f] = new Vector3[numJoints];
    _targetJointSequence[f][(int)Bone.Spine0_BLegConnect] = sourceJointSequence[f][(int)Bone.Spine0_BLegConnect];

    foreach (var childIdx in retargetOrder)
    {
        // Compute the local directions in source pose
        int parentIdx = (int)GetParentBone(childIdx); // Get immediate parent
        Vector3 sourcePoseParentPosition = sourceJointSequence[f][parentIdx];
        Vector3 sourcePoseChildPosition = sourceJointSequence[f][childIdx];
            
        Vector3 sourcePoseDirection = (sourcePoseChildPosition - sourcePoseParentPosition).normalized;
            
        // Compute the length and rotation in source pose
        float sourcePoseBoneLength = Vector3.Distance(sourcePoseChildPosition, sourcePoseParentPosition);
        Quaternion sourceRotation = Quaternion.FromToRotation(sourceRestDirections[childIdx], sourcePoseDirection);
            
        // Apply the rotation to the target
        Vector3 targetPoseDirection = sourceRotation * targetRestDirections[childIdx];
            
        // Compute the global positions in target pose
        Vector3 targetParentPosePosition = _targetJointSequence[f][parentIdx];
        Vector3 targetChildPosePosition = targetParentPosePosition + targetPoseDirection * restLengthRatios[childIdx] * sourcePoseBoneLength;
            
        // Set the target pose position
        _targetJointSequence[f][childIdx] = targetChildPosePosition;
    }

Bone positions at rest position: Source (left) and Target (right)
The joints I use for retargeting are marked with spheres (blue: head, red: spine, white: legs, green: tail) enter image description here enter image description here

Bone positions at a pose: Source (left) and Target (right)
enter image description here enter image description here

At a pose similar to the rest pose but different orientation enter image description here enter image description here

0

There are 0 best solutions below