Parameterizing poses with Anny¶

First we start by instantiating the model, and a few helper objects for visualization.

In [1]:
import torch
import roma # A PyTorch library useful to deal with space transformations.
import anny # The main library for the Anny model.
import trimesh # For 3D mesh visualization.

# Instantiate the model.
# Remark: the first instantiation may take a while. Latter calls will be faster thanks to caching.
anny_model = anny.create_fullbody_model(eyes=True, tongue=False, remove_unattached_vertices=True).to(dtype=torch.float32, device='cpu')

# Some helper objects for visualization.
trimesh_scene_transform = roma.Rigid(linear=roma.euler_to_rotmat('x', [-90.], degrees=True), translation=None).to_homogeneous().cpu().numpy()

mesh_material = trimesh.visual.material.PBRMaterial(baseColorFactor=[0.6, 0.8, 0.7, 0.5],
                                                        metallicFactor=0.5,
                                                        doubleSided=False,
                                                        alphaMode='BLEND')

world_axis = trimesh.creation.axis(axis_length=1.)
axis = trimesh.creation.axis(axis_length=0.1)

Warp 1.8.1 initialized:
   CUDA Toolkit 12.8, Driver 13.0
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA RTX 1000 Ada Generation Laptop GPU" (6 GiB, sm_89, mempool enabled)
   Kernel cache:
some_path

Pose parameterization¶

A pose is described by a set of 4x4 rigid transformation matrices, one per bone. Here we set all bone transformations to be the identity, which corresponds to the rest pose:

Remark: we use direct transforms and column vectors conventions, so that coordinates $X_i$ of a vector are transformed by a matrix $M_{i,j}$ into $Y_i = \sum_k M_{ik} X_k$.

In [2]:
# The code supports batched inputs. Here we use a batch size of 1.
batch_size = 1
pose_parameters = {label: torch.eye(4)[None].repeat(batch_size, 1, 1) for label in anny_model.bone_labels}
output = anny_model(pose_parameters=pose_parameters)

mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze(0), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
scene.apply_transform(trimesh_scene_transform)
scene.show()

Module anny.skinning.warp_skinning 4408221 load on device 'cpu' took 3.19 ms  (cached)
Out[2]:

Alternatively, pose parameters can be specified by a single tensor stacking all bone transformations. The ordering of the bones is consistent with the one of anny_model.bone_labels.

In [3]:
pose_parameters = torch.eye(4, dtype=torch.float32)[None, None].repeat(batch_size, anny_model.bone_count, 1, 1)

output = anny_model(pose_parameters=pose_parameters)

mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze(0), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[3]:

Setting bone pose¶

In general settings, the transformation associated to a bone affects this bone and its children along the kinematic chain of the model. It describes the transformation applied to the bone with respect to its parent bone, and is expressed with respect to the rest coordinates system of this bone.

Let us consider a particular bone, the left shoulder for example. We show below its default pose:

In [4]:
bone_id = anny_model.bone_labels.index('shoulder01.L')
print(f"Bone 'shoulder01.L' has id {bone_id}.")

pose_parameters = {label: torch.eye(4)[None] for label in anny_model.bone_labels}
output = anny_model(pose_parameters=pose_parameters)
mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(mesh)
scene.add_geometry(world_axis)
scene.add_geometry(axis, transform=output["bone_poses"].squeeze(0)[bone_id].cpu().numpy())
scene.apply_transform(trimesh_scene_transform)
scene.show()

Bone 'shoulder01.L' has id 49.
Out[4]:

Let us apply a rotation around the Z axis (in blue) to rotate this bone with respect to its parent. Note how it affects the whole arm.

In [5]:
pose_parameters = {label: torch.eye(4)[None] for label in anny_model.bone_labels}
pose_parameters["shoulder01.L"] = roma.Rigid(roma.euler_to_rotmat("z", [30.], degrees=True), translation=None).to_homogeneous()[None]
output = anny_model(pose_parameters=pose_parameters)
mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(mesh)
scene.add_geometry(world_axis)
scene.add_geometry(axis, transform=output["bone_poses"].squeeze(0)[bone_id].cpu().numpy())
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[5]:

Global transformation¶

The root bone is the first bone of the kinematic chain of the model. Changing the root transformation affects the model globally.

Anny supports different pose parameterizations that behave differently. The choice of parameterization can be changed at instantiation through the default_pose_parameterization argument, or at each call of Anny using the pose_parameterization argument.

pose_parameterization="root_relative"

In root_relative mode, the transformation associated with the root bone describes the pose of the root bone, with respect to the world coordinate system. When using the identity tranformation, the coordinate system of the root bone is therefore aligned with the world coordinate system as illustrated below:

In [6]:
pose_parameters = {label: torch.eye(4)[None] for label in anny_model.bone_labels}
output = anny_model(pose_parameters=pose_parameters, pose_parameterization="root_relative")

mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze().cpu().numpy(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
bone_id = 0
scene.add_geometry(axis, transform=output["bone_poses"].squeeze(0)[bone_id])
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[6]:

Using this parameterization, one can easily control the pose of the root bone, but the default orientation of the model may be less intuitive for the user.

In [7]:
pose_parameters = {label: torch.eye(4)[None] for label in anny_model.bone_labels}
# Translate the root bone of 20cm along the X axis.
pose_parameters['root'] = roma.Rigid(linear=torch.eye(3), translation=torch.tensor([0.2, 0, 0.])).to_homogeneous()[None]

output = anny_model(pose_parameters=pose_parameters, pose_parameterization="root_relative")

mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze().cpu().numpy(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
bone_id = 0
scene.add_geometry(axis, transform=output["bone_poses"].squeeze(0)[bone_id])
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[7]:

pose_parameterization="root_relative_world" (the default)

In root_relative_world mode (the default fullbody parameterization), the model in resting position is centered on the root joint and oriented upright, assuming the Z axis of the world coordinate pointing upwards. The root transformation describes a transformation applied to this default configuration, and is expressed with respect to world coordinates.

In [8]:
pose_parameters = {label: torch.eye(4)[None] for label in anny_model.bone_labels}
# Rotate the model of 20 degrees along the vertical Z axis and translate it of 20cm along the X world axis.
pose_parameters['root'] = roma.Rigid(linear=roma.euler_to_rotmat("z", [-20.], degrees=True), translation=torch.tensor([0.2, 0, 0.])).to_homogeneous()[None]

output = anny_model(pose_parameters=pose_parameters, pose_parameterization="root_relative_world")

mesh = trimesh.Trimesh(vertices=output['vertices'].squeeze().cpu().numpy(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
bone_id = 0
scene.add_geometry(axis, transform=output["bone_poses"].squeeze(0)[bone_id])
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[8]:

pose_parameterization="absolute" and switching between representations¶

Lastly, in absolute mode, the pose parameterization consists of rigid transformations that describe the pose of each bone individually. An absolute parameterization can be used to perform local deformation, as illustrated below. Here, we use the method get_pose_parameterization to convert a pose parameterization into an other.

In [9]:
# Retrieve an absolute parameterization of the previous output mesh.
absolute_parameters = anny_model.get_pose_parameterization(output, target_pose_parameterization="absolute").clone()

# Translate and scale the left upperarm bone (note how it does not affect the other bones of the kinematic chain).
bone_id = anny_model.bone_labels.index('upperarm01.L')
absolute_parameters[:,bone_id,2,3] += 0.2 # translate of 20cm in absolute Z direction
absolute_parameters[:,bone_id,:3,:3] *= 1.2 # scale the bone by 20%

new_output = anny_model(pose_parameters=absolute_parameters, pose_parameterization="absolute")
mesh = trimesh.Trimesh(vertices=new_output['vertices'].squeeze().cpu().numpy(), faces=anny_model.faces, process=False)
mesh.visual = trimesh.visual.TextureVisuals(material=mesh_material)
scene = trimesh.Scene()
scene.add_geometry(world_axis)
scene.add_geometry(mesh)
scene.add_geometry(axis, transform=new_output["bone_poses"].squeeze(0)[bone_id])
scene.apply_transform(trimesh_scene_transform)
scene.show()
Out[9]: