Alternative models¶
In this tutorial, we show how to use different rigs and topologies with Anny.
Imports and helper functions¶
In [1]:
import torch
import numpy as np
import roma # A PyTorch library useful to deal with space transformations.
import anny # The main library for the Anny model.
import trimesh # For 3D mesh visualization.
# Create and show multiple rigs in one cell
from IPython.display import display, Markdown
import trimesh.viewer.notebook as nb
# Some helper objects for visualization.
trimesh_scene_transform = roma.Rigid(linear=roma.euler_to_rotmat('x', [-90.], degrees=True), translation=None).to_homogeneous().cpu().numpy()
mesh_material = trimesh.visual.material.PBRMaterial(baseColorFactor=[0.6, 0.8, 0.7, 0.5],
metallicFactor=0.5,
doubleSided=False,
alphaMode='BLEND')
world_axis = trimesh.creation.axis(axis_length=1.)
axis = trimesh.creation.axis(axis_length=0.1)
def add_skeleton_to_scene(scene, model, output):
# Add bones visualization
bone_heads, bone_tails = output['bone_heads'], output['bone_tails']
bone_heads = bone_heads.squeeze(dim=0).cpu()
bone_tails = bone_tails.squeeze(dim=0).cpu()
#bone_colors = [[0.8, 0.4, 0.2, 1.0], [0.8, 0.2, 0.4, 1.0]]
bone_colors = [[0.8, 0.3, 0.3, 1.0]]
bone_visuals = [trimesh.visual.TextureVisuals(material=trimesh.visual.material.PBRMaterial(baseColorFactor=color,
metallicFactor=0.,
roughnessFactor=1.,
doubleSided=True,
alphaMode='BLEND')) for color in bone_colors]
for i in range(len(bone_heads)):
bone_head = bone_heads[i]
bone_tail = bone_tails[i]
cylinder = trimesh.creation.cylinder(radius=0.005, height=torch.norm(bone_tail - bone_head).item(), sections=16)
t = (bone_head + bone_tail) / 2
M = roma.special_gramschmidt(torch.stack([bone_tail - bone_head, torch.tensor([0., 0., 1.], dtype=torch.float32)], dim=-1))
R = torch.stack([M[:, 2], M[:, 1], M[:,0]], dim=-1)
cylinder.visual = bone_visuals[i % len(bone_colors)]
scene.add_geometry(cylinder, transform=roma.Rigid(R, t).to_homogeneous().numpy(),
node_name=f"bone_{model.bone_labels[i]}")
# Add some spheres at the joints
bone_poses = output["bone_poses"].squeeze(dim=0).cpu()
joint_sphere = trimesh.creation.icosphere(radius=0.008, subdivisions=2)
joint_sphere.visual = trimesh.visual.TextureVisuals(material=trimesh.visual.material.PBRMaterial(
baseColorFactor=[0.1, 0.1, 0.1, 1.0],
metallicFactor=0.5,
roughnessFactor=1.,
doubleSided=True,
alphaMode='OPAQUE'))
for i in range(len(bone_poses)):
scene.add_geometry(joint_sphere, transform=bone_poses[i], node_name=f"joint_{model.bone_labels[i]}")
Rigs and topology¶
Anny supports two main rigs: "default" and "mixamo".
Bones that are not useful for your application can be removed of the default rig. Using "default-notoes" will ignore bones animating individual toes, for example.
Anny also supports various mesh topologies. A topology such as "notoes_collapse5pc" provides coarser mesh output for example, allowing to speed up inference and reduce memory consumption.
We visualize below a few combinations of meshes and topologies supported by the model.
In [2]:
viewers = []
for rig, topology in [("default", "default"),
("mixamo", "default"),
("default-notoes-noexpression-nobreasts", "default"),
("default-notoes-nohands-noexpression-nobreasts", "notoes_collapse5pc"),
]:
model = anny.create_fullbody_model(rig=rig, topology=topology, remove_unattached_vertices=True)
output = model(return_bone_ends=True)
mesh = trimesh.Trimesh(
vertices=output["vertices"].squeeze(0).cpu().numpy(),
faces=model.faces.cpu().numpy()
)
mesh.visual.material = mesh_material
scene = trimesh.Scene([mesh])
add_skeleton_to_scene(scene, model, output)
scene.apply_transform(trimesh_scene_transform)
# Convert to a notebook widget/HTML
viewers.append(Markdown(f"#### '{rig}' rig ({model.bone_count} bones)"))
viewers.append(Markdown( " - " + ", ".join([label for label in model.bone_labels])))
viewers.append(Markdown(f"#### '{topology}' topology ({len(output['vertices'].squeeze(0))} vertices, {len(model.faces)} faces)"))
viewers.append(nb.scene_to_notebook(scene))
# Display all viewers
display(*viewers)
Warp 1.8.1 initialized:
CUDA Toolkit 12.8, Driver 13.0
Devices:
"cpu" : "x86_64"
"cuda:0" : "NVIDIA RTX 1000 Ada Generation Laptop GPU" (6 GiB, sm_89, mempool enabled)
Kernel cache:
some_path
Module anny.skinning.warp_skinning 0b51ecc load on device 'cpu' took 0.80 ms (cached)
Reducing the number of influencing bones to 5 Module anny.skinning.warp_skinning c07988a load on device 'cpu' took 0.24 ms (cached)
Module anny.skinning.warp_skinning 0b51ecc load on device 'cpu' took 0.22 ms (cached)
'default' rig (163 bones)¶
- root, pelvis.L, upperleg01.L, upperleg02.L, lowerleg01.L, lowerleg02.L, foot.L, toe1-1.L, toe1-2.L, toe2-1.L, toe2-2.L, toe2-3.L, toe3-1.L, toe3-2.L, toe3-3.L, toe4-1.L, toe4-2.L, toe4-3.L, toe5-1.L, toe5-2.L, toe5-3.L, pelvis.R, upperleg01.R, upperleg02.R, lowerleg01.R, lowerleg02.R, foot.R, toe1-1.R, toe1-2.R, toe2-1.R, toe2-2.R, toe2-3.R, toe3-1.R, toe3-2.R, toe3-3.R, toe4-1.R, toe4-2.R, toe4-3.R, toe5-1.R, toe5-2.R, toe5-3.R, spine05, spine04, spine03, spine02, breast.L, breast.R, spine01, clavicle.L, shoulder01.L, upperarm01.L, upperarm02.L, lowerarm01.L, lowerarm02.L, wrist.L, finger1-1.L, finger1-2.L, finger1-3.L, metacarpal1.L, finger2-1.L, finger2-2.L, finger2-3.L, metacarpal2.L, finger3-1.L, finger3-2.L, finger3-3.L, metacarpal3.L, finger4-1.L, finger4-2.L, finger4-3.L, metacarpal4.L, finger5-1.L, finger5-2.L, finger5-3.L, clavicle.R, shoulder01.R, upperarm01.R, upperarm02.R, lowerarm01.R, lowerarm02.R, wrist.R, finger1-1.R, finger1-2.R, finger1-3.R, metacarpal1.R, finger2-1.R, finger2-2.R, finger2-3.R, metacarpal2.R, finger3-1.R, finger3-2.R, finger3-3.R, metacarpal3.R, finger4-1.R, finger4-2.R, finger4-3.R, metacarpal4.R, finger5-1.R, finger5-2.R, finger5-3.R, neck01, neck02, neck03, head, jaw, special04, oris02, oris01, oris06.L, oris07.L, oris06.R, oris07.R, tongue00, tongue01, tongue02, tongue03, tongue04, tongue07.L, tongue07.R, tongue06.L, tongue06.R, tongue05.L, tongue05.R, levator02.L, levator03.L, levator04.L, levator05.L, levator02.R, levator03.R, levator04.R, levator05.R, special01, oris04.L, oris03.L, oris04.R, oris03.R, oris06, oris05, special03, levator06.L, levator06.R, special06.L, special05.L, eye.L, orbicularis03.L, orbicularis04.L, special06.R, special05.R, eye.R, orbicularis03.R, orbicularis04.R, temporalis01.L, oculi02.L, oculi01.L, temporalis01.R, oculi02.R, oculi01.R, temporalis02.L, risorius02.L, risorius03.L, temporalis02.R, risorius02.R, risorius03.R
'default' topology (13718 vertices, 13710 faces)¶
'mixamo' rig (52 bones)¶
- mixamorig:Hips, mixamorig:LeftUpLeg, mixamorig:LeftLeg, mixamorig:LeftFoot, mixamorig:LeftToeBase, mixamorig:RightUpLeg, mixamorig:RightLeg, mixamorig:RightFoot, mixamorig:RightToeBase, mixamorig:Spine, mixamorig:Spine1, mixamorig:Spine2, mixamorig:LeftShoulder, mixamorig:LeftArm, mixamorig:LeftForeArm, mixamorig:LeftHand, mixamorig:LeftHandIndex1, mixamorig:LeftHandIndex2, mixamorig:LeftHandIndex3, mixamorig:LeftHandMiddle1, mixamorig:LeftHandMiddle2, mixamorig:LeftHandMiddle3, mixamorig:LeftHandPinky1, mixamorig:LeftHandPinky2, mixamorig:LeftHandPinky3, mixamorig:LeftHandRing1, mixamorig:LeftHandRing2, mixamorig:LeftHandRing3, mixamorig:LeftHandThumb1, mixamorig:LeftHandThumb2, mixamorig:LeftHandThumb3, mixamorig:Neck, mixamorig:Head, mixamorig:RightShoulder, mixamorig:RightArm, mixamorig:RightForeArm, mixamorig:RightHand, mixamorig:RightHandIndex1, mixamorig:RightHandIndex2, mixamorig:RightHandIndex3, mixamorig:RightHandMiddle1, mixamorig:RightHandMiddle2, mixamorig:RightHandMiddle3, mixamorig:RightHandPinky1, mixamorig:RightHandPinky2, mixamorig:RightHandPinky3, mixamorig:RightHandRing1, mixamorig:RightHandRing2, mixamorig:RightHandRing3, mixamorig:RightHandThumb1, mixamorig:RightHandThumb2, mixamorig:RightHandThumb3
'default' topology (13718 vertices, 13710 faces)¶
'default-notoes-noexpression-nobreasts' rig (74 bones)¶
- root, pelvis.L, upperleg01.L, upperleg02.L, lowerleg01.L, lowerleg02.L, foot.L, pelvis.R, upperleg01.R, upperleg02.R, lowerleg01.R, lowerleg02.R, foot.R, spine05, spine04, spine03, spine02, spine01, clavicle.L, shoulder01.L, upperarm01.L, upperarm02.L, lowerarm01.L, lowerarm02.L, wrist.L, finger1-1.L, finger1-2.L, finger1-3.L, metacarpal1.L, finger2-1.L, finger2-2.L, finger2-3.L, metacarpal2.L, finger3-1.L, finger3-2.L, finger3-3.L, metacarpal3.L, finger4-1.L, finger4-2.L, finger4-3.L, metacarpal4.L, finger5-1.L, finger5-2.L, finger5-3.L, clavicle.R, shoulder01.R, upperarm01.R, upperarm02.R, lowerarm01.R, lowerarm02.R, wrist.R, finger1-1.R, finger1-2.R, finger1-3.R, metacarpal1.R, finger2-1.R, finger2-2.R, finger2-3.R, metacarpal2.R, finger3-1.R, finger3-2.R, finger3-3.R, metacarpal3.R, finger4-1.R, finger4-2.R, finger4-3.R, metacarpal4.R, finger5-1.R, finger5-2.R, finger5-3.R, neck01, neck02, neck03, head
'default' topology (13718 vertices, 13710 faces)¶
'default-notoes-nohands-noexpression-nobreasts' rig (36 bones)¶
- root, pelvis.L, upperleg01.L, upperleg02.L, lowerleg01.L, lowerleg02.L, foot.L, pelvis.R, upperleg01.R, upperleg02.R, lowerleg01.R, lowerleg02.R, foot.R, spine05, spine04, spine03, spine02, spine01, clavicle.L, shoulder01.L, upperarm01.L, upperarm02.L, lowerarm01.L, lowerarm02.L, wrist.L, clavicle.R, shoulder01.R, upperarm01.R, upperarm02.R, lowerarm01.R, lowerarm02.R, wrist.R, neck01, neck02, neck03, head