Multi-camera Mocap Run

This guides you through the process of creating a mocap run when using multiple capture inputs.

To create a mocap run you will need to create an instance of MocapRun and provision that through a specific compute class.

However, you will need to go through a specific process to be able to do this:

1. Define camera settings and capture devices

2. Create a capture volume

   1. You will need to create capture inputs for this

   2. This can be either an aruco volume or an actor volume

3. Create multiple capture inputs

   1. This will be the inputs used for the mocap run

4. [Optional] NOT YET IMPLEMENTED Create actor skeletons for all actors in your capture inputs

5. Create a mocap run

1. Define camera settings and capture devices

You will need to define the camera settings and capture devices that you used

from moveai_mocap.models.captures.camera_settings import CameraSettings
from moveai_mocap.models.captures.capture_device import CaptureDevice

camera_settings = CameraSettings(
    model="GoProHero7-FHD",
    resolution_x=1920,
    resolution_y=1080,
    focal_length_x=1038.37,
    focal_length_y=1029.15,
    principal_point_x=953.06,
    principal_point_y=719.21,
    distortion_coeffs=(0, 0, 0, 0, 0),
)

# We had 4 capture devices each 90 degrees  apart from each other. They
# were all the same type of camera
capture_device_0 = CaptureDevice(
    device_id="capture_device_0",
    camera_settings=camera_settings,
)
capture_device_90 = CaptureDevice(
    device_id="capture_device_90",
    camera_settings=camera_settings,
)

capture_device_180 = CaptureDevice(
    device_id="capture_device_180",
    camera_settings=camera_settings,
)

capture_device_270 = CaptureDevice(
    device_id="capture_device_270",
    camera_settings=camera_settings,
)

2. Create a capture volume

Create capture inputs

We then define the VideoCaptureInputs that our capture devices created in order for us to create the volume. Ensure that only one actor is present on camera for these inputs.

from moveai_mocap.models.captures.capture_input import VideoCaptureInput
from moveai_mocap.models.resources.enums.resource_type import ResourceType
from moveai_mocap.models.resources.enums.service import Service
from moveai_mocap.models.resources.resource import Resource


video_capture_input_0 = VideoCaptureInput(
    capture_device=capture_device_0,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceType.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_0.mp4",
        service=Service.s3,
    ),
)
video_capture_input_90 = VideoCaptureInput(
    capture_device=capture_device_90,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceType.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_90.mp4",
        service=Service.s3,
    ),
)
video_capture_input_180 = VideoCaptureInput(
    capture_device=capture_device_180,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceType.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_180.mp4",
        service=Service.s3,
    ),
)
video_capture_input_270 = VideoCaptureInput(
    capture_device=capture_device_270,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceType.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_270.mp4",
        service=Service.s3,
    ),
)

Create capture

We then define our capture - which pulls all the video capture inputs together

from moveai_mocap.models.captures.capture import Capture
from moveai_mocap.models.captures.sync_methods.clap_audio import ClapAudioSync

capture = Capture(
      inputs=[
          video_capture_input_0,
          video_capture_input_90,
          video_capture_input_180,
          video_capture_input_270,
      ],
      # Other sync methods are available. If not provided the engine
      # assumes the videos are fully synced
      sync_method=ClapAudioSync(clap_window=(0.0, 2.0)),
)

Create actor volume run

Once all that is done we can create our ActorVolumeRun

from moveai_mocap.models.volumes.runs.actor import ActorVolumeRun
from moveai_mocap.models.volumes.enums.area import Area
from moveai_mocap.models.actors.actor import Actor
from moveai_mocap.models.resources.enums.service import Service

actor = Actor(
    name="actor_1",
    height=1.8,
)

actor_volume_run = ActorVolumeRun(
    # Only one actor is required for creating an actor volume
    actor=actor,
    # different size areas are available
    area=Area.large,
    capture=capture,
    volume_service=Service.s3,
    volume_uri_prefix="s3://my-bucket/path/to/directory",
)

Provision the engine

Now that we’ve generated the python object for our actor volume we can provision an engine with it

from moveai_mocap.compute.ec2 import EC2ComputeManager
from moveai_mocap.models.notifiers.sns import SNSNotifier

sns_notifier = SNSNotifier(
    endpoint="arn:aws:sns:",
    payload={"foo": "bar"},
)

compute = EC2ComputeManager(
    # configure these settings according to your AWS account
    iam_instance_profile_name="instance-profile-name",
    ami_id="ami-12345678",
    subnet_ids=["subnet-1"],
    security_group_ids=["sg-1"],
    notifier=sns_notifier,
)
# Calling provision will launch an EC2 instance and generate the output resources
# defined on `volume_uri_prefix` on the `actor_volume_run`
compute.provision(actor_volume_run)

3. Create capture inputs

Now that you have created your volume you can film as normal with multiple actors and create capture inputs. These capture inputs will be used to generate the mocap resources.

from moveai_mocap.models.captures.capture_input import VideoCaptureInput
from moveai_mocap.models.resources.enums.resource_type import ResourceType
from moveai_mocap.models.resources.enums.service import Service
from moveai_mocap.models.resources.resource import Resource

video_capture_input_0 = VideoCaptureInput(
    capture_device=capture_device_0,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceTypeEnum.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_0.mp4",
        service=Service.s3,
    ),
)
video_capture_input_90 = VideoCaptureInput(
    capture_device=capture_device_90,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceTypeEnum.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_90.mp4",
        service=Service.s3,
    ),
)
video_capture_input_180 = VideoCaptureInput(
    capture_device=capture_device_180,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceTypeEnum.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_180.mp4",
        service=Service.s3,
    ),
)
video_capture_input_270 = VideoCaptureInput(
    capture_device=capture_device_270,
    # The Resource defines that this file is stored on S3 and is an input_mp4 type
    input=Resource(
        resource_type=ResourceTypeEnum.video_capture_input,
        uri="s3://my-bucket/path/to/capture/capture_device_270.mp4",
        service=Service.s3,
    ),
)

Create capture

We then define our capture - which pulls all the video capture inputs together

from moveai_mocap.models.captures.capture import Capture
from moveai_mocap.models.captures.sync_methods.clap_audio import ClapAudioSync

capture = Capture(
      inputs=[
          video_capture_input_0,
          video_capture_input_90,
          video_capture_input_180,
          video_capture_input_270,
      ],
      # Other sync methods are available. If not provided the engine
      # assumes the videos are fully synced
      sync_method=ClapAudioSync(clap_window=(0.0, 2.0)),
)

4. [OPTIONAL] Create actor skeleton

Warning

Not yet implemented

You can optionally create a skeleton for one or more of the actors that are in your capture inputs. This will enable the engine to more accurately generate mocap resources

from moveai_mocap.models.actors.runs.actor import ActorSkeletonRun
from moveai_mocap.models.actors.actor import Actor
from moveai_mocap.models.resources.enums.service import Service


actor = Actor(
    name="actor_1",
    height=1.8,
)

actor_skeleton_run = ActorSkeletonRun(
    capture=capture,
    volume_service=Service.s3,
    volume_uri_prefix="s3://my-bucket/path/to/directory",
    actor=actor,
    # Where the actor skeleton file should be saved to
    actor_skeleton_uri_prefix="s3://my-bucket/path/to/skeleton/directory",
    actor_skeleton_service=Service.s3
)

Provision the engine

from moveai_mocap.compute.ec2 import EC2ComputeManager
from moveai_mocap.models.notifiers.sns import SNSNotifier

sns_notifier = SNSNotifier(
    endpoint="arn:aws:sns:",
    payload={"foo": "bar"},
)

compute = EC2ComputeManager(
    # configure these settings according to your AWS account
    iam_instance_profile_name="instance-profile-name",
    ami_id="ami-12345678",
    subnet_ids=["subnet-1"],
    security_group_ids=["sg-1"],
    notifier=sns_notifier,
)
# Calling provision will launch an EC2 instance and generate the output resources
# defined on `volume_uri_prefix` on the `actor_skeleton_run`
compute.provision(actor_skeleton_run)

5. Create a mocap run

Warning

actor_mapping not yet implemented

We can now use all the data created in the previous steps to provision a mocap run

from moveai_mocap.models.actors.actor_mapping import ActorMapping
from moveai_mocap.models.motion.runs.mocap import MocapRun
from moveai_mocap.models.resources.enums.service import Service
from moveai_mocap.models.resources.resource import Resource
from moveai_mocap.models.resources.enums.resource_type import ResourceType

mocap_run = MocapRun(
    # The capture defined in step 3
    capture=capture,
    # set the number of actors in the capture
    number_of_actors=2,
    # This mapping tells compute which actors are where on each video input
    # The actors here must have a skeleton. If the actors don't have skeletons
    # then you cannot provide actor mapping and have to use `number_of_actors` only
    actor_mapping=[
        ActorMapping(
            actor=actor_1,
            # Where the actor is in the frame
            bounding_box=(0, 100, 0, 0),
            video_capture_input=video_capture_input,
        ),
        ActorMapping(
            actor=actor_2,
            # Where the actor is in the frame
            bounding_box=(0, 0, 0, 500),
            video_capture_input=video_capture_input,
        ),
    ],
    volume_service=Service.s3,
    volume_uri_prefix="s3://my-bucket/path/to/directory",
    # Where outputs should be saved. There are many types of output
    # resource
    outputs=[
            Resource(
            resource_type=ResourceType.main_fbx,
            uri="s3://my-bucket/path/to/mocap/scene.fbx",
            service=Service.s3,
        ),
        Resource(
            resource_type=ResourceType.all_fbx,
            uri="s3://my-bucket/path/to/mocap/all_actors.zip",
            service=Service.s3,
        )
    ],
)

Provision the engine

Then provision the engine

from moveai_mocap.compute.ec2 import EC2ComputeManager
from moveai_mocap.models.notifiers.sns import SNSNotifier

sns_notifier = SNSNotifier(
    endpoint="arn:aws:sns:",
    payload={"foo": "bar"},
)

compute = EC2ComputeManager(
    # configure these settings according to your AWS account
    iam_instance_profile_name="instance-profile-name",
    ami_id="ami-12345678",
    subnet_ids=["subnet-1"],
    security_group_ids=["sg-1"],
    notifier=sns_notifier,
)
# Calling provision will launch an EC2 instance and generate the output resources
# defined on `outputs` on the `mocap_run`
compute.provision(mocap_run)