Working with Point Clouds - Creating a Custom Mask

When processing a dataset with 360 video for image extraction or colorization, one of the essential steps is applying a mask to the extracted frames. This is because there may be areas in the frame that you do not want to show. Emesent Aura comes with several pre-defined masks that you can use for this purpose but you can also create your custom mask in case none are suitable for your dataset.

To create a custom mask

The following process is demonstrated using GIMP (a freely downloadable editing software). However, you can utilize any third-party image editing software, as the techniques discussed here are applicable across various platforms.

Run an Extract 360 images workflow with Image masking turned off.Note: If you have a large video, it is recommended to do a subsample first by setting a high Frame interval (e.g., 250).
Once the frame extraction process is completed, navigate to the extracted frames folder.
Select a frame you want to create a mask for and open it in GIMP.
Adjust the display to ensure the image fits your screen properly.
Start with an existing mask instead of creating one from scratch. Open the folder containing the predefined masks by navigating to Program Files > Aura > Aura> Plugins > EmtProcessWorkflows > Content > ProcessWorkflows > ImageMasks.
Select a suitable mask from either the Colourise or FrameExtract folder then drag this mask onto your current image in GIMP. It will appear as a new layer.
Use the Fuzzy Select Tool (aka Magic Wand) and click on the black area of the mask layer to select it.
In the Layer panel on the right, every layer in the image appears as a thumbnail. The upper layer in the list is the first one visible. Click the Eye icon before the mask layer to hide the mask.
Notice that the selected mask area is visible on top of the image.
Use the Free Select Tool to add or refine areas to the mask as needed. Press the Enter key to add the area to the current selection.
Note: Make sure the Mode is set to Add to the current selection. You can do this by clicking on the second mode icon or pressing the Shift key while selecting.
Keep the frames folder handy and add more frames if necessary. Review each added frame, adjusting the mask to ensure proper coverage, especially around challenging areas like hands or cables.
Ensure that the active background color is set to black.
Once satisfied with the mask, go to the Layer panel and select the base layer containing the extracted image.
Press Delete on your keyboard.
Go to Select > Invert then press Delete again. The entire image is now black.
Click the Bucket Fill Tool. Ensure the active foreground color is white.
Click the top area to fill it with the active foreground color (white).
Go to File > Export As. Save the mask in PNG format to avoid the lossy compression issues associated with JPG files.
Click Export.
Note: If using Emesent Aura 1.5 or earlier, change the pixel format to 8bpc RGB.
Close GIMP and return to the Emesent Aura.

To add the custom mask in Aura

Run a Colorize or Extract 360 images workflow again.
Click Processing Settings.
In the Colorize or Extract 360 images tab, enable Image masking.
In Mask template, click the + icon.
Enter a name for the new mask, click Create then browse for the newly created mask.
Click Save to finish adding the custom mask to Emesent Aura.