Segment by Example

1. Overview

Segment by Example (SBE) is Aivia’s smart image segmentation tool that eliminates the guess work from image segmentation. Instead of struggling with complex parameter tuning, just show SBE what you want to detect through simple annotations. SBE then guides AI models to deliver accurate segmentation result for your specific task.

2. Interface

When you select Segment by Example, the tool selects the most recently used segmenter set or starts a new segmenter sets, if you have an image currently in the viewer.

The main Segment by Example GUI is shown below. There are 7 sections in the interface:

• The SBE Toolbar along the top of the GUI allows you to perform file actions related to the Segment by Example tool.

• The Tutorial section displays helpful tips to get you started.

• The Common Parameters section allows you to configure share detection options across segmentation types.

• The Types section lets you add/remove/rename a type in a segmenter set.

  • The Compartments section lets you select the output compartment and its input channels and the color of the output compartment.

  • The Parameters section for each compartment lets you adjust the estimated parameters after they’ve been calculated from your annotations. You can also disable Select for Output here if you don’t want that compartment included in the final output.

• The Drawing section lets you select a tool for defining training regions in the Image Panel.

• The Preview section lets you add and display Regions of Interest (ROIs) for previewing. You can also choose to use the preview ROI as the final segmentation.

• The Apply Controls along the bottom of the GUI allow you to run a segmenter set, choose the output set, or output each type as a separate subset

2.1 SBE Toolbar

The SBE Toolbar is located at the top of the SBE tab. The toolbar lets you perform file actions related to the SBE, such as creating a new segmenter set; saving a segmenter set and/or annotations; and loading segmenter files. 

Since Aivia 16, the concept of segmenter set was introduced to support multiple segmenters, replacing the previous single-segmenter approach.

The function for each item in the toolbar is summarized in the table below.

Function	Icon	Description

Function	Icon	Description
Select Segmenter Set	Open Editing Mode Open Applying Mode	Shows the name and type (2D or 3D) of the currently selected segmenter; click on the item to expand a dropdown menu from which you may select a loaded/created segmenter for estimation and/or applying When a pencil icon appears, it means the segmenter set is in editing mode. When a lock icon appears, it means the segmenter set is in applying mode, where users cannot add types, train segmenters, or modify the segmenter set except for applying it.
Create New	Open	Allows you to create a new, blank segmenter set for estimation
Delete	Open	Removes the current segmenter set from Aivia; this does not delete the segmenter set from its file if the segmenter set was loaded from a file
Remove All Segmenters	Open	Removes all currently loaded segmenter sets
Load	Open	Allows you to load a saved off segmenter set (.segmenter), compartment annotations(.annotations) or training set (.sbetraining) from a file. See https://aivia-software.atlassian.net/wiki/spaces/AW/pages/2576252929/Segment+by+Example#3.3-Saving-and-Loading.
Save	Open	Allows you to save the current segmenter set, segmenter training set, or annotations to a file. See https://aivia-software.atlassian.net/wiki/spaces/AW/pages/2576252929/Segment+by+Example#3.3-Saving-and-Loading.

2.2 Tutorial

The tutorial section contains example images to help you get acquainted with how to use the the SBE tool. Starting from left to right, it gives examples of what steps to follow to successfully create a segmenter set.

2.3 Common Parameters

• Detection method:

  • Standard: Applying original Cellpose based cell segmentation method. For cell detection, it applies the Cellpose Cyto2 pre-trained model; for nucleus detection, SBE uses the Cellpose Nuclei pre-trained model.

  • Fast (2D only): This approach leverages a high-speed, U-Net–based deep learning model developed by Leica for accurate cell and nucleus segmentation. It is optimized for large-scale 2D datasets, such as multi-well assay applications, and delivers faster performance than the standard Cellpose pre-trained models.

  • Flow-based (3D only): This method applies a pre-trained Cellpose model to generate vertical and horizontal flow fields, which are then used to create confidence maps, perform 2D cell or nucleus segmentation, and reconstruct cells or nuclei in 3D with improved Z-axis separation compared to the standard method.

• Remove objects touching boundary:

  • If a detected object touches the image or ROI boundary, it will be excluded from the output. This option is enabled by default.

2.4 Segmentation Types

Add/Rename/Remove Segmentation Type. Starting with Aivia 16, each segmenter is referred to as a segmentation type (or simply type), which represents different cell types. Within each type, SBE can detect various compartments, such as cytoplasm, nucleus, or vesicles. This feature is particularly useful for assay applications. For example, in a Live/Dead assay, the user can define two distinct types: one for live cells and another for dead cells.

Under the Types section, there are two sub-sections: Compartments and Parameters

2.4.1 Compartments

In the compartments section, you’ll see a diagram showing compartments: Cell, Nucleus, and Vesicles. This section becomes available when both an active image and an active segmenter set are present.

Clicking on a compartment highlights it, indicating that it has been selected for annotation. For example, in the illustration on the right, the Nucleus compartment is highlighted, and the field below the diagram also displays “Nucleus,” confirming it as the segmentation target for SBE. On the right side of this field, a dropdown menu allows the user to select input channels. After choosing the channels, users can begin drawing examples for SBE to use in parameter estimation.

2.4.2 Parameters

The Parameters section allows you to review and adjust the values estimated by SBE based on your annotations. These parameters are then applied during segmentation. Any changes to the annotations will automatically trigger a new estimation process, and the updated values will replace the previously estimated or user-defined parameters in this section.

By default, this section is collapsed, as users typically do not need to manually adjust these settings.

• Select for Output: User can decide whether to generate output for this specific compartment in this Segmenter Type.

• Diameter: The average diameter of the object.

• Min/Max Diameter: The range of the object diameters.

• Probability Threshold: The minimum confidence threshold for determining whether a pixel belongs to an object. Lower values result in more detected objects and include more pixels in segmentation masks, while higher values result in fewer detected objects.

2.5 Drawing

The SBE uses annotations to estimate the parameters required for the segmentation. The drawing section provides a variety of tools for marking objects in the image. For best results, annotate at least 3 diverse objects in your image, including some of the smallest and biggest. All annotations must be created in the Main View (2D).

2.5.1 Drawing tools

The function and description for each of the drawing tools are summarized in the table below. For advanced drawing modes and tips, please refer to the appendix.

Drawing mode or Function	Icon	Description

Drawing mode or Function	Icon	Description
Paint	Open	Paints a circle under the cursor; the brush size is adjustable
Erase	Open	Erases a circle under the cursor; the eraser size is adjustable
Flood	Open	Fills in enclosed regions
Region Drawing	Open	Draws a freehand drawing that automatically closes
Auto Draw	Open	Predicts the contour on the next z-plane based on the previous drawing; see the "Auto Draw mode" section for more details
Magic Wand	Open	Uses flood-filling to paint a region that has similar intensity and is connected to the selected location; see the "Magic Wand" section for more details
Size Selector	Open	Adjusts the size of the brush or eraser. Equivalent to ctrl + shift + mouse scroll
Jump to Previous Teaching Frame	Open	Jumps to the previous frame with an annotation drawn on it, moving back through Z and then through time
Jump to Next Teaching Frame	Open	Jumps to the next frame with an annotation drawn on it, moving forward through Z and then through time
Automatically Move to Next Frame (only available for 3D images)	Open	Sets the mode for moving between z-frames after a drawing is completed; the modes are as follows: Stay Still: The viewer does not change the z-frame after a drawing is completed. This is the default mode. Up: After a drawing is completed, the viewer moves up in the z-stack. Down: After a drawing is completed, the viewer moves down in the z-stack. Auto: After a drawing is completed, the viewer moves in the last z-direction the user cycled through.
View Training Image		This button will let you go back to the training FOV when you train a segmenter set using multi-well experiment files

2.6 Preview

Click on the Preview icon (see above) to initiate preview generation; while previewing is toggled on, there is live previewing that updates as changes to the drawings/annotations are made. All output types may be previewed in Main View (2D).

Function	Icon	Description

Function	Icon	Description
Toggle Preview	Open	Toggles the preview on/off
Select ROI for Preview	Open	A dropdown menu lists all available ROIs, allowing the user to select one as the preview target
Toggle ROI	Open	Toggles the ROI on/off
Delete ROI	Open	Deletes the current selected preview ROI
Use for Final Segmentation		Toggles the selected preview ROIs to use as the ROI to use when running the full segmentation

2.7 Apply Controls

The Apply Controls, located at the bottom of the SBE, allow you to run segmentation and configure the output set. Before running segmentation, you must perform parameter estimation at least once.

3. How to Use SBE

3.1 Single/Multi Compartment Segmentation

1. Create a New Segmenter Set
   Click the create new icon in the SBE Toolbar. This will create a new segmenter set along with a blank segmentation type.

2. Select Compartment
   Click the compartment to detect in the cartoon diagram

3. Choose Input Channels
   Select the input channel(s) that represent the chosen compartment

4. Draw teaching examples
   Use any tools in the Drawing section to paint desired features of interest. At least 3 examples are required to estimate, preview, and run the segmenter.

5. Preview Results
   A preview ROI appears by default after selecting the compartment and input channels.

6. Refine with More Drawings (Optional)
   If the preview results are not ideal, you can draw more examples to improve the estimation quality

7. Annotate another compartment (Optional, Multicompartment detection only)
   Repeat 2. - 6. for another compartment. For vesicle compartments, the user can add up to 50 different kinds of vesicles.

8. Run Segmentation
   Apply segmentation to detect and create an output set on your image

3.2 Multi-Type Segmentation

A segmenter set in Aivia allows you to have multiple segmentation type. If you have different cell types, you can create multiple segmentation types to keep things organized and easier for downstream analysis. To do so, simply click Add a Segmentation Type icon in the Types section and train the second segmentation type.

Please ensure that every type has identical compartment selections before running segmentation; otherwise, you cannot apply segmentation, and a warning will appear.

See also https://aivia-software.atlassian.net/wiki/spaces/AW/pages/2980577283 wiki for more Multi-Type Segmentation Examples.

3.3 Saving and Loading

You can save three types of files in SBE: Segmenter Set, Annotation, and Training Set. The latter two can be reloaded into Aivia for estimation in a future session, while Segmenter Sets allow you to reuse a segmenter on the same or similar images.

3.3.1 Segmenter Set (.segmenter)

Saving a Segmenter Set

1. Make sure your taught segmenter set needs no further tuning.

2. Click the downward arrow next to the Save icon in the SBE toolbar.

3. Select Save Segmenter.

4. Choose the file name and location for the .segmenter file.

5. Click Save.

Loading a Segmenter Set

1. Click the Load Files icon in the SBE toolbar.

2. Navigate to the .segmenter file and click Open,
   or drag and drop the .segmenter file onto the Segment by Example interface.

3.3.2 Compartment Annotations (.annotations)

Annotation files contain the drawn teaching regions for a compartment.

Saving Annotations

1. click on the downward-pointing triangle next to the Save icon in the SBE Toolbar

2. Select Save Compartment Annotations

3. In the dialog, choose the file name and location for the .annotations file.

Loading Annotations

1. Make sure the corresponding image, compartment, and segmentation type are already open. Aivia assumes these are ready before loading annotations.

2. Click the Load Files icon in the SBE toolbar.

3. Navigate to the .annotations file and click Open.

4. Confirm loading into Current Type Name and Current Compartment Name when prompted.

5. The previously drawn regions will appear and can be edited, previewed, and used for estimation or running segmenters.

3.3.3 Training Set (.sbetraining)

A training set file includes image path, annotations, segmenter set and its settings

Saving a Training Set

1. Click the downward arrow next to the Save icon in the SBE toolbar.

2. Select Save Training Set.

3. In the dialog, choose the file name and location for the .sbetraining file.

4. Click Save.

Loading a Training Set

1. Click the Load Files icon in the SBE toolbar.

2. Navigate to the .sbetraining file and click Open,
   or drag and drop the file onto the Segment by Example interface.

3. Aivia will load the image from the training set into the workspace, along with the previously drawn annotations and segmenter settings.

4. You can now continue add annotations, refining, or using your segmenter set.

A. Warnings Messages

When a warning icon appears in SBE, click the icon to read the message. The table below explains each warning and how to address it.