Aivia Software

Pixel Classifier

The Pixel Classifier is an image enhancement tool that enhances specific features on an image based on user-input teaching patterns. A number of image transformations are applied to the user-defined teaching regions (and their respective adjacent pixels) internally to identify unique features (such as edges or speckles) for classification.

Interface

The Pixel Classifier is a tab in the Analysis Panel, which is pinned and found on the right side of the Aivia window by default. When you launch the Pixel Classifier, the tool selects the most recently used classifier or starts a new classifier with two (2) predefined classes to help you get started right away.

The main Pixel Classifier GUI is shown below. There are five (5) sections in the Pixel Classifier interface:

  • The Classifier Toolbar along the top of the GUI allows you to perform file actions related to the Pixel Classifier.

  • The Input & Training section allows you to specify the image channel(s) to use for training and applying the classifier.

  • The Drawing section lets you select a tool for defining training regions in the Image Panel.

  • The Classifier section lets you select the pixel class to teach as well as specify output options.

  • The Apply Controls along the bottom of the GUI allow you to preview, teach, or apply a classifier and specify regions of interest (ROIs) for previewing and applying.

Pixel Classifier GUI

Classifier Toolbar

The Classifier Toolbar is located at the top of the Pixel Classifier tab. The toolbar lets you perform file actions related to the Pixel Classifier, such as creating a new classifier; saving a classifier, training set, or annotations; and loading classifier files. 

Classifier Toolbar

The function for each item in the toolbar is summarized in the table below.

Function

Icon

Description

Function

Icon

Description

Select Classifier



Shows the name and type (2D or 3D) of the currently selected classifier; click on the item to expand a dropdown menu from which you may select a loaded/created pixel classifier for training and/or applying

Create New

Allows you to create a new, blank pixel classifier for teaching; click on the downward-pointing triangle to expand a menu where you may specify whether to create a 2D or 3D classifier

Delete

Removes the current pixel classifier from Aivia; this does not delete the pixel classifier from its file if the classifier was loaded from a file

Remove All Classifiers

Removes all currently loaded classifiers

Load

Allows you to load a pre-trained pixel classifier (.pxclassifier), training set (.pxtraining), or set of annotations (.pxannotations) from a file

Save

Allows you to save the current pixel classifier, training set, or annotations to a file

 

Input & Training

The Input & Training section lets you select the image channel(s) to use for training when creating a new pixel classifier or to specify the image channel(s) to use as input for applying a previously trained and finalized (saved as a .pxclassifier file) pixel classifier. During training, the teaching images are automatically set as the input images. 

You may specify the number of channels to use as inputs for the classifier as well as the names of the channel inputs in the Input Definitions column. The table below describes the options available in the Input Definitions column.

Function

Icon

Description

Edit Input Name

Enables editing of the classifier input name; you can also double-click on an input name to enable editing of the name

Add Input Channel

Adds a channel input

Remove Input

Removes the corresponding classifier input

In the Training Channels column, you may use the dropdown menu(s) to select which channel(s) on the current image to use as which input(s) for training the pixel classifier.

The Manage Training Set button opens a window with options for saving the annotations for the individual images in the training set as well as removing images from the training set. See the "Train a pixel classifier over multiple sessions" section of this page for more details about saving training data.

For teaching on colocalized regions (regions with overlapping signals on multiple channels), make sure to include all the colocalized channels as training channels.

Only the Input section is displayed when you are applying a previously trained and finalized pixel classifier to an image. You may use the dropdown menu(s) to select which image channel(s) on the current image to provide as which input(s) to the classifier.

 

Input & Training section during training




Input section for applying a previously trained classifier



Drawing

When teaching a pixel classifier, use the variety of drawing tools to draw some examples of your classes. All drawing, except with the Magic Wand, must be done in Main View (2D).

 

Drawing tools

 

The function and description for each of the drawing tools are summarized in the table below.

Function

Icon

Description

Function

Icon

Description

Paint 

Paints a circle under the cursor for the currently selected class; the brush size is adjustable

Erase

Erases a circle under the cursor for every class; the eraser size is adjustable

Flood

Fills in enclosed regions with the selected class; the enclosure can be made up of any class(es)

Region Drawing

Draws a freehand drawing that automatically closes

Auto Draw 

Predicts the contour on the next z-plane based on previous drawing; see the "Auto Draw mode" section for more details

Magic Wand

Uses flood-filling to paint a region that has similar intensity and is connected to the selected location; in 3D, the Magic Wand paints regions for two (2) classes, one (1) for the selected class and one (1) for the background class or other class (see the "Magic Wand" section)

Size Selector

Adjusts the size of the brush or eraser

Jump to Previous Teaching Frame

Jumps to the previous frame with a teaching region drawn on it, moving back through Z and then through time

Jump to Next Teaching Frame

Jumps to the next frame with a teaching region drawn on it, moving forward through Z and then through time

Automatically Move to Next Frame

(only available for 3D images)

Sets the mode for moving between z-frames after a drawing is completed; the modes are as follows:

  • Stay Still: The viewer does not change the z-frame after a drawing is completed. This is the default mode.

  • Up: After a drawing is completed, the viewer moves up in the z-stack.

  • Down: After a drawing is completed, the viewer moves down in the z-stack.

  • Auto: After a drawing is completed, the viewer moves in the last z-direction the user cycled through.

Auto Draw mode

Auto Draw options

When Auto Draw mode is selected, a set of options are shown underneath the row of drawing tools (see above). First, select the channel to use for prediction in the dropdown menu on the left, which shows "GFP" in the example above. Click and drag the mouse to draw the outline of a region on the image; when the mouse button is released, the ends of the drawn line are automatically connected to create a closed loop. Use the Interpolate Down button and Interpolate Up button (or hotkeys S and D) to interpolate the next drawing down or up in Z respectively. The Step to Bottom button and Step to Top button (or hotkeys A and F) can be used to visit the bottom and top drawings respectively. Click on the Clear icon to remove all drawing from the current z-slice or on the Reset icon to clear all drawing. Click on Finish Drawing to assign the drawn regions (enclosed by the blue outlines) on all slices to the selected class.

Magic Wand

Magic Wand options

 

Options for the Magic Wand tool are provided below the row of drawing tools when the Magic Wand is selected (see above). The Magic Wand may be used for annotation in 2D or 3D viewing modes. In 2D or 3D, the first step is to select the image channel to use from the dropdown menu, which shows "GFP" in the image above.

In Main View (2D), expand the Thresholding Mode dropdown menu to choose one of the following modes:

  1. Bidirectional mode accepts connected pixels within the sensitivity limit above and below the intensity at the selected location.

  2. Peak Finding mode accepts connected pixels within the sensitivity limit above the intensity at the selected location.

  3. Valley Finding mode accepts connected pixels within the sensitivity limit below the intensity at the selected location.

Select a location on the image for the Magic Wand to search from. A preview of the region to be painted is shown in blue (see right). Adjust the Sensitivity slider (or use hotkeys W and R) to change the tolerance for intensity differences. The narrowest range of accepted intensity levels is used when the slider is all the way to the left. Adjust the Search Range slider (or use Ctrl + Shift + Mouse Scroll over the Image Panel) to increase or decrease the area over which the Magic Wand will search. No pixels outside of the search region are included in the painted region. The search range is indicated by the green box around the cursor when the Magic Wand is used (see right). Check the Fill Multiple Z Frames box to extend the user-defined search range limits in the z-direction; when this box is not checked, the Magic Wand only searches on the current z-frame. Finally, click on Paint (or press E) to paint the region and assign it to the selected class.

When the Fill Multiple Z Frames option is used, the painted regions may not exactly match the blue preview.

With the Magic Wand in 3D View, click on the image to show a surface preview. You may adjust the search range using either the Search Range slider or the Ctrl + Shift + Mouse Scroll hotkey combination. To adjust the range of intensities included in the region, adjust the LUT of the channel being used for the Magic Wand and click again on the image. Regions do not have to be connected to be painted and assigned to the selected class when the Magic Wand is used in 3D. The Magic Wand paints to two (2) classes: one (1) is the selected class, and the other is what is presumed to be the background class. This means that the total search volume is painted. When you click on Paint, a yellow frame around the painted region appears in 3D View. Return to Main View (2D) to closely examine the resulting annotations.



Selected location (small green square), search limits (large green square), and region preview (blue) for the Magic Wand

Hotkeys for drawing

Drawing mode

Hotkey

Description

Drawing mode

Hotkey

Description

Paint

Ctrl

Switches to erasing while held down

Ctrl + Shift + Mouse Scroll

Adjusts the brush size when done over the Image Panel

Erase

Ctrl

Switches to painting while held down

Ctrl + Shift + Mouse Scroll

Adjusts the eraser size when done over the Image Panel

Auto Draw

A

Goes to the bottom drawing

S

Predicts the next drawing down

D

Predicts the next drawing up

F

Goes to the top drawing

E

Clears all drawing

Magic Wand

W

Decreases the sensitivity

R

Increases the sensitivity

E

Paints the selected region

Ctrl + Shift + Mouse Scroll

Adjusts the search range when done over the Image Panel

 

Classifier section

The Classifier section is the primary space for interacting with individual pixel classes in the Pixel Classifier. To begin training a pixel class, select the class's row prior to drawing the regions in the Image Viewer. The Classifier section also lets you specify the output type and options for each pixel class and add additional training features.

To edit the name of the classifier, double-click on the classifier name above the "[s]elect pixel class" text and then edit the text.

The function and description for each item in the Classifier section are summarized in the table below.

Classifier section

Function

Icon

Description

Function

Icon

Description

Expand/Collapse Output Parameters

Expands or collapses the output parameter section for each class

Advanced Options

Lets you select different features and kernel sizes to use for the pixel classifier

Hide/Show Class Annotations

Hides/Shows the training regions on the image

Select Output Type

Lets you select from the following four (4) output types for each pixel class:

  • Confidence: Output the confidence image, which contains the determined likelihood from 0 (least likely) to 255 (most likely) at each pixel location that the pixel belongs to the associated class.

  • Masked Channel: Threshold the confidence image and keep the portions of the selected input channel(s) whose confidence is over the threshold.

  • Segmentation: Threshold the confidence image and generates objects (outlines for 2D, meshes for 3D).

  • Smart Segmentation: Take the training regions as input to determine the best parameters for object detection, size, and partitioning, and then use the parameters to generate objects (outlines for 2D, meshes for 3D) using a custom object detection algorithm.

Class Color

Specifies the color of the drawn training region and, if Confidence is selected as the output, the channel color for the preview and output of the given class

Select Output

Opens the options for enabling/disabling output(s); you can disable the output for a given class by unchecking each available output; if Masked Channel is selected as the output, you may have the option to select multiple channels to mask out.

Remove Class

Removes the class from the pixel classifier

Add Class

Adds a new class to the pixel classifier

Output parameters

Depending on the output type chosen, there are additional parameters that allow you to fine-tune the final output. To expand the parameters, click on the caret to the left of the Hide/Show Class Annotations button. The parameters are summarized in the table below by output type.

Output type

Parameter name

Description

Output type

Parameter name

Description

Masked Channel

Detection

Specifies the confidence threshold for the masked region; moving the slider to the right lowers the minimum confidence threshold and results in more pixels being included in the output

Remove Small Objects

Removes small objects that are likely the result of noise

Segmentation

Detection

Specifies the confidence threshold for the segmented objects; moving the slider to the right lowers the minimum confidence threshold and results in detection of more low-confidence objects as well as larger and more connected objects

Remove Small Objects

Removes small objects that are likely the result of noise

Smart Segmentation

Detection

Specifies the confidence threshold for the segmented objects; moving the slider to the right lowers the minimum confidence threshold and results in detection of more low-confidence objects as well as larger and more connected objects

Partition

Specifies the aggressiveness of the partitioning algorithm applied to the segmented regions; moving the slider to the right increases the aggressiveness of partitioning and results in more partitioned objects

Object Size

Specifies the object size range based on the user-drawn regions; use the lower threshold (left slider control) to modify the minimum size and the upper threshold (right slider control) to modify the maximum size

 

Advanced Options

The Advanced Options menu lets the user specify the image features to use for creating the pixel classifier. Click on a feature name in the table to add the feature to the list; features with the GPU-Computed symbol (see below) can be processed on a GPU. 

 

GPU-Computed symbol

 

Additionally, Aivia provides three (3) preset groups of features:

  • Clear deselects all features; you will not be able to teach the Pixel Classifier until at least one (1) feature is selected.

  • Standard includes the default features that are used in the Pixel Classifier.

  • Line includes specialized features designed to highlight fine, line-like features.

The Kernel Size option lets you choose Small (micro), Medium, and/or Large (macro) kernels for applying the image feature transformations. Smaller kernels are more attuned to highly localized features, while larger kernels are more accurate for detecting major image features.



Advanced Options menu



Apply Controls

The Apply Controls are located at the bottom of the Pixel Classifier tab and include options for previewing, teaching, and applying as well as specifying ROIs. Prior to applying a pixel classifier, the user must click the Teach button to train the pixel classifier.

Preview

Preview icon highlighted in the Apply Controls section

 

Click on the Preview icon (see above) to initiate preview generation; while previewing is toggled on, there is live previewing that updates as changes to the drawings/annotations are made. All output types may be previewed in Main View (2D), and Segmentation and Smart Segmentation outputs can be previewed in 3D View.

Click on the downward-pointing triangle next to the Preview icon to reveal the Speed/Accuracy slider (see below), which lets you control the quality of the preview. Moving the slider toward Speed leads to faster preview generation (at the expense of accuracy), while moving the slider toward Accuracy leads to the generation of previews that more closely resemble the output (but take longer to generate). If any ROIs are selected, previews are only generated in those ROIs.

Speed/Accuracy slider for previews

 

Specify regions of interest

Click on the ROI Processing Options icon to open the Regions Of Interest Table (see right). When the ROI Processing Options icon appears as nested squares, a ROI is selected. Check the box in an ROI's row of the Regions Of Interest Table to select it for previewing and/or applying. Use the topmost checkbox, which is in the header row, to (de)select all ROIs. See the Regions of Interest page for more information about using ROIs.

Teach

The Teach button creates a pixel classifier using the drawn training regions for all training images and the feature calculation from those regions. An asterisk to the left of the Teach button indicates that changes have been made to the pixel classifier since it was last taught. At least two (2) classes must have training regions and at least one (1) feature must be selected before a pixel classifier can be taught.

Apply

The Apply button applies the taught pixel classifier to the selected ROIs or, if no ROIs are selected, the whole image. You can choose to apply to the whole image sequence (over all of T and Z), just the current t-frame, or just the current t- and z-frame (except if you are creating 3D objects).

The downward-pointing triangle to the right of the Apply button provides additional options for generating pixel classifier output. Select one (1) of the output options from the dropdown menu to apply the pixel classifier. There are four (4) apply options:

  • Apply applies the pixel classifier to the entire image sequence (over all of T and Z).

  • Apply to the current time frame applies the pixel classifier to the current t-frame.

  • Apply to the current time and depth frame applies the pixel classifier to the currently displayed t- and z-frame only.

  • Batch apply the classifier opens a window where you can specify a collection of images to apply the pixel classifier to as well as an output folder for the results.

ROI Processing Options icon (bottom right) and Regions Of Interest (ROI) Table

General usage of the Pixel Classifier

Create a new pixel classifier

To create a new pixel classifier, click on the Create New icon in the Classifier Toolbar. A blank pixel classifier will then be created with two (2) pre-configured classes: "Want" and "Background." The general workflow for training a new pixel classifier is as follows:

1. Draw teaching regions using any tools in the Drawing section to paint desired features of interest. Select a different class from the class table to draw teaching regions for that class.

2. Preview the results. Use ROIs for faster previews.

3. Add more teaching regions to the image, following the same instructions as are in Step 1; keep previewing on to determine the quality of the teaching as you add more teaching regions.

4. Teach the pixel classifier by pressing the Teach button.

5. Select the output types and destinations using the corresponding menus in each class's row of the class table. 

6. Apply the pixel classifier to the image.

 

Train a pixel classifier over multiple sessions

There are two (2) types of files that you can save and load back into Aivia for training at a later date: annotation files (.pxannotations) and training set files (.pxtraining). 

Annotation files

Annotation files contain the drawn teaching regions for an image. To save annotations for the displayed image and current classifier, click on the downward-pointing triangle next to the Save icon in the Classifier Toolbar and then select Save Annotations for the current image (see right); this opens a window where you may specify the location and name for the .pxannotations file. Another option is to click on the Manage Training Set button and then on the Save icon for the image in the training set whose annotations you would like to save; note that this method allows you to save annotations for an image in the training set that is not currently displayed in the Image Viewer.

It is assumed that when annotations are loaded, the image that they are from and the classifier that was being trained (with the same number of classes, class names, etc.) are already open and displayed. To load annotations, click on the Load icon in the Classifier Toolbar, navigate to the .pxannotations file, and then click on Open. The previously drawn teaching regions are then displayed and may be edited and used for previewing, training, and applying pixel classifiers.

Training set files

Training set files include the following:

  • Paths to all images in the training set

  • Drawn teaching regions (annotations) for all images in the training set

  • Classifier settings

  • The taught classifier, if the training set was saved with a taught pixel classifier

To save a training set file, click on the downward-pointing triangle next to the Save icon in the Classifier Toolbar and select Save Training Set; this opens a window where you may specify the location and name for the .pxtraining file. Another option is to not only save the .pxtraining file but also copy the images in the training set to another location. To do this, click on the downward-pointing triangle next to the Save icon in the Classifier Toolbar, select the Copy all the training fields and training data to one spot option, navigate to the folder where you would like to copy the training images and .pxtraining file to, and then click on Select Folder.

To load a training set, click on the Load icon in the Classifier Toolbar, navigate to the .pxtraining file, and then click on Open or drag and drop the .pxtraining file onto the Pixel Classifier interface; this will load all of the images in the training set into the workspace as well as load the previously drawn training regions and classifier settings for the in-progress pixel classifier. You may then continue drawing teaching regions, adding images to the training set, and otherwise building upon and/or using your pixel classifier.



Pixel Classifier saving options

Save a pixel classifier

When you have a taught classifier that needs no further tuning, you may save the classifier as a .pxclassifier file; to do so, click on the downward-pointing triangle next to the Save icon in the Classifier Toolbar, select Save Classifier, choose the name and location for the file, and then click on Save. You may then load the classifier by clicking on the Load icon in the Classifier Toolbar, navigating to the .pxclassifier file, and then clicking on Open or by dragging and dropping the .pxclassifier onto the Pixel Classifier interface.

Technical details

The Pixel Classifier consists of two (2) fundamental steps:

  1. Feature Calculation: Image features (such as edges) are calculated for every pixel of the input image to generate a feature vector.

  2. Classification Application: The feature vector is used to train a classifier to classify unknown feature vectors into one (1) of the classes specified by the user.

Each image feature is derived from applying one (1) of several processing filters on the image over several scales, or kernel sizes, to identify both regional (macro) and local (micro) features inherent in the image. A random forest algorithm is used for creating the pixel classifier from the calculated features.

The classification result is output as new image channels that map the probability that a given pixel belongs to a specific class from 0 (high likelihood of not belonging) to 255 (high likelihood of belonging). The probability map channel can then be used for additional analysis.

Related articles