Review Labeled Images and Send Feedback
As a review task owner, you are responsible for completing the assigned review task, sending the correctly labeled images to the project owner, and sending feedback on the incomplete images to the labeling task owner. For more information on the tasks performed by each role, see the Using This Example section of Get Started with the Image Labeler.
The Image Labeler app enables you to review a set of images as a review task owner. The project owner assigns the review task to you as part of a distributed labeling project with multiple team members. For more information about how to set up a team labeling project as a project owner, see Create Project for a Labeling Team and Publish Tasks and Create Review Tasks for Labeled Images.
This example details the steps a review task owner must follow to:
Open a project shared by the project owner.
Load the assigned review task.
Send the review feedback.
Open Labeling Project and Load Assigned Review Task
A review task owner with access to a full MATLAB® license, can open a labeling project and review images following these instructions.
Open the Image Labeler app using one of these options:
MATLAB® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.
In the MATLAB command prompt, enter
To install the Image Labeler app and access the shared labeling project, a task owner without access to a full MATLAB license must use a compiled executable file shared by the project owner. For more information on the executable file created by the project owner, see Create Executable for Review Task Owners Without Full MATLAB License.
The labeling tasks, along with labeling instructions have been predefined. These are part of the team project created and shared with you by the project owner.
To open an existing team project, in the Image Labeler app interface,click Open Project.
In the file explorer, navigate to the project folder location shared with you. Select the
.prjfile, and click Open.
Load Assigned Review Task
After you select and open the team labeling project, the app provides the option to open the project in Project Owner or Task Owner mode.
To access the review task assigned to you as a review task owner, follow these steps:
Select the Task Owner mode. The app then displays an interface with a Task Owner Name list and a Task List that contains the Task Name and Task Type for each task assigned to the selected task owner.
From the Task Owner Name list, select
Team Member E. The Task List shows Task 5 in the Task Name column and
Reviewin the Task Type column.
Task 5and click Open. The Image Labeler app opens the review
Task 5in the task owner mode.
Image Labeler Layout in Task Owner mode
The ROI Label Definitions, Scene Label Definitions, and Task instructions panes contain the label definitions for all the images in the task that you must review for the current project.
Each ROI label definition is associated with a specific color listed in the ROI Label Definitions pane. The colors of labels on the images correspond to these label definitions.
The Review Labels, Sublabels and Attributes pane lists options for reviewing the labels. You can accept or reject individual labels, and provide feedback, in this pane.
The Image Browser pane displays thumbnails of the images in
the review task. In this example, the Image Browser shows the
50 images assigned to
Team Member E for the review
Review Labeled Images
To complete the review process for the task, you must accept or reject the label definitions for all images in the task. You can also provide a few lines of feedback for each reviewed image. When you reject the labels for an image, this enables you to specify concrete actions for the labelers to correct the labels.
Accept Correctly Labeled Images
Consider the following image for review. Note that the ROI label definition
VideoCamera is labeled correctly on the image. The scene
BustPresent is correctly marked as
No, as there is no bust present in the picture. There is no
lamp in the picture, and thus the ROI label definition
correctly absent in the image. Thus, you can accept the labels on this image. You
can do so in the Review Labels, Sublabels, and Attributes pane
using these steps:
Because all the labels are marked correctly, you can click Accept All.
Alternatively, you can accept the labels individually by selecting Accept next to the each label.
Reject Incorrectly Labeled Images
Consider the following image for review. Note that the labeler has used the
incorrect ROI label definition of
Lamp to label the video camera
in the image. Furthermore, the label does not fully cover the video camera in the
image. The scene label definition
BustPresent is correctly marked
No, as there is no bust present in the picture. Thus, you can
use the Review Labels, Sublabels, and Attributes pane to accept
the scene label and reject the ROI label, with feedback for correction, using these steps:
Accept the scene label by selecting Accept next to the
Reject the ROI label by selecting Reject next to the
Alternatively, instead of rejecting an incorrect ROI label, you can delete or modify it and accept it as a reviewer. You can also view or hide each label and sublabel in a labeled image by clicking the icon next to the name of that label or sublabel in the ROI Label Definitions pane.
For pixel labels, you can only send acceptance or rejection for each image and provide feedback for the labeling task owner.
Send Review Feedback
After reviewing a set of images, you can send the review feedback to the project owner and labeling task owner. Once you send the feedback, the images are locked for review until the project owner and the respective labeling task owner take next steps. Images with at least one rejected label go back to the original labeling task owner to re-label correctly. Images with all labels accepted are marked ready for export and the project owner can export them.
To send your review feedback, from the toolstrip select Send Review Results > Send reviewed images. While you are working on the review task, you can save the project at any stage. This enables the project owner and labeling task owners to access the feedback for the images that you have completed reviewing so far.
In the Image Browser pane the symbol on the corner of each image thumbnail indicates the status of the image. You can access descriptions for the thumbnail badges by selecting Image Status Legend from the toolstrip. This table shows the review progress for each image based on the displayed badge.
Image labeling accepted
Image labeling rejected
Image submitted with review feedback