Create Review Tasks for Labeled Images
The Image Labeler app enables the project owner of the team to create a labeling project setup and distribute the images into labeling tasks before assigning them to labeling task owners. For more information about how to set up a team labeling project as a project owner, see Create Project for a Labeling Team and Publish Tasks. After the labeling task owners complete the assigned labeling tasks, they share the labeled images with the project owner. For more information about how to complete an assigned labeling task as a labeling task owner, see Label Images and Send for Review.
To ensure that the labels received from the distributed labeling tasks are correct, the project owner creates distributed review tasks for the review task owners. The review task owners are the team members who review the shared labeled images and provide feedback to the labeling task owners.
This example shows you the steps that a project owner must follow to:
Open an existing project.
Check the progress of shared labeling task images.
Create review tasks for labeled images.
Distribute the labeled images for review among the team members.
Open Existing Labeling Project
Open the Image Labeler app using one of these options:
MATLAB® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.
In the MATLAB command prompt, enter
To open an existing team project, such as the one you created in the Create Project for a Labeling Team and Publish Tasks step, follow these steps:
In the Image Labeler app interface, select Open Project.
In the file explorer, navigate to the folder that contains your previously created project. Select the
.prjfile, and click Open.
Once you select and open the team labeling project, the app enables you to open the project in Project Owner or Task Owner mode. To access the project you created as a project owner, select Project Owner.
Check Status of Images Assigned for Labeling
In project owner mode, the Image Labeler app displays the published labeling tasks and the individual labeling task progress in the Tasks pane.
In Create Project for a Labeling Team and Publish Tasks, you created three
labeling tasks. In Label Images and Send for Review step, the task
owners labeled the assigned images. The Progress column indicates
that all 50 images each of
Task 1 and
Task 3 have
been labeled and shared for review. For
Task 2, 44 images have been
labeled and shared for review, while the remaining 6 images have not yet been labeled or
the labeling is in progress.
Progress is defined as C/N. For a labeling task, C is the number of labeled images, and N is the total number of images in the labeling task.
C varies over time in the project. For example:
If the labeling task owner shares all the images in the labeling task for review, then C = N.
If the project owner creates a review task for labeled images and the review task owner rejects two images and accepts the rest, the labeling task progress is (C-2)/N.
The Image Browser pane displays the thumbnails of all the images
in the project. Using the Image Filters list, you can filter the
images based on status. Set the filter to
Images that can be assigned to a
review task, and the Image Browser displays the
144 images that have been labeled and shared for review.
Changing the Image Filters option updates the current project images with the most recently saved images and labels from other project contributors and re-saves the project.
The symbol on the top right corner of each image thumbnail indicates the status of the image. You can access descriptions for the thumbnail badges by selecting Image Status Legend from the app toolstrip. This table shows the image labeling progress for each image based on the displayed badge.
Image pending in an unpublished label
|Image is assigned to a label task that is not yet published.
Image locked by a label task owner
|Image is assigned to a label task, and labeling is in progress.
Image labeled and ready for review
|Image labeling is complete, and the image is ready to be assigned to a review task.
Image pending in an unpublished review
|Labeled image is assigned to a review task that is not yet published.
Create Review Tasks and Assign to Task Owners
To ensure labeling accuracy in the labeled data shared by the labeling task owners, the project owner can assign the labeled data to review task owners for review. To distribute the shared labeled images for review, you must create one or more review tasks for the team members, add labeled images to each review task, and then assign the tasks to the individual reviewers.
Create Review Task
The project owner can assign only images with a
Image labeled and ready
for review status to a review task.
Image labeled and ready
for review images are the images that are marked as labeled and shared
for review by the labeling task
In the Image Browser pane, set Image Filters to
Images that can be assigned to a review task. The Image Browser displays only the
Image labeled and ready for reviewimages.
On the app toolstrip, select Add Review Task. The app adds a new task to the Tasks table, of Type
In the Task Information pane, set Name to
Set Owner to
Team Member D.
In the Instruction field, enter
Review labeled data and modify incorrect labels.
Add the first 50 images in the Image Browser pane to this task.
Select the first image in the Image Browser.
In the Task Information pane, in the Select Images field, enter
50and press Enter. This selects the first 50 images sequentially, starting from the selected image. The Image Browser highlights all selected images in blue.
Click Add Images To List to add the selected images to the task. Once added, the image names show up as entries in the Images List. In the Image Browser, the thumbnail for each added image now has an image status legend,
Image pending in an unpublished review task, in its corner, indicating that it is associated with a review task.
You cannot assign the same image to two labeling tasks or two review tasks simultaneously. Each image can be part of only one labeling task and only one review task.
Repeat steps 1–5 to create a new task named
Team Member E, adding the next 50 labeled images in the project.
Use the same process to create a new review task named
Team Member F, and add the remaining 44 labeled images.
The Tasks pane now lists
Task 5, and
Task 6 with and their assigned
task owners along with the previously published labeling tasks,
Task 2, and
Publish Review Tasks and Share Project with Review Task Owners
Publish Review Tasks
Publishing the tasks saves the project in a state that enables individual review task owners to open the project and start working on their assigned review tasks.
For this example, publish all the unpublished review tasks at once. To do so, on the app toolstrip, select Publish Tasks > All Unpublished Tasks.
Alternatively, to publish a single review task, in the Tasks pane, select the task you want to publish. Then, on the app toolstrip, select Publish Tasks > Selected Task.
Share Project with Review Task Owners
Save the project after publishing the review tasks. Then, on the app toolstrip, select Publish Tasks > Share Instructions to open a dialog box with a shareable link to the project location. Ensure that the review task owners have read/write permissions for the shared location, and then share the link with the task owners so they can start working on their review tasks.
Distribute the shareable link to the task owners through email, or as a message outside the Image Labeler app.
Create Executable for Review Task Owners Without Full MATLAB License
For review task owners who do not have access to a full MATLAB® license, you can create a compiled version of the review project in the form of a standalone executable using these steps:
On the Image Labeler tab of the app toolstrip, click Compile.
Choose the app options for compilation based on the task. For this example, set App Type to Review as you are creating this executable for review task owners.
Specify the App Name, Location and include any automation algorithms necessary for labeling.
Click Start Packaging. This creates an executable that you can share with the review task owners.
Review task owners can use the executable to install the standalone app, open the app, and then complete the labeling tasks.