How can I improve segmentation of these images?

I'm trying to segment ants in several thousand images (I've attached two images here). I'm then applying a colour detection function, with the ultimate aim of detecting paint marks on the abdomen of ants.

I've had some success with the below segmentation, but it also sometimes picks up the brood (the pale oval-shaped things in the image), as well as some of the border. I can't just use something like imclearborder, because often there are ants connected to these border pixels, right on the edge of the image.

I have also tried

• K means clustering - but often the colour blobs are treated seperately to the ant body
• imfindcircles (to try and isolate just the abdomen of ants) - not much success so far

I'm just wondering if anyone has any input or other methods to improve the segmentation.

ant = imread("ant.jpg") % read original image, call it "ant"
burnedAnt = createAntSegmentation(ant) % apply local function to ant image

%% display images side by side, check segmentation not too strict
fig = figure();
ax(1) = axes('Units','normalized','Position', [ .1 .1 .4 .8]);
ax(2) = axes('Units','normalized','Position', [ .5 .1 .4 .8]);
imshow(ant,'Parent', ax(1)) % display original image
imshow(burnedAnt, 'Parent', ax(2)) % display new burned image
linkaxes(ax)

%% create function to segment ants, and burn the mask onto the original
% image
function burnedAnt = createAntSegmentation(ant) 

greyant = rgb2gray(ant); % convert image to grayscale
% threshold image. Higher = more lenient filering
adaptedAnt = adaptthresh(greyant, 0.38,"ForegroundPolarity","dark"); 

BW = imbinarize(greyant,adaptedAnt); % binarize ant image, using the above thresholding
BWopen = bwareaopen(~BW,800); % exclude pixels smaller than X

se = strel("disk",32); % create shape used for dilating image
% dilate image, expand the white pixels to ensure not cutting off relevant areas
BWdilate = imdilate(BWopen,se); 

BWfilled = imfill(BWdilate,"holes"); % fill in any holes
%produce final image, mask burned onto original image, with black (k) fill.
burnedAnt = imoverlay(ant,~BWfilled,"k"); 

end

Answer 1

[Image Analyst]

Did you try the Color Thresholder on the Apps tab of the tool ribbon? Set it up for hsv color space and tell it to find green blobs and then export the code. Do you need the whole ant, or just need to know how many green blobs there are? It could get tricky if some of the ants touch each other, connecting the blobs.

[BC]

So the end goal is to detect paint marks and extract their coordinates, which is mostly working in a separate function. So it doesn't matter if the segmentation connects ants at all, and even just segmenting the abdomens could work too. I don't need to know the number of ants present.

I think I had experimented with the colour thresholder app, but the issue I had is that sometimes the paint from the ants smudges onto the arena, and these smudges aren't always smaller than a paint blob on an ant that I actually want to detect.

So I first have this segmentation step to get rid of some of these paint smudges, before running a colour detection function similar to the app, in which I create a shape around the paint blob, and use this as a reference in LAB space to detect similar pixels. The colour detection itself seems to run ok, though I think I will have issues with blue paint marks due to shadows in the JPGs being detected as blue (I can't avoid using JPG format unfortunately). Though that is perhaps more suited for a separate question!

So, the segmentation above definitely helps avoid paint smudges on the arena, but it's not perfect. But maybe this is as good as I can get it without cutting off any ants. Thanks again for your time, appreciate it!

[Image Analyst]

If this is a video, you can preprocess all the frames to get all the green blobs. Then identify those blobs where the centroid is the same. These will be paint marks on the arena. Can we assume that all the paint marks on the arena were present before the video started? Then make a second pass where, if the centroid is the same, mask out that blob to remove the spot on the arena.

[BC]

It's not a video unfortunately, it's a still photo taken roughly every 15 minutes. Depending on the trial, there are 1000+ images.

But that idea sounds good - if not possible to preprocess with images, I could always e.g. just check the coordinates in Excel later and just sort them so I can see if a given centroid appears a large number of times and remove it?

I will have to check the images when I have access again on Monday, but yes I imagine most of the paint marks will be present at the start of the trial, though not 100% sure.

[Image Analyst]

Yeah, it doesn't have to be a video. A video is essentially just a series of still photos anyway, just like you got. I'm sure it can get tricky quickly, like if ants crawl over each other or touch, and some ants don't seem to have a very vivid green dot, if an ant is covering an arena mark in the first image, etc. So I'm just describing the simplest case. I'm sure you'll run into lots of cases where it gets tricky. Good luck though.

[BC]

Yeah there's lots of little issues that'll crop up for sure! It'll never be 100% accurate but I'm trying to get it as good as I can. Thanks again for the help :)