How to classify images depending on the shape of each image's object ?

[Mahesh Taparia]

Hi

Can you give more details what you want to do with these images?

[Mohamed Elbeialy]

I am looking to classify imageData store depending on the shape of each object inside each image. This can happened by find a max value then start training and classifying. The above images are just example of the images inside ImageData store. I want to make sure that the process of finding max value in image will cover all Datastore images

[Walter Roberson]

Max value of what?

You had a recent Question in which you did not explain what you wanted a max value of, but went along when people asked if you were looking for maximum brightness.

[Image Analyst]

What classes do you have?

[Mohamed Elbeialy]

5 classes of cars, flowers, buidlings, trees, dogs. The point is to classify them according to the shape of the object inside each image. Using max value of the image is just a hint. if you have another way to determine the shape of object, ues it .

[Amit]

Follow following steps,

1. First of all you need to binarize the image and find edges of image using canny edge detection.
2. Then you need use regionprops to extract various isolated regions in the image.
3. You need to find centroid of regions.
4. Then at various angles, you can find distance between centroid of region and point on edge of image.
5. This makes your shape descriptors.
6. You can compare these descriptors with descriptors of known share to categorize your query or unkown object.

This should work for you.

I have my IEEE paper published regarding above process, you can send request to me on, [e-mail], I will send you my IEEE paper where this process is explained in details with images showing intermediate results.

Answer 1

[Mahesh Taparia]

Hi

There is already an existing answer similar to this problem. You can refer this link for that.

[Mohamed Elbeialy]

This link does not have any relation with my question.

Answer 2

[Image Analyst]

Try the transfer learning example with CNN/AlexNet. There should be demos in the Deep Learning Toolbox.

[Mohamed Elbeialy]

Do you have other solutions?

[Image Analyst]

Why do you need to do this? What is your use case?

[Mohamed Elbeialy]

The point is to classify images depends on the shape in each image instead of RCS info of each image

[Walter Roberson]

RCS - radar cross-section??

Do you have the same scene in multiple modalities?

[Mohamed Elbeialy]

yes, RCS is radar cross section, which I am looking to substitute with shape info of each image

[Walter Roberson]

Radar has a wavelength of about 3.4 cm. You are viewing through air, which has an index of refraction of 1. You cannot use optical glass with radar to raise the index of refraction. According to Rayleigh you cannot do better resolution than 1.22*λ/refraction so you are not going to be able to resolve much better than 1.22*3.4 cm... so roughly 4cm.

This is a problem for you as most flowers (one of the classes you have to deal with) are less than 4 cm. There are larger flowers, such as amaryllis or sunflower, but you have a problem.

For example in the farm image, the yellow patch is canola flowers, and those flowers are appropriately 17 mm wide, which is at best 1/20th of the resolution possible with normal radar systems.

[Mohamed Elbeialy]

for this reason, I asked to find the shape of each image according to the index and can adjust all images' index to value (1)

[Walter Roberson]

The only things that I can think of that you might mean by "shape", you already said are not appropriate for your situation, when you said that the link that @Mahesh Taparia provided was not relevant to your work. The link Mahesh gave included information on Feature Extraction, and the "shape" of something is very often characterized by doing Feature Extraction. Any kind of measurements that return sequences of information about how a blob turns, how circular it is, and so on, is considered Feature Extraction, but you have told us that is not something you want to do, so we do not understand what you are after.

[Mohamed Elbeialy]

This is the code I built, and looking to edit it to a code that classifies the shape of each image. I am not looking to specific shape such as triangle, circle, square, but the shape of the object inside each image, which can be anything random shape.

imds = imageDatastore('images','IncludeSubfolders',true,'LabelSource','foldernames');
[imdsValidation,imdsTrain1,imdstest,imdsTrain2] = splitEachLabel(imds,0.1,0.1,0.8)

layers = [
    imageInputLayer([227 227 3],"Name","data")
    convolution2dLayer([11 11],94,"Name","conv1","BiasLearnRateFactor",2,"Stride",[4 4])
    reluLayer("Name","relu1")
    crossChannelNormalizationLayer(5,"Name","norm1","K",1)
    maxPooling2dLayer([3 3],"Name","pool2","Stride",[2 2])
    convolution2dLayer([3 3],384,"Name","conv3","BiasLearnRateFactor",2,"Padding",[1 1 1 1])
   fullyConnectedLayer(5,"Name","new fc","BiasLearnRateFactor",10,"WeightLearnRateFactor",10)
    softmaxLayer("Name","prob")
    classificationLayer("Name","classoutput")];

miniBatchSize = 25;
valFrequency = floor(numel(augimdsTrain.Files)/miniBatchSize);
options = trainingOptions('sgdm', ...
    'MiniBatchSize',25, ...
    'MaxEpochs',8, ...
    'ValidationData',augimdsValidation, ...
    'ValidationFrequency',valFrequency, ...
    'ValidationPatience',4
    'Plots','training-progress');

trainedNet = trainNetwork(augimdsTrain,layers,options);  % train the network 
[YPred,probs] = classify(trainedNet,augimdsValidation);   % classify the validation images
fracCorrect = accuracy/numel(YPred)

[Walter Roberson]

What difficulty are you having running that code the way it is?

You will need a lot of examples to train on; and you might need Transform Store in order to resize the images https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.transform.html

[Mohamed Elbeialy]

I do not need to use transform store. The code is ok with me. Just need to add detecting the shape part only

[Walter Roberson]

Suppose that you had a routine detect_shape() that took in an image as input, then what should the function return?

For example for the farm scene, what should be returned? It looks to me as if that scene has cars, trucks, buildings, trees, canola (rapeseed) in bloom, and that the red might possibly be something else in bloom, possibly Crimson Clover. What should be returned from the hypothetical detect_shape() return for it?

[Image Analyst]

Why are you not posting your radar images? Why are you posting only RGB optical images? You can't have one algorithm that works for every possible kind of image out there.

[Mohamed Elbeialy]

these are my radar heatmap images that looking to classifiy by shape using max value of 1.

[Image Analyst]

What do you mean by "using the max value of 1"? You should have posted this in the first place instead of those other images that had nothing to do with your problem. Don't send us on while goose chases about cars and trees and animals. So for these 4 images, what shape differences do you see and what class would each be?

See this link:

Morphometrics - Wikipedia, the free encyclopedia

[Mohamed Elbeialy]

classes are same as before, dogs, buildings, flowers, cars. each image represents a heatmap image for radar data. Making index of 1 to all images will help finding the shape of each image. The points is detemining the shape of the object in each image. I think It's clear now. You do not have to think further than this edge. Your link does not have anything to do with my inquiry

[Walter Roberson]

I notice that your code uses augimdsTrain which is a variable name consistent with using an Augmented Image Data Store, but you have not defined any Augmented Image Data Store, and have indicated firmly that you do not need one?

[Mohamed Elbeialy]

There is already an image data store I use. I need only to detect the shape of each image in that datastore

[Walter Roberson]

imds = imageDatastore('images','IncludeSubfolders',true,'LabelSource','foldernames');

Okay, you have an image data store named imds

[imdsValidation,imdsTrain1,imdstest,imdsTrain2] = splitEachLabel(imds,0.1,0.1,0.8)

Okay, you have four more image data store objects whose name begin with imds

valFrequency = floor(numel(augimdsTrain.Files)/miniBatchSize);

augimdsTrain is not defined. The variable name is consistent with an augmented image data store, not with an image data store.

options = trainingOptions('sgdm', ...
    'MiniBatchSize',25, ...
    'MaxEpochs',8, ...
    'ValidationData',augimdsValidation, ...
    'ValidationFrequency',valFrequency, ...
    'ValidationPatience',4
   'Plots','training-progress');

augimdsValidation is not defined. The variable name is consistent with an augmented image data store, not with an image data store.

valFrequency is not defined.

You are missing a line continuation on the ValidationPatience line.

   trainedNet = trainNetwork(augimdsTrain,layers,options);  % train the network 
[YPred,probs] = classify(trainedNet,augimdsValidation);   % classify the validation images

augimdsTrain and augimdsValidation are not defined.

[Mohamed Elbeialy]

How about detecting the shape of each image. What will be the code to do this step

[Walter Roberson]

Suppose that you had a routine detect_shape() that took in an image as input, then what should the function return?

For example, for your image

img = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/596385/image.png');
imshow(img)

What is the data type of the expected return value? What is the size of the expected return value?

[Mohamed Elbeialy]

The size of image should be same but the detected shape will be changed according to the inside object.

[Walter Roberson]

Are you implying that if you did have a detect_shape function, that the value returned from it should be the same size as the original gray image? Or since your input layer resizes, is it the resized image that would be input to the function and you would want the output to be the same size as that?

What is the data type you would want such a function to return?

Are you asking for a function that will detect a shape in the image and return 1 in each location that is considered to belong to the shape, and 0 otherwise?

Are you looking for this as a layer for use in a CNN?

Or are you looking for a function that would use CNN in order to figure out which locations in the image were part of a shape?

[Mahesh Taparia]

Hi

@Mohamed Elbeialy Can you specify what kind of output you want from those SAR images? What is the label of those images? What kind of network you want? What it should predict based on those SAR images?

Can you post a sample input image and its corresponding output that you want from the network?

[Mohamed Elbeialy]

Network is ALexNet, and labels are dogs, cars, trees, buildings. I am looking to classify image after determining the shape of each image. I do not know how outcome is, for this reason, I asked for help.

[Mahesh Taparia]

You mentioned that labels are dogs, car etc. Then is it not a image classification task?

What is the meaning of shape of each image? Is is not a MXN matrix? What you will do with shape?

Again you mentioned that you don't know about the outcome. What kind of output is not known? You already mentioned that labels are dogs, car etc.

It is not clear to me what you are trying to achieve.

[Mohamed Elbeialy]

dogs , cars, etc are classes that have a lot of images related to each class. I have classified those classes using RCS of image. Now, I want to use the shape of each image and classify them to compare the outcome of classifing accuracy between RCS and shape

[Walter Roberson]

Could you give us an example of what the "shape" of the image would be as a matrix? For example for your first image,

img = imread('https://www.mathworks.com/matlabcentral/answers/uploaded_files/596385/image.png');
imshow(img)

could you draw a sketch of what parts of that should have what value as a "shape", since you have indicated that you do not mean maximum brightness over the whole image?

[Mohamed Elbeialy]

Here it is. how to make sure that the network will detect all random shapes of all images inside the imageData store

[Image Analyst]

To get a binary image mask of inside where you traced the outline:

binaryImage = grayImage < someValue;

You'd have to train (label) your training images all with the outline and with the class you know them to be (car, dog, etc.)

Answer 3

[Walter Roberson]

imageInputLayer([227 227 3],"Name","data","Normalization","rescale-zero-one")

This will rescale each input image to have a maximum value of 1.

[Mohamed Elbeialy]

how do I verfiy the outcome that each image has normalized according to max value of 1

[Walter Roberson]

Use the debugger with trainNetwork ?

[Mohamed Elbeialy]

Your code line does not work. It gives a code error. Could you try the whole code and show me the result

[Walter Roberson]

imageInputLayer([227 227 3],"Name","data","Normalization","rescale-zero-one")

ans = 
  ImageInputLayer with properties:

                      Name: 'data'
                 InputSize: [227 227 3]

   Hyperparameters
          DataAugmentation: 'none'
             Normalization: 'rescale-zero-one'
    NormalizationDimension: 'auto'
                       Max: []
                       Min: []

R2021a says it is okay. The documentation for R2020a says it is okay. I will install R2020a and check it there.... Yes, R2020a says it is fine.

[Mohamed Elbeialy]

Rescale image is not same as detecting object's shape. I do look for object detection in each image

[Walter Roberson]

You wrote, and I quote, "Making index of 1 to all images will help finding the shape of each image.". Using the normalize option makes the index 1 for each of the images, if you interpret "index" as brightness. We had asked you what you meant by "index" and as you did not answer, we had to guess that you meant brightness like you did in your previous question.

[Walter Roberson]

There is no object detection layer. https://www.mathworks.com/help/vision/object-detection.html

[Mohamed Elbeialy]

I did not mention index or brightness, I mentioned max value of the shape of the object inside each image. so, max value related to the shape not brightness. You could work on the code according to the shape max value not brightness

[Walter Roberson]

You wrote, and I quote, "Making index of 1 to all images will help finding the shape of each image.", so you did mention index.

What, to you, is "max value of the shape of the object inside each image"??

[Mohamed Elbeialy]

max value is visible pixel on the shape of image. also, what rescale-zero-one represents

[Walter Roberson]

When you create an image data store and specify 'normalization', 'rescale-zero-one' but no 'Min' or 'Max' value, then by default the code does the operation

minval = double( min(ThisImage(:)) );
maxval = double( max(ThisImage(:)) );
ScaledImage = (double(ThisImage) - minval) ./ (maxval - minval);

The outcome of this is that the maximum value of ScaledImage will be 1 and the minimum will be 0. The data will be rescaled from whatever range it happens to be. For example if the image happened to be grayscale uint8 data in the range 10 to 62, then you would subtract 10 from each location and divide the result by (62-10 = 52). This would map the 10 values to 0, and would map the 62 values to 1, and everything in-between would be linear scaled -- so for example (36 - 10)/(62-10) -> 1/2

In other words makes the value of the maximum pixel into 1 and all other values will be less than 1.

The difference between what this does (which you said earlier you do not want to do) and what you want to do is... difficult for us to understand. Perhaps you only want to look at a subset of the image and scale that subset, that you want to detect a "shape" first with the shape not including all of the image, and then you want to find the maximum value.

Or perhaps you see some difference between the values of the pixels as passed in by reading the image, as being different than "brightness" ??

It is hypothetically possible that even though a particular pixel might have the maximum uint8 value of 255 (maximum brightness), that you have a non-linear mapping between pixel value and the value you want to scale against. For example it is hypothetically possible that the intensities at the pixels represent the distance from the "center", with the "center" value being near 128, and that what you are looking for as the "maximum" might mean "closest to the center value" instead of maximum displacement that displays as "brightest". We would tend to think that you would have already told us if that was the case, but since you have not told us otherwise, and have told us that you don't mean "brightness", then we cannot rule it out.

[Mohamed Elbeialy]

why are you insisting about rescale. I am not looking to rescale image, only detect the shape of the image to classify according to shape info instead of all image info which represented by (RCS). I did not expect that the matter will be too hard like this.

[Walter Roberson]

Well, if you could respond to https://www.mathworks.com/matlabcentral/answers/808630-how-to-classify-images-depending-on-the-shape-of-each-image-s-object#comment_1483345 with a sketch of the output you expect, then we could understand better.

You want the maximum value of something to be mapped to 1, which is a rescale operation acting on something . We are having a difficult time understanding what you are wanting to have become 1.

[Mohamed Elbeialy]

Forget about max value. just focus on detecting the shape of the image, then classify it to get the best validation accuracy

[Image Analyst]

I'm going to bow out soon. You said you want to determine shape for certain blobs in radar images and then gave us 4 images that looked virtually identical.

Then I gave you a link to a Wikipedia article on Morphometrics, which by definition is the study and characterization of shapes, and you said that it did not apply

My final suggestion will be to look at Fourier Descriptors, which is another field that studies shapes. Find lots of web sites on it here: Search Google for "Fourier Descriptors".

If transfer learning/cnn, Morphometrics, and Fourier Descriptors all don't work, then you'll just have to invent your own custom classification algorithm.

[Walter Roberson]

Just focus on detecting the shape of the image

Please reply to https://www.mathworks.com/matlabcentral/answers/808630-how-to-classify-images-depending-on-the-shape-of-each-image-s-object#comment_1483345 with a sketch of what you expect the result of detecting the shape to look like for that image.

[Mohamed Elbeialy]

I do find the code for detecting object inside image, but I do not know how to apply to the whole imageDatastore. Any suggestions?

[imds,blds] = objectDetectorTrainingData(gTruth)

[Image Analyst]

Just get the datastore and loop over all the filenames in it and call the code you have. Can you share "the code for detecting object inside image" that you say you have? I'll show you.

Also, answer all Walter's questions that you have not answered yet, such as outlining the shape of the car, tree, or whatever that you see in the image(s) you posted?

[Mohamed Elbeialy]

here it is, however, I stucked with (gTruth ) which does not allow me to insert all imageData store images [imds,blds] = objectDetectorTrainingData(gTruth)

[Image Analyst]

You asked: "I do find the code for detecting object inside image, but I do not know how to apply to the whole imageDatastore." So, try this:

ds = imageDatastore('*.png')
numFiles = numel(ds.Files)
% Apply "code for detecting object inside image" to "the whole
% imageDatastore" - apply to every image in the image datastore.
for k = 1 : numFiles
	thisFullFileName = ds.Files{k};
	fprintf('Analyzing #%d of %d : "%s" ...\n', k, numFiles, thisFullFileName);
	theImage = imread(thisFullFileName);
	imshow(theImage);
	[folder, baseFileNameNoExt, ext] = fileparts(thisFullFileName);
	title(baseFileNameNoExt, 'Interpreter', 'none');
	drawnow;
	% Now give code to do something to analyze theImage...
        % Put your existing code "for detecting object inside image" here:
end