How can I detect the dark black socket from the dark or very reflective bacground?

I need to detect the CCS2 connector of electric cars. It is very dark, with dark and poor textures.

I have some results with intensity transformations using imadjust and gamma correction when I am deparating the connector from the background. Later, I am using morphology.

Still, when I place the camera little far from the connector, 40cm-50cm the image spoils, and the detection fails, specially at daylight. The detection from 20-25cm is 100%.

I am using ZED2i depth camera and diffuse light sources, but the depth image is very poor due to the socket texutes, and now first I need to detect the connector position from the color image.

I would like to use image processing techniques, not deep learning if it is possible (I was thinking about RCNN network, or mask-RCNN, but I can not find the detail description how can I make the training with my own data/images).

I have attached the images too.

Please, any suggestion to improve the detection from 50cm distance (the greater distance would more better).

Naturally, a new camera could help, but the goal is to generate the depth image too, not only the color image.

Thanks!

[Mathias Smeets]

Is it an option to crop the image in advance?

[L]

Hello Mathias,

No, there is no option to crop the image.

The goal is to record the image from about 50cm and detect/extract the connector.

Also, the detection should work with various illumination conditions (daylight, artificial light, etc.)

Answer 1

[Image Analyst]

imadjust won't help. Increasing global brightness or contrast will not help with segmentation. Just think about it and you should know why.

Local contrast enhancement like with adapthisteq or imlocalbrighten might help.

If the connectors are of the same type, you might be able to use normalized cross correlation to find them. Demo attached. But it will depend on the template having the same size as the connector, which is unlikely from images collected from the field.

Otherwise if you had hundreds of photos and wanted to label them all with ground truth of known connector pixels, then you could use segnet.

[L]

Hello Image Analyst,

Thank you for the response. Your advice is very useful, as always.

I think you saw the attached images.

imadjust helped at the beginning to make more visible the socket, naturally it is not enough for reliable detection in every case, since the illumination conditions, position, size can change, also the demand is the 50cm distance between the camera and the socket.

I have tried the adapthisteq and imlocalbrighten functions. The problem remained. The detection from 50cm sometime fails. The detection from 20-25cm is good (since the socket is closer and there are less components around it).

The correlation is a good operation, however it cannot help in this situation. The connectors are always the same, but the connector size can vary in the image and it is impossible to know the template size. It is difficult to set the car and the camera at the same position in every case.

The goal is to use common image processing operations for the detection and extraction.

Maybe some very special industrial camera or sensor (profile sensor, etc.) can help, I don't know? I have ZED2i and ZED cameras (and a commercial photo camera and a phone camera).

I have many photos, it is not a problem to capture more.

I never used CNNs and segnet. Is there a tutorial (Matlab, Python) with details how to train the own data/images with segnet? The segnet would provide the connector area (like the mask-rcnn, or the rcnn), or it would segment and label the image regions including the connector too (then further processing is needed)? Also, what do you think about 6DOF Pose Estimation with DOPE? Did you know something about that? It should detect and label the bounding box around the object (I saw a video about this, I am not familiar with it).

Do you have an idea how to limit the capturing conditions from 50cm to ensure a better image (because the connector is very dark, without visible textures in the RGB image)?

Thank you in advance.

[Image Analyst]

Well you accepted @Mathias Smeets answer so I guess you got it all figured out. Again imadjust will not enable a better segmentation. It will make it look nicer to you but that won't affect the segmentation. For example, let's say your image histogram went from 40 to 60 gray levels with a best threshold at 50. Then using imadjust will now make the image go from 0 to 255 and it will look better and brighter. But what happens to the threshold? Well now the threshold would be at 128 instead of 50, but that will give the same binary image as if you hadn't done imadjust. You just get it with a different threshold value.

I do have a nice segnet trainer app but I'm not sure I've generalized it enough for general purpose use yet. It's on my to do list. I'm using it today as a matter of fact to find some items on a human body.

[L]

Thank you. I have accepted your answer, as always.

I see, but could I replace the imadjust with other function?

The first problem is when the car and the socket are the same black color. The second problem is when the background is very reflective and the socket is not visible enough. The third problem is when the socket is in shadow and it is not visible.

I have tried the thresholding without the imadjust. Mainly, the socket melts into the background, specially when the background is black. Futher, there are too many components in the image after the thresholding and it is hard to extract the socket, specially from 50cm. From 20cm the thresholded image is better, but still it is a problem to extract the socket.

Generally, is it possible to detect the socket realiably with classiccal image processing techniques (under certain conditions of image capturing)? Now, I am thinking about that too.

The segnet results with direct detection of the socket, or it segments the image in regions?

Yor segnet app is trained in Matlab, or you have used other platform?

Also, the problems of black background and socket invisibility can affect the segnet (or any other CNN)? I am not familiar with CNNs, but I think the low image quality and the darkness could be a problem to CNNs too. Generally, the illumination in this case is a problem due to the reflection from the car.

Is it possible to use some texture filter? The socket is mainlye textureless due to the darkness, and that is the reason I didn't try. I have tried the edge detection and the morphological gradient, but still the socket is not elevated suitably from the background.

[Image Analyst]

Well black on black with no contrast is almost impossible. Just what do you need to find exactly and how accurate does it need to be? Maybe you don't need to find the exact location of every perimeter pixel but maybe just the centroid is good enough. Why do you need to know anyway? Are you going to have a robot try to plug a cable in using computer vision or something? Maybe you could have a light as well as a camera on the end of the robot arm.

[L]

The best result would be the binary mask of the connector, with its contacts. Hence, the accuracy should be high.

The other possibility could be the detection of the centroid too. Maybe after the centroid detection, in the second step the accuracy can be improved? Maybe this could be a good solution?

The goal is to detect reliably the connector for the scientific project, and the possible application could be the plugging of the cable (with robot, with automated arm, etc.), however, this is only the beginning of the project. The topic of this project is interesting from the image processing point of view, since it is very difficult the detection of the dark socket.

I am aware that is difficult the detection (specially with black environment), I have mounted two LED lights near the camera. It helped, however the detection fails in some examples. The light can cause reflections too, and they can blind the camera.

I have attached you a good result. I have cropped the good results.

[Image Analyst]

There are 3-D cameras now. Maybe using a camera of that kind might help.

[L]

I am using a 3D depth camera, but the depth information of the socket is bad since the socket is very dark and the depth camera cannot estimate the socket depth information. Also, the reflections and shadows often degrade the depth image and it not contains useful informations.

[L]

Hello Image Analyst,

I have tried a new approach, as you have suggested. I have used a narrow light source to highly illuminate only the socket. Now, the socket is quite visible. Next, I have used the Sobel edge detector. The edges are detected, however there some components in the left part of the binary image to eliminate. Now, I am wondering, what could be the further step.

On first look, this looks good.

What do you think? Maybe some dilation operation to connect the components related to the socket area?

[Image Analyst]

What exactly do you want? A mask of the rectangular plate surrounding the connectors? The location of the round things? Attach the original image, not a screenshot with caption and white surround.

[L]

That is the original image. The white part is the model with the socket inside, the white area is the simulation of white colored car. I have captured the image using the narrow light source (a common LED hand lamp).

The mask of the socket is the goal, not the rectangular plate.

[Image Analyst]

Are you sure? I've never seen a machine vision camera add a ton of white padding around the image and a caption "RGB Image" above the image in the white padding area. Usually they just give you the image alone without any frame/padding around the image. I guess we could crop out the real image since we really don't want that white surround, but it's just extra work that shouldn't be necessary.

[L]

I have captured the image using the camera and the LED hand lamp. I have manually set up the lamp.

Sorry, I have attached the wrong images with the captions. These are the good images. Sorry, my mistake.

[L]

Hello,

This is an update to previous post. I did the adaptive thresholding. The image is attached. The global thresholding is not useful since the enviroment is white.

Now, I am thinking about simple operations to extract the mask of the socket area.

Maybe the combination of the edge image with this can produce a reliable solution (the goal is with any enviroment)?

[L]

This is the example with a silver (alu-foil) background. The socket is illuminated with the same hand lamp as before, however now the sunlight is affecting the scene from the left side. The results of the adaptive thresholding and the Sobel edge detector are shown in the sample images.

The possible solution is to illuminate the socket in dark. However, is it possible to separate and extract the socket from this kind of input image (to get the mask of the socket)?

Possibly this is one of the worst cases.

Answer 2

[Mathias Smeets]

I don't know a specific solution.

However, here is a link that might help. I had a similar (easier) problem and I tried out a lot of different things on this link under 'segmentation techniques'.

Another thing you might try is using the functions imadjust (for enhancing contrast) and imlocalbrighten (for brightening) before you do the actual processing.

[L]

I have used the imadjust before the processing. It helped, however when the connector is too small in the scene (when the camera is about 50cm far from the connector), sometimes severeal similar blobs are remained and then it is difficult to extract the connector.

I have used the brightening option too, but it didn't help, The results are the same.

I think some kind of imadjust function (a similar function) could be the solution to separate the connector from the background, but I didn't find out what.

[Mathias Smeets]

Can you provide an image of this result when there are several similar blobs detected?

[L]

I am sending you two good results when the detection is correct, and I am sending you the image after imadjust and few operations. In this example the detection fails. Generally, if there are several white areas with similar dimensions as the connector itself, that is the problem.

[L]

I have attached now the images.

The file names explaine the images.