How can I change the traffic light state in Traffic light Negotiation model to not display any colours (no state)?

I am trying to train a detector to recongise a traffic light that is broken and doesn't display any colours, Green, Red or, Yellow. Is it possible to change the state to no state in the Traffic Light Negotiation with Unreal Engine Visualization Simulink model so I can use this model to train and test my detector?

Answer 1

[Nishan Nekoo]

Hi Meg,

Unfortunately, doing this is not a trivial process but I have been able to do it on my end.

The existing traffic light enumeration in the Level Blueprint of the US City Block Scene only has three states, 0,1, and 2 corresponding to the three states of Red, Yellow and Green respectively. You need to use the Unreal Editor to customize the scene and add a fourth enumeration where all the lights are off. Please follow all the steps closely at the link I sent you above (they are broken down into 4 different pages).

At a high level, these are the steps:

1. First search for the "Enum-TrafficLightsState" in the Content Browser the USCityBlock map, then modify it to add a new Enum for an All Lights Off state.
2. Then, in a few places in the Level Blueprint where this enum is used, you need to replicate the logic used for the other colors for this new enum. In one of the places, you can specify the color used for this enum as well.
3. Finally, you'll need to either hard code or modify the logic in the "Simulation 3D Traffic Light Controller" block to send the new enum to Unreal during simulation.

Hope this helps and please reach out if you need more help.

Nishan

[Meg Wilson]

Thank you, this has been an enormous help.

I have got to a point and I'm stuck with how to get the variable for the new enum state time.

I have attached the parts I have changed so far and have copied the blocks ( changing the state) to configure the logic. The last picture shows the block I am referreing to. The block takes in the Traffic light object and outputs the float variable being Green Time which needs to be changed to the Notate time.

Thanks,

Meg

[Meg Wilson]

How would you go about hard coding or changing the "Simulation 3D Traffic Light Controller" block? Would this be in the state flow or the actual block iteself.

Also are there any only tutorials on this kind of implementation as I am doing my final year project and need to integrate a yolov2 detector to detect these 4 states, navigate through the traffic intersection and I would like to get it working as soon as possible?

Thanks again

[Nishan Nekoo]

Hi Meg,

I would recommend reaching out to [e-mail] for help with specific questions.

Here are some high level suggestions from me:

1. The new enum state time should be set just like the other enum state times. I believe these values are sent to Unreal from Simulink inside the "Simulation 3D Traffic Light Controller". When I tried it myself, I did not add an enum state time for the new enum like you are suggesting, I simply hard coded the value '3' to replace "State" on line 34 of the block 'TLNWithUnrealTestBench/Sensors and Environment/Simulation 3D Traffic Light Controller/State-based'.
2. This block has a mask but you can look under by right clicking on the block and selecting Mask > Look Under Mask or by using Ctrl + U. Here you can see the implementation and how information sent to Unreal is packaged.
3. This block has a library link, meaning you have to right click on it, select Locked Library Link > Go To Library Block. Click the lock icon on the bottom left corner to unlock the library block and modify it. You can also copy out the contents of this library block if you wanted to leave the original unmodified. Whether you change the Stateflow or this library block depends on what you are trying to accomplish.

What topics are you looking for specifically for a tutorial?

Nishan

[Meg Wilson]

Hi Nishan,

I really appreciate all the help and responses.

I just want to confirm how you added the state to the traffic lights to enable a state of no colours on. I opened up the AutoVrtlEnv which is the copied US city block using the Vehicle Dynamics Blockset interface with Unreal Engine 4 toolbox. I then added an enum, 4th state in unreal engine and named it (Nostate). I then edited the logic in the Traffic Lights (BluePrint class) in the unreal engine content browser. This is where I added the 4th state block and set the source material to M_TrafficLights. From the previous email I just want to check I don't have to edit any logic in the Traffic Lights Group (Blueprint Class)?

Once this Unreal Engine is completed I need to edit the 3D simulation Traffic light block how you explained above (which I have done).

The simulation gives an error and cannot open unreal engine when running. Please let me know what I have done incorrectly.

I also just want to ask once I have configured the traffic light to these four states. Is it possible to simulate each state one after each other using the state flow block like a normal traffic light. For example the light is green for 5 secs then yellow for 5 seconds and then red for 5 seconds and lastly no colours displayed for 5 seconds. I want to use the simulation to train a yolov2 detect to detect each state. I am recording the video of the vehicle front camera and using this for the training in Video Labeller app.

I'm not sure this is your area of expertise's but to train the detector do I need one video file of all states in the file or can I train different detectors and merge the detectors of each state to create one large detector for all states.

Sorry for the long post.

Thanks again,

Meg

[Nishan Nekoo]

Hi Meg,

None of this is my area of expertise :) I am just figuring things out as I go. Unfortunately, I do not know the answer to your question about needing one video vs multiple videos. However, from a cursory search, it appears you can "Resume Training a Detector" which seems to imply that you can train it on multiple different data sets: https://www.mathworks.com/help/vision/ref/trainyolov2objectdetector.html#mw_a47a1ef1-9d20-4624-8ceb-597c96e690b6

Coming back to the Blueprint modifications, I modified the blueprint at only the places shown in my screenshots here. I did not edit any logic in the Traffic Lights Group (Blueprint Class). I have not taken the time to fully understand why this wasn't necessary to get it to work for me, but perhaps that is something you can look into. I am not saying this is the only way or best way to do this, it is simply one way that I did it.

One thing I didn't mention -- after modifying the blueprints, you need to use the Simulation 3D Scene Configuration block to switch the Scene Source to "Unreal Editor" since it is now a modified scene. Then when you press Run, wait for Simulink to enter "initialize" phase, then press play in Unreal Editor -- the video will then play in Unreal Editor. I attached a screenshot of the blank traffic lights for me.

Regarding your question is about whether it is possible to simulate one state after the other, I am sure that it is and your workflow seems reasonable to me. Again I don't know exactly how to implement it but I can tell you that it seems like a reasonable approach. However, I am not implementing any logic in the Traffic Lights Group (Blueprint Class) and I am not sure how that might affect a simulation where you want to change the traffic light colors during the simulation. I have only tried it with the color hard coded.

I do not know what you did incorrectly that led to an error, but I'd recommend taking a close look at the error and trying to understand what it is saying, and whether you can debug it yourself.

Unfortunately, I have provided all the information I have so please Accept my answer if you think this helped! If you need anything more, please reach out to [e-mail].

Thanks,

Nishan