Key Issues for Realizing Autonomy of Inland Vessels | MATLAB Day for Marine Robotics & Autonomous Systems, Part 1

Hello to everybody. My name is Dirk Soffker. I'm the head of the Chair of Dynamics and Control from the University of Duisburg-Essen. And my team-- so next slide, please-- can be detected here, from the next slide. You see we will have four persons working on inland vessels in different versions-- Mr. Boschmann, Mr. Thind, and Mr. Bejaoui. And you will see them in a few minutes. So next slide, please.

So we want to talk about autonomous vessels. And we want to talk about remote-controlled vessels. So for autonomous vessels, we need some sensors. The system has to know how to sail from A to B, has to know where the system is, and-- very important-- the system has to know where the system is within the environment, and what this environment is doing. So we call this object detection, and this will be one of our key parts we intend to talk about today.

So the next question is, if we know that there is an object, we have to know what this object is intending to do. That's the second important part. And of course, in all of this stuff, we have to answer questions like, what happens when? Because the functionality is very important, but we have to rely on this functionality. So this will be addressed. This is the overall question we want to address.

In the third part, we are talking about remote-controlled vessels. In this case, we also need sensors, we need data connection, we need a suitably equipped control stand. But all of this is not part of our discussion. Our discussion is to supervise the captain. And we introduce here in our third three part the workflow management.

So from my first words, you can clearly detect that the safety and reliability aspects are very important. And this is exactly what we want to address with the title "Key Issues for Realizing Autonomy and Remote-Control Inland Vessels." So next slide, please.

This means our presentation is structured as given here. First, we want to say something about the realization of reliable object detection. Second part, the realization of reliable intention prediction. And the third part is about the supervision of remotely controlling captains. So have fun.

OK. Thanks for the introduction. So in the next part, we will talk about situational variation, redundant object detection, and finally about the fusion of different object-detection candidates.

First of all, object detection-- what does it mean? So we understand here object detection as the task of recognizing and localizing multiple objects in a specific scene. This is usually performed by artificial neural networks or other machine-learning-based approaches that are trained on recorded and annotated data. This data is usually provided by different sensors, like camera systems, LiDAR, or radar systems. And the training process is usually applied to learn or teach to detect specific patterns that can be associated with certain objects of interest.

So why is reliable object detection actually needed and really required? First of all, it is a really essential task for highly automated systems or tasks like autonomous navigation since we have to perceive the environmental conditions that we have, like other traffic participants, or obstacles that we have to avoid, and so on. The overall objective here is to detect all relevant objects in the given environment and increase the actual safety level, which will finally allow us a fully driverless operation without any captain, remote control, or active communications.

To achieve the goal of reliable object detection, we are applying a redundant-object-detection approach. So we basically assume that there are different situations and using different approaches. Different approaches can have advantages in certain situations while having drawbacks in others.

So we are actually trying to collect a set of experts that are performing well for their specific conditions. And each of them is providing one set of potential object candidates. And finally, we are trying to utilize situational information that we collect from the sensor data to fuse these object detection sets to one final result that we then use in the next steps.

Coming to situational variation-- so situation can be defined by conditions given by the environment or by internal conditions like, for some system states, like velocity or currently driven maneuver. So we can differentiate the situations based on, for example, weather conditions or light conditions or general illumination of the environment, or other properties like distance or conditions of the actual object of interest.

Once we perform the object detection in the training test scenario, we can analyze the detection performance based on different process parameters that can vary with each prediction. So at the bottom, we can see a sample of our data set from the AutoBin project where we can see, shown on the left side, an actually kind of ideal scene with good illumination. Everything is clearly visible and fine, so we expect better performance, kind of.

In the middle, we see a little bit more difficult situation where we have the harbor at night with rain. So we have the starry effects, so that different patterns have to be recognized and interpreted. And then our eyesight gets really difficult, because it's super-dark and there's barely something to be visible. So we can conclude here that real-world applications have to perform in a set of very diverse situations.

So talking about situational dependencies, so we can expect the change on the operating conditions can affect the sensed information; therefore, also the performance of object-detection systems. As initially mentioned, we are trying to obtain a set of diverse object-detection approaches that is then finally able to provide the best result. So we have object-detection systems that are trained for different weather and light conditions to address certain situations. And we analyze the performance based on easily available process parameter.

In this example, we are using the brightness level that we discretize into five levels. So to analyze the dependencies here, we trained one module as a reference, which is the base model, which is train with the default training process. And we have three other models that are trained for specific situations.

The actual results are shown here, with the recall and the precision separately. The interesting point here is that we can see that while the base model is possibly performing as an average, for each brightness level that can be seen, the models are performing differently. So for example, on the right side, with the precision, while the base model performs well in general, for each brightness level there is another model with another specification that still performs better. So we are able to see, first of all, the dependency on this first parameter, brightness. And second of all, we can see that the performance is not equal. So we can choose different strategies for different operating conditions.

Second of all, given a redundant set of object-detection approach, we also have a redundant set of predictions for each time step. Each prediction is provided here with a detection score, which is reflecting the estimated certainty of the predicted object. So as we can see here on the right side, the detection score can be associated with the true-positive rate. So we see here, for example, that the low detection score is associated with a low detection rate, and vice versa.

So now we can associate redundant predictions with each other based on semantic consistency, so same class-- basically, ships are ships and pedestrians and pedestrians and so on-- and geometric consistency like, for example, center distance as a criterion.

Once we did that, we have the information whether there are associated predictions available for each individual prediction. And we can then conclude that in a case where multiple predictions are available for one object, that this is like reinforcing data, and we can verify the detection, or there is no further information about that object, and we should doubt the available detection.

As you can see on the right side, we have here this example of these four digits. Each digit is basically representing one detection system. And in this case, we have just one detection system perceiving the object of interest here. So we can see that in this case, we separate a subset of detections. And even with a high detection score, we see that we have a quite low true-positive rate. So we can conclude that these detections are not that strong, basically.

In another case, we only consider the detections that are detected by all four detection systems. And we can see that for each detection score, we have a highly increased true-positive rate. So we can actually assume that this detection is more reliable-- more strong, kind of. So finally, we can conclude that the redundant detection system of each individual prediction.

Finally, we have to fuse the object-prediction results. And therefore, the fusion here is performed by adjusting the detection score with the set of associated predictions. So the results in this case are shown with a mean value. So in a set of associated predictions, we will average the detection score, basically, and kind of re-weight the prediction by doing this. And to differentiate different cases, we take again the scheme of four digit for all different detection systems.

And in case of four times 0, this is the subset of possible objects that is not detected, which is like 2.5%. And in the case of all digits 1, this is the case where all detection systems detected the object, which is like 85% of all possible cases in our test data set.

The mean performance seen here is the fused result. And on the right side, we see the improvement compared to the best individual result. And what we can see here is that for most of these cases, the fused results is actually achieving improved performance, like up to 6.8% here. So we can conclude here that the fused predictions of redundant systems outperform the standalone performance. So actually, the initial goal can be achieved with this approach here.

I'm going to hand over the second part to my colleague, Mr. Thind.

Thank you, Mr. Boschmann. Hello, everyone. So now the task is that we should predict the intentions. And there are different approaches being used. First is data-driven and model-based approach, and then we have combined the approach to reduce the prediction error. And the third approach is transition probability. Next slide, please.

So I will try to explain this with the help of a diagram where we have two ships in the environment. And ship A is the ego ship, or the considered ship. And ship B is the encountering ship. So once the objects are detected, or the position of the encountering ship available through AIS data, the first task is that we should know the behavior of the ship-- how it is going to behave. And with that we can improve the situation awareness. And that way we can also calculate the collision risk once we do have knowledge about the ship.

The third part is that once the intentions are predicted, we can optimize the behavior of the ship A. And until now, the requirement is that we need to know how reliable are the predictions. So for that, we require this module.

The problem here is that for encountering ship, we can't get the knowledge about rudder angle or the engine speed. So the measurements which are available from encountering ships are just position measurements, or it can be more like speed over ground can be used. So the vessel-motion models which are available until now consider rudder angle and all those hydrodynamic calculations, which is the problem in this approach. And every ship behaves differently, so we have to consider every ship differently.

And the third method is that we have different parameters which have to be adapted for the different kind of ships. And as I mentioned, the model is unknown. And there are also different environmental situations like different flow velocities in different seasons of the year. And even wind velocity occurs as a different environment as the variables.

So here we tried to use the data-driven approach. And the reason is that, for example, if we are using the linear model, then we can't predict the actual behavior of the ship. But when we are using the data from the past year, one year, then we can predict the behavior of the ship with more detail than using data-driven approach.

And then in the next slide we are using the data-driven approach, where we have the tracks of the ship from past one year. We are using that data. Then we are pre-filtering the non-required information with all the non-integer value sensor. Then we are clustering based on-- because the ship behaves, we assume that ship is behaving in different situations like either the ship can be sailing upstream or downstream.

It might have different loading conditions. And we have considered the different seasons as well, because during different seasons, the depth of the water changes. So that's how we are clustering the data.

And then we are comparing the ego track of the considered ship with those clusters and finding the best trajectory using Bayes' algorithm. And the best trajectory will define the predicted behavior of the vessel. And the problem here is that the prediction is based on clustering, so clustering affects the prediction performance.

And the second approach-- until now, we haven't considered and while other variables like flow velocity. For that, we have considered the model-based approach in the next slide, which is working based on this sliding-window approach. Here we can see in the state space that we have assumed a key parameter as an error term, so that every time we are updating the window, to update this parameter.

This parameter is assumed to contain the information related to flow velocity or wind velocity. And so we are updating every 30 seconds this parameter to adapt to the local situations.

So now we have two things-- the model, which is considering the previous information, local previous information like flow velocity and wind velocity-- and the Bayes' approach, which is predicting the next behavior of the ship, like how it can be-- as we can see here, the error is increasing with the prediction time horizon. The reason is that it is assuming the future values as constant values for the whole prediction horizon. But as in the type of in-the-ship COG behavior, it might happen that the ship in the next times wants to reduce the velocity. So this is not included in this model-based approach.

So we are going to, in the next slide, we have defined combined approach where we are predicting the behavior using the local model and the intentions. Here in the figure a, we can see that the model predicts the next position values. And from future intentions, we have taken the speed-over-ground values and course-over-ground to resolve the velocities in x- and y-direction, and then iteratively applying the model to get the predictions for the whole prediction horizon.

So in the next slide, we can see that the combination of these approaches reduces the error. As in the figure, we can see that with the combined approach, the error reduces. And every 30 seconds, we can see that there is a very dark black line, which represents that the local model is adapted every 30 seconds so that the error should not keep on increasing for the whole horizon. So we are also updating the local model every 30 seconds.

And as I already mentioned, the clustering approach still affects the Bayes-approach performance. So we come with a new idea, transition probability approach, which is explained in the next slide.

So what was the idea behind it? First I will explain from the diagram. In the diagram, we can see that the river has hectometer-stone information available every 100 seconds, which are shown here in the pink points. And it is on both sides of the river, that same hectometer-stone information on the side of river. And then we have divided the river into different section, which is shown in the next slide.

Like for the first hectometer-stone, we have divided the river into four different sections. And then we are seeing that whether, from first hectometer section to the next hectometer section, whether there is a transition probability to another section or it is remaining in the same section. So that's how we are classifying the trajectories, so that we can get the best trajectories from the past AIS data.

So that's how we are going to get the clusters, and then apply the same model-based approach to get the error. The results can be seen here in the video.

Here. So here we can see that the error is reduced than the Bayes approach. So now Mr. Abderahman will explain the next task. Thank you.

Thank you, Mr. Thind. This part, the supervision of remote-controlled vessels, will be presented, and so how the captain's behavior can be supervised. So we can go to the next slide.

The challenge is that there are a lack of qualified nautical personnel, and that the amount of connected traffic is increasing. So the idea is that a remote-control station in the land can be developed which allows the control of several vessels in the same zone.

For this goal, an action space will be developed which enable the decision-making so that the possibilities to reach the goal can be compared to each other, and so that the captain's behavior and the captain's actions can be evaluated and compared to each other, so that errors can be detected and alarms can be reported.

For this task, the situation-operator-modeling approach is used. And monitoring strategies will be developed and integrated into action space. So with the combination between the action space and the monitoring strategies, it will be possible that the captain's behavior can be supervised and that the captain can be supported. So we can go to the next slide.

The workflow management is developed which allows the monitoring of the captain's behavior, so that everyone's can be detected, and if necessary, the driver functionality can be taken over. Data are coming from the environment-- so for example, AIS data, our infrastructure data, which will be pre-filtered, fused, and compressed so that characteristics will be extracted. So characteristics describe this current situation, and can be physical or logical terms. So for example, the speed of a ground, the course of a ground of the vessel, or information like if the driving area is free.

So the structured-situation operator and the vector operators are stored in a knowledge base, and each current situation will be compared with the goal as desired situation. And if the two situations are identical, this means that the goal is reached. If not, this means that there are a human error, and an alarm can be reported. So we can go to the next slide.

For the supervision, a concept for the detection of the human errors is developed. So in the lower level, inadmissible operators has to be detected. This is operators and actions which are not inadmissible and don't match with the actual situation and can lead to dangerous situations. And these operators are declared as errors.

In the higher level, there are a distinction between admissible operators which are meaningful and admissible operators which are not meaningful. So admissible operators with respect to assumptions are operators which are accepted, but not meaningful for considering the actual situation. Admissible operators lead to the goal, are suitable, and are declared as useful actions.

So we can see an action space integrate these supervisory strategies, and is consisting of all options which can lead to the goal. For us, it's very important that the supervisory strategies will be integrated into action space so that the actions of the captain can be compared to each other, and so that there are possible alternative actions. So we can go to the next slide.

So in this slide, I want to define the situation vector and the operator. This is the most important terms to get the action space. The situation is represented here as a gray ellipse. And I refer to that same. It's time-fixed and consisting of characteristics-- for example, the speed of all the state of the vessel, information about the environment, if the driving area is free, information about the roads.

The operator describes the change in the actions of the real world-- for example, acceleration, deceleration, if I want to turn. And so with these two terms, action space can be composed. And an example is shown in the next slide.

So an example of a turnaround maneuver is shown here. So we can see that there are more than one possibility to reach a desired situation. So the goal in this situation is that the captain want to turn around, and has to consider other vessels. The white vessel in there is coming from the other direction. And the first possibility is that the captain can pass on the white vessel, wait, then can operate so the bow thruster and the rudder accelerate one time-- so the first possibility, against the direction of the water currents; and the second possibility, in the direction of the water currents.

These two possibilities lead to the goal, but turning around against the direction of the water currents is not meaningful, so the second possibility is useful. So the third possibility-- the captain can decelerate, wait of the passing of the other vessel, and can turn around one time. So the third possibility is turning around in the direction of the water current.

And the last option is against the direction of the water current. So there are one more possibility to reach the goal. So two possibilities here are useful and meaningful, and the two other lead to goal but are not very meaningful. And the graph-based model is represented using situation vectors and operator. And this is a description of how this tree and the action space can be programmed, for example.

So we come to the summary. So in this presentation, the application of the reliability analysis for real-world systems is shown and discussed, and the application of redundant-object-detection approach is discussed. So for the trajectory prediction, two approaches combined with two others, so the Bayes filter and the local model, and are presented to realize an improved result at the end. So the captain's behavior is illustrated using the situation-operator modeling, including supervisory strategies to realize the remote-controlling station.

So in the future, we want to realize redundant object predictions should be fused considering situational variations. The reliability approach will be integrated, so in the trajectory prediction and for the remote-control station. So supervision strategies will be integrated into action space so that the captain's behavior can be evaluated, and the results will be shown and validated in a driving simulator and in a real inland vessel.

So thank you for your attention.