Reinforcement Learning: Use trained agent in Simulink

Hello,

I have a problem to use my trained agent in Simulink. I try to structure this thread so good as possible. First of all I want to mention that I use Matlab 2020b.

At first I want to introduce the structure of my neural network:

mdl = 'Simulink_Model'                      % Speichern des Simulink-Modells in der Variable mdl

% Actions
actionNum = 1                               % Anzahl der Aktionen
actionInfo = rlNumericSpec([1 1],...        % Erstellung des Action Spezifikations Objekts
    'LowerLimit', -9,...
    'UpperLimit', 9);
actionInfo.Name = 'Voltage Supply';         % Bezeichnung des Action Spezifikations Objekts

% Observations
observationNum = 4                          % Anzahl der Beobachtungen
observationInfo = rlNumericSpec([4 1],...   % Erstellung des Observation Spezifikations Objekts
    'LowerLimit', [-inf -inf -pi()/2 -inf]',...
    'UpperLimit', [inf inf pi()/2 inf]');
observationInfo.Name = 'Observations';
observationInfo.Description = 'Wheel Velocity, Wheel Position, Body Angle, Angle Velocity'; % Bezeichnung des Oberservation Spezifikations Objekts

% Environment
env = rlSimulinkEnv(mdl, 'Simulink_Model/RL Agent', observationInfo, actionInfo)
env.ResetFcn = @(in)localResetFunction(in)      % Reset Function
rng(0)                                          % Fix the random generator seed for reproducibility.

% Neural Network of Agent
statePath = [
    featureInputLayer(observationNum,'Normalization','none','Name','observation')
    fullyConnectedLayer(400,'Name','CriticStateFC1')
    reluLayer('Name', 'CriticRelu1')
    fullyConnectedLayer(300,'Name','CriticStateFC2')];
actionPath = [
    featureInputLayer(1,'Normalization','none','Name','action')
    fullyConnectedLayer(300,'Name','CriticActionFC1','BiasLearnRateFactor',0)];
commonPath = [
    additionLayer(2,'Name','add')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(1,'Name','CriticOutput')];

criticNetwork = layerGraph();
criticNetwork = addLayers(criticNetwork,statePath);
criticNetwork = addLayers(criticNetwork,actionPath);
criticNetwork = addLayers(criticNetwork,commonPath);
    
criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');
criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');
criticOptions = rlRepresentationOptions('LearnRate',2e-03,'GradientThreshold',1);
% Critic
critic = rlQValueRepresentation(criticNetwork,observationInfo,actionInfo,'Observation',{'observation'},'Action',{'action'},criticOptions);
% Actor
actorNetwork = [
    featureInputLayer(observationNum,'Normalization','none','Name','observation')
    fullyConnectedLayer(400,'Name','ActorFC1')
    reluLayer('Name','ActorRelu1')
    fullyConnectedLayer(300,'Name','ActorFC2')
    reluLayer('Name','ActorRelu2')
    fullyConnectedLayer(1,'Name','ActorFC3')
    tanhLayer('Name','ActorTanh')
    scalingLayer('Name','ActorScaling','Scale',max(actionInfo.UpperLimit))];

actorOpts = rlRepresentationOptions('LearnRate',1e-04,'GradientThreshold',1);

actor = rlDeterministicActorRepresentation(actorNetwork,observationInfo,actionInfo,'Observation',{'observation'},'Action',{'ActorScaling'},actorOpts);

% Agent
Ts = str2num(get_param(mdl, 'FixedStep')) % Sample Time des Simulink Modells
agentOptions = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'TargetSmoothFactor',1e-1,...
    'ExperienceBufferLength',1e6 ,...
    'DiscountFactor',0.99,...
    'MiniBatchSize',256);
agentOptions.NoiseOptions.Variance = 0.6;
agentOptions.NoiseOptions.VarianceDecayRate = 1e-5
% Creating the Agent
agent_DDPG = rlDDPGAgent(actor,critic,agentOptions)

After I trained the agent, I have wanted to realize the trained agent in the real world. I have read that I have to use a matlab function instead of my RL Agent in Simulink here. So I generated with 'generatePolicyFunction' a function evaluatePolicy.m, that is structured liked that:

function action1 = evaluatePolicy(observation1)
%#codegen

% Reinforcement Learning Toolbox
% Generated on: 10-Dec-2021 12:48:51

action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
	policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
observation1 = reshape(observation1,[1 1 4]);
action1 = predict(policy,observation1);
end

My Simulink looks like that:

An the MATLAB function is structured like that:

function y = fcn(u)
    ausgabe = evaluatePolicy(u)
    y = ausgabe
end

If I try 'Monitor and Tune', I get the following error code:

Invalid value set for 'DLTargetLibrary' for deep learning code generation. The allowed values are: 'MKL-DNN' , 'ARM Compute'.

Function 'evaluatePolicy.m' (#49.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.

Function 'evaluatePolicy.m' (#49.411.417), line 16, column 19:
"policy"
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'evaluatePolicy.m' (#49.140.167), line 7, column 11:
"localEvaluate(observation1)"
Component:MATLAB Function | Category:Coder error
Function call failed.

Function 'MATLAB Function' (#36.34.51), line 2, column 15:
"evaluatePolicy(u)"
Component:MATLAB Function | Category:Coder error
Undefined function or variable 'ausgabe'. The first assignment to a local variable determines its class.

Function 'MATLAB Function' (#36.60.67), line 3, column 9:
"ausgabe"
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'Real_Model/MATLAB Function'

In my opinion this error doesn't make any sense, because I have set the value for DLTargetLibrary at 'MKL-DNN' as you can see here:

I would really appreciate it if somebody can help me with that issue.

[Filip Poloczek]

I have also followed the instruction here to build the Intel MKL-DNN library. Unfortunately I get the same error code.

Answer 1

[Sayan Saha]

Hi Filip,

Can you check if you have selected "MKL-DNN" as the Deep Learning Target Libray under "Interface" in "Code Generation" pane? See below image for reference:

The option that you have selected currently is only for the simulation mode. For code-generation the target library needs to be set too as mentioned above and these settings for simulation and code-generation are independent of each other.

Thanks,

Sayan

[Filip Poloczek]

Hi Sayan,

I have checked the Settings and as you can see below, it is not possible for me to change 'Target library'. Do you know if I have to install some extra library?

I want to mention that I have activated 'Use Simulink Coder features' and 'Use Embedded Coder features' under 'Advanced Parameters' in Hardware Implementation pan.

Thanks,

Filip

[Filip Poloczek]

I solved the problem. I had to set the Language to C++. Unfortunately my board only support C as language.

[Sayan Saha]

Hi Filip,

Yes to generate code for "MKLDNN" or "ARM-Compute" target libraries, target language should be set to C++ as those libraries only support C++ language. In 21a release we added support to generate generic C/C++ code for deep learning networks that does not depend on any third party libraries. You can find a video showcasing that workflow in Simulink here: https://www.mathworks.com/videos/generate-generic-c-c-code-for-deep-learning-networks-in-simulink-1622706081351.html With this feature you can generate generic C/C++ code that should be compilable by any board that has a C/C++ compiler without requiring additional libraries. The performance of the generated code for generic C/C++ is not on par with "MKLDNN" yet but we have done some great performance improvements for the new 22a release.

Hope this helps,

Sayan

[Filip Poloczek]

I watched the workflow you attached in your thread and tried it, but get a new error code:

Code generation for rl.layer.ScalingLayer is not supported for target library 'none'. See documentation for a list of supported layers with each target library.

Function 'evaluatePolicy.m' (#80.403.431), line 16, column 11:
"predict(policy,observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed.

Function 'evaluatePolicy.m' (#80.140.167), line 7, column 11:
"localEvaluate(observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed.

Function 'MATLAB Function' (#84.34.51), line 2, column 15:
"evaluatePolicy(u)"
Component: MATLAB Function | Category: Coder error
Undefined function or variable 'ausgabe'. The first assignment to a local variable determines its class.

Function 'MATLAB Function' (#84.60.67), line 3, column 9:
"ausgabe"
Component: MATLAB Function | Category: Coder error
Errors occurred during parsing of MATLAB function 'MATLAB0/MATLAB Function'
Component: MATLAB Function | Category: Coder error

[Sayan Saha]

Hi Filip,

I'm assuming you tried this in a 21a release. rl.layer.ScalingLayer is not supported for library-free generic C/C++ codegen in 21a release. We added that support in 21b. You can call coder.getDeepLearningLayers(TargetLibrary="none") to get a list of layers supported for library-free generic C/C++ codegen in any MATLAB release. Here is the doc for that function: https://www.mathworks.com/help/coder/ref/coder.getdeeplearninglayers.html

~Sayan

[Filip Poloczek]

Hi Sayan,

I would like to use Matlab R2021a so I deleted the ScalingLayer and trained my agent from beginning. In the simulation my agent worked well, so I wanted to use it now in external mode. But this time I get the following error code:

Starting build procedure for: Real_Model
Maximum variable size allowed by the program is exceeded. Function 'evaluatePolicy.m' (#92.397.426), line 16, column 18: "predict(policy, observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed. Function 'evaluatePolicy.m' (#92.140.167), line 7, column 11: "localEvaluate(observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed. Function 'MATLAB Function' (#67.35.52), line 2, column 15: "evaluatePolicy(u)"
Component: MATLAB Function | Category: Coder error
Undefined function or variable 'ausgabe'. The first assignment to a local variable determines its class. Function 'MATLAB Function' (#67.61.68), line 3, column 9: "ausgabe"
Component: MATLAB Function | Category: Coder error
Errors occurred during parsing of MATLAB function 'Real_Model/MATLAB Function'
Component: MATLAB Function | Category: Coder error

I couldn't find any answer how I can solve this problem. Do you now maybe a way to solve that problem?

Sincerely,

Filip

[Sayan Saha]

Hi Filip,

From the error message it seems like the RL model might be too large to fit in the memory available in the board. What is the model architecture? Can you try deploying a very simple model to check if that generates code and deploys to the board correctly?

~Sayan