Check for Run-Time Issues failed on GPU but successed on CPU（cudaEr​rorIllegal​Instructio​n）

When I use GPU coder to convert MATLAB source code, I get an error when I choose GPU to run the MEX file in Check for Run-Time Issues, but it runs successfully if I choose to test on the CPU. The error log is as follows.

Error using TABU_ALNS (line 5655)
Error generated while running CUDA-enabled program: [715,cudaErrorIllegalInstruction] an illegal instruction was encountered

Error in MainScript (line 89)
 [BestSolution_M1, Best_obj] = TABU_ALNS(Order_inf,SN_nu,SN_rand,SBR_REL,R_qi_range,q_B,c_B,C_UNIT,t_ib,Tabu_PARA,ALNS_PARA);

It seems that there is a problem of using illegal instructions, but I don't know how to further locate the error, or what causes the file to run on the CPU but not on the GPU?

All suggestions are appreciated

[Chao Luo]

Hi Shi,

It is hard to tell the issue from this error message. If you want the function TABU_ALNS to run on CPU, you can add coder.inline('never') inside the function and remove ahh coder.gpu.kernelfun and coder.gpu.kernel statememts if there are any.

We can investigate more if you can provide us the reproduction and your system info, like OS and GPU info.

[Shi]

Hi Luo,

Thanks very much for your reply.

In fact, I want all the functions I designed (TABU_ALNS and its sub-functions) to run on GPU, because it takes a lot of time to run on CPU. My MATLAB source code runs well. The problem I have encountered is that when I use GPU coder to convert the code into a MEX file, if I choose check for issues on GPU, it passes the Generating trial code and building MEX stages, and then in the Running test file with MEX stage, MATALB suddenly crashes and exits( as shown in the figure). But if I choose check for issues on CPU, it can pass all stages and generate MEX files, but when I call the generated file in MATLAB, the above error message appears.

I think there may be errors in my code caused by the difference between GPU and CPU operation, but I don’t know how to identify and locate the error. I tried to use Visual Studio for debugging, but it reported an error before the mexfunction.I wonder if the problem is caused by the following operations done before code conversion: adding coder.gpu.kernelfun declarations at the beginning of TABU_ALNS and its subfunctions, declaring all data as single types, using structures as input parameters of TABU_ALNS and declaring them through coder.cstructname.

My OS is Win10 64-bit, the C/C++ compileer is Microsoft Visual C++ 2019, and the GPU information are as follows:

Name: 'NVIDIA RTX A2000'

Index: 1

ComputeCapability: '8.6'

SupportsDouble: 1

GraphicsDriverVersion: '572.83'

DriverModel: 'WDDM'

ToolkitVersion: 11.8000

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152 (49.15 KB)

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 6032850944 (6.03 GB)

CachePolicy: 'balanced'

MultiprocessorCount: 26

ClockRateKHz: 1200000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 1

CanMapHostMemory: 1

DeviceSupported: 1

DeviceAvailable: 1

DeviceSelected: 1

AvailableMemory: NaN

[Chao Luo]

Hi Shi,

We'd be happy to help identify the issue if we have your code. Just from the error and system info, we cannot tell what is wrong.

[Shi]

Hi Luo,

Many Thanks! The following is my entry-point function and the structure of its input parameters. This function also calls other functions designed by myself, such as Tabu_LocalSearch, ALNS_for_M2, etc. However, since they all run normally in MATLAB and the MEX file generated by MATLAB coder, I think it should be able to find the issues by only looking at the entry-point function.

Hope to get your reply!

function [BestSolution_M1,Best_obj] = TABU_ALNS(Order_inf,SN_nu,SN_rand,SBR_REL,R_qi_range,q_B,C_UNIT,t_ib,Tabu_PARA,ALNS_PARA) %#codegen
coder.gpu.kernelfun
coder.cstructname(Tabu_PARA, 'Tabu_PARA_struct');
coder.cstructname(ALNS_PARA, 'ALNS_PARA_struct');
coder.cstructname(SBR_REL, 'SBR_REL_struct');
coder.cstructname(C_UNIT, 'C_UNIT_struct');

tabuLoop = Tabu_PARA.tabuLoop;
tabuTenure = Tabu_PARA.tabuTenure;
tabuNeighbors= Tabu_PARA.tabuNeighbors;
so=Order_inf(:,1);
qo=Order_inf(:,2);
to=Order_inf(:,3);
S_B=SBR_REL.S_B;
B_R=SBR_REL.B_R;
S_R=SBR_REL.S_R;
c_B=C_UNIT.c_B;
B=single(length(q_B));
%% Tabu Search loop to optimize Solution_M1
% Initialize the M1 solution, BUS schedules' availability
InitialSolution_M1 = M1InitialSolutionGeneration(B_R, R_qi_range, q_B, c_B);
BestSolution_M1 =InitialSolution_M1;
Best_obj = single(Inf); % Initialize the best objective value

tabuMoves = single(zeros(3,tabuTenure)); % Initialize the tabuMove list as a matrix
tabuIndex = single(1); % Index to manage the circular tabu list

% Main Tabu Search loop
for iter = 1:tabuLoop
    disp(['Iteration: ', sprintf('%f', iter)]);
    % Generate neighborhood solutions
    [numNeighbors,neighborhoodSolutions,candidateMoves] = Tabu_LocalSearch(BestSolution_M1, B_R, S_R, q_B, R_qi_range, tabuNeighbors, tabuMoves);
    % Evaluate neighborhood solutions in parallel
    for i = 1:numNeighbors
        candidateSolution_M1 = neighborhoodSolutions(:,i);
        % Calculate the fixed operating cost
        M1_obj = sum(candidateSolution_M1 .* c_B);

        % Calculate the second-stage cost using ALNS
        M2_obj = single(0);
        q_B_adjusted = q_B .* candidateSolution_M1;
        currentso = so;
        currentqo = qo;
        currentto = to;
        so_SN=currentso(SN_rand);
        qo_SN=currentqo(SN_rand);
        to_SN=currentto(SN_rand);
        for sn = 1:SN_nu
            Order_inf_SN = [qo_SN(:, sn), so_SN(:, sn), to_SN(:, sn)];
            IniS_M2 = M2InitialSolutionGeneration(Order_inf_SN, SBR_REL, q_B_adjusted, t_ib);
            BestS_Found_M2 = ALNS_for_M2(IniS_M2,Order_inf_SN, SBR_REL, q_B_adjusted, C_UNIT, t_ib);
            BestS_M2_obj = GetM2_obj(BestS_Found_M2,Order_inf_SN,B);
            M2_obj = M2_obj + BestS_M2_obj/ SN_nu;
        end
        % Calculate the total objective value of neighborhood solution
        neighborObj = M1_obj + M2_obj;

        % Update the best solution if the neighbor is better
        if  neighborObj < Best_obj
            BestSolution_M1 = neighborhoodSolutions(:,i);
            Best_obj = neighborObj;
            % Add the best neighbor to the tabu list
            BestMove = candidateMoves(:,i);
            if any(BestMove)
                tabuMoves(:,tabuIndex) = BestMove;
                tabuIndex = mod(tabuIndex, tabuTenure) + 1;
            end % Circular index for tabu list
        end

    end

end

end

[Chao Luo]

Hi Shi,

We cannot identify the issue with only the entry-point function. Can you call customer support to create a case with all the code and reproduction steps? You can ask the customer support to escalate to dev team directly.