Why does my prediction always show high risk?

Hello everyone.

I want to ask a question regarding my final year project.

I'm currently doing a prediction system using:

1. UCI heart disease dataset.
2. Ensemble method (Subspace Discriminant) trained model from Classifier Learner App.
3. a GUI from app designer.

The system shown no problem when detecting a high parameter for high risk patient data and correctly shows 'Heart Disease Detected' but for a low parameter for a normal patient data it still shows the output as 'Heart Disease Detected'. It is because of the dataset's value comes from patients that already suspect with heart disease or the model used is sensitive. The GUI screenshot is suppose to be a normal patient data.

If you have an insight about this please share with me. Thank you.

I'll share the coding used in the app designer and the GUI with some of the patient attribute data where the target is suppose to be the outcome of this system prediction.

    properties (Access = private)
        modelData = load('HeartModel.mat','trainedModel');
    end
  

    methods (Access = private)
        
        function startupFcn(app)

            % Set DropDown items and ItemsData for numerical values

            % Sex
            app.SexDropDown.Items = {'Female', 'Male'};
            app.SexDropDown.ItemsData = [0, 1];

            % Chest Pain (cp)
            app.ChestPainDropDown.Items = {'Normal','Typical Angina', 'Atypical Angina', 'Non-anginal Pain', 'Asymptomatic'};
            app.ChestPainDropDown.ItemsData = [0, 1, 2, 3, 4];

            % Fasting Blood Sugar (fbs)
            app.FastingBloodSugarDropDown.Items = {'<= 120 mg/dL', '> 120 mg/dL'};
            app.FastingBloodSugarDropDown.ItemsData = [0, 1];

            % Rest ECG (restecg)
            app.RestECGDropDown.Items = {'Normal', 'ST-T Wave Abnormality', 'LV Hypertrophy'};
            app.RestECGDropDown.ItemsData = [0, 1, 2];

            % Slope
            app.SlopeDropDown.Items = {'Upsloping', 'Flat', 'Downsloping'};
            app.SlopeDropDown.ItemsData = [1, 2, 3];

            % Exercise Angina (exang)
            app.ExerciseAnginaDropDown.Items = {'No', 'Yes'};
            app.ExerciseAnginaDropDown.ItemsData = [0, 1];

            % CA
            app.CADropDown.Items = {'0', '1', '2', '3'};
            app.CADropDown.ItemsData = [0, 1, 2, 3];

            % Thal
            app.ThalDropDown.Items = {'Normal', 'Fixed Defect', 'Reversible Defect'};
            app.ThalDropDown.ItemsData = [3, 6, 7];
   
        end
        end

        

                % Button pushed function: PredictButton
        function PredictButtonPushed(app, event)

        % Collect all inputs (convert to double to be safe)
        age = double(app.AgeEditField.Value);
        sex = double(app.SexDropDown.Value);
        cp = double(app.ChestPainDropDown.Value);
        trestbps = double(app.RestingBPEditField.Value);
        chol = double(app.CholesterolEditField.Value);
        fbs = double(app.FastingBloodSugarDropDown.Value);
        restecg = double(app.RestECGDropDown.Value);
        thalach = double(app.MaxHeartRateEditField.Value);
        exang = double(app.ExerciseAnginaDropDown.Value);
        oldpeak = double(app.OldpeakEditField.Value);
        slope = double(app.SlopeDropDown.Value);
        ca = double(app.CADropDown.Value);
        thal = double(app.ThalDropDown.Value);
        

        % Create input table
        T = table(age, sex, cp, trestbps, chol, fbs, restecg, thalach, ...
                           exang, oldpeak, slope, ca, thal, ...
                           'VariableNames', {'age','sex','cp','trestbps','chol','fbs',...
                                             'restecg','thalach','exang','oldpeak',...
                                             'slope','ca','thal'});
        
        %Prediction
        label = app.modelData.trainedModel.predictFcn(T);
        
        % Display result
        if label == 0
            app.ResultLabel.Text = 'No Heart Disease';
            app.ResultLabel.FontColor = [0 1 0];  % Green
        else
            app.ResultLabel.Text = 'Heart Disease Detected';
            app.ResultLabel.FontColor = [1 0 0];  % Red
        end

[dpb]

Everything is tied up in one line

label = app.modelData.trainedModel.predictFcn(T);

You will have to have sufficient normal data as well as unhealthy conditions in the dataset in order to be able to effectively train and then discrimante between the two cases.

We have no way to be able to say anything definitive from just the general description -- one first thing might be to try discriminant analyses and do coditional plots of the measured parameters to see which are those that actually do correlate with the condition to be detected.

[Halimatun Saadiah]

Sure I will try as you suggest. If the dataset contain class imbalance, will it be the cause of what's happening to the system where it will always detected a disease?

[dpb]

The other comment indicates that the model itself apparently isn't the problem, presuming the conclusion drawn there is correct that the model does predict good/bad as expected.

The above would only be of real use if, as was implied initially, that the model itself is always returning one value -- as noted, that other comment indicates the problem is in either the GUI isn't calling the model with the correct inputs/getting the expected results or in the conversion of the correct output to the displayed value if the model itself is being called correctly.

This could be something wrong in a logical expression that is miscoded such that it will always return true or false regardless of the input.

If the model returns a value in the range 0-1, for example, then checking for nonzero alone wouldn't work as expected because even a likelihood value of 1% chance of disease would still test as True.

Answer 1

[Image Analyst]

When you train the model in the Classifier Learner App, are you including in the data a variable that is exactly predicting/matching the response you expect? In other words if you have a response variable of 1 or 0 for diseased or normal, your table of data (observations) should NOT also include that response variable or something almost identical to it or else it will basically just pay attention to that variable as a good 100% reliable predictor and ignore all the other data. And it should also not include any unrelated data like name, date, hospital ID number, etc.

This statement you make "The GUI screenshot is suppose to be a normal patient data." seems to be contrary to this statement "It is because of the dataset's value comes from patients that already suspect with heart disease." If that's right, then perhaps ALL of the suspected cases truly do have heart disease and the model is predicting correctly.

[Halimatun Saadiah]

During the model training process, the target variable is automatically set as a target response. Only the 13 other variables was chosen as input predictor. The table of data shown was from the original dataset where I need to handle some of the missing value in the dataset.

Does the other attributes with only 1 or 0 variable affect the output?

[Image Analyst]

That should be ok I believe.

Have you tried other models? Was the "Ensemble method (Subspace Discriminant) " model the one with the least error?

Again, why is your test set "patients that already suspect with heart disease" and you expect it to report those as normal? It should report them as diseased, not normal.

[Halimatun Saadiah]

Yeah, the ensemble model have a higher accuracy and the least error when I trained the models and I test with other model too which is Tree, and SVM. But the Tree model does not give the accurate prediction while SVM only detected Normal/No disease even with high parameter.

I already check the dataset and it contained a balance disease and normal patient data.

I also test it on the command window for the new data and it give the prediction that I want but not with the GUI.

[dpb]

"... on the command window for the new data and it give the prediction that I want but not with the GUI."

That then says the problem is not in the actual model itself but in your use of it inside the GUI.

You'll have to set a breakpoint and execute the code that evaluates the model and check it first is provided the correct input values and returns the expected result before the fancy GUI display.

Once that is correct for all cases, then it shouldn't be difficult to figure out where your logic/coding error is in converting that to the display.

[Image Analyst]

Good point that dpb makes above. I thought of that too but I didn't see anything obvious when I looked over the code. If I had the .mlapp file I maybe could have debugged it.

Another thing to consider is, since your data have vastly different ranges, from single digit to triple digits, is to standardize all your data to the same range - normalize it. I don't recall off the top of my head but in the app is there a way to normalize the data before training? Or you might try checking the PCA box. Then once you have the model, you'd need to double check whether you need to normalize the test data yourself before passing in to predictFcn, or does the model do it for you automatically/internally and you can pass in unnormalized/original data.

[dpb]

If HS can, as he says, run the model interactively and get the expected results, then the problem has to be that inside the app he's either not passing it the inputs he thinks he is or is not correctly interpreting the results for display if is running the model correctly.

As noted above, he's got to set a breakpoint and verify that he actually is getting the correct inputs/outputs first, then process those to present to the user.

[Halimatun Saadiah]

I appreciate it if you guys could help me debug it. Here I attach the .mlapp file and .mat file (save model classifier) for you.

[dpb]

d=dir('*.mlapp');
l=split(string(evalc(['type ' d.name ])),newline);
%type(d.name)
l(contains(l,'predict','ignorecase',0))

ans = 2×1 string array
    "% Make prediction"
    "        label = app.modelData.trainedModel.predictFcn(T);"

again we don't have the prediction function so can't run the model.

But, to debug, set a breakpoint here and inspect the result; if it is not what you would expect/get from the model at the command line, then you've not given it the right inputs.

out=find(startsWith(strip(l),'label'));
disp(l(out:out+10))

    "        label = app.modelData.trainedModel.predictFcn(T);"
    ""
    "        if label == 0"
    "            app.ResultLabel.Text = "Normal/No Disease Detected";"
    "            app.ResultLabel.FontColor = [0 1 0];"
    "        else "
    "            app.ResultLabel.Text = "Heart Disease Detected";"
    "            app.ResultLabel.FontColor = [1 0 0];"
    "        end"
    ""
    "        end"

And here's where the issue may be -- unless your model only outputs identically integer 0 every case is going to be "Disease".

What is the type of label? It seems a peculiar name for a prediction output; have you by chance turned it into a char() value of a property in which case the condition comparison to integer 0 would not be correct.

You'll just have to look at what, exactly, is going on here...

[Halimatun Saadiah]

I checked it just now, the label type is double and in the file, it clearly shows the class 0 and 1.

I also tested it when assigned the breakpoint in the line and whenever I put a normal value data, the output always shows class 1 (heart disease).

[Halimatun Saadiah]

Thank you so much for helping me. I already find the solution to the problem and now my GUI is working properly! I really appreciate all the help that you guys have given.

[dpb]

"... I already find the solution to the problem"

Glad to be able to point to the way to find and fix your own problems...

It would be most interesting if you would tell/show us what the actual issue was (and might be of value to someone else who happens on to this thread sometime in the future).

[Halimatun Saadiah]

There is no problem with the button callback function. The problem lies in the startupFcn, where all the dropdown item and itemdata didn't match with the GUI. After change dropdown item and itemdata on the GUI same as the coding, it work perfectly. It was my mistake for not clearly check the dropdown data 😅.

[dpb]

Gotcha'. Thanks, that is, indeed, a valuable lesson!