Why am I getting incorrect scores from AlexNet?

I have been using MATLAB transfer learning with AlexNet to train a neural network on my own set of images. I have been using ~350 images for training, on three different categories of object, with a separate set of ~6500 images for testing afterwards, roughly an even mix of each category. The images are individually cropped down to the specified area and the rescaled to the required size (227x227 px). At the end of the network training, it typically reports an accuracy of >95%.

However, regardless of the images which are given to the output network, the scores are always roughly the same; regardless of whether the images are from the training set or from the images used to train the network itself (reporting about 0.80, 0.19, 0.01 per image).

The code I used for the transfer learning is shown here:

alexLayers = net.Layers(1:end-3);
numClasses = numel(categories(trainingImages.Labels));
layers = [
    alexLayers
    fullyConnectedLayer(numClasses,'WeightLearnRateFactor',20,'BiasLearnRateFactor',20)
    softmaxLayer
    classificationLayer];
miniBatchSize = 10;
numIterationsPerEpoch = floor(numel(trainingImages.Labels)/miniBatchSize);
options = trainingOptions('sgdm',...
    'MiniBatchSize',miniBatchSize,...
    'MaxEpochs',4,...
    'InitialLearnRate',1e-4,...
    'Verbose',true,...
    'VerboseFrequency',1,...
    'ExecutionEnvironment','gpu');
newNet = trainNetwork(trainingImages,layers,options);

A plot of the output scores is shown here:

Why is this happening?