Apparently the technique is to overload your own function in place of mse (or one of the other 3 standard functions). It sounds like you are doing this correctly, "C", but you comment:
"The code within this m file is hard to understand. I believe it should return the value of the cost function, and it should return the gradient at a given point. Can someone tell me where these two things are within this m file? The output arguments, out1 and out2, aren't even described in the function's help text. Am I right in guessing that these two outputs are the cost function value and gradient?"
I don't think you should be trying to modify the function call definition itself. This is all part of the standard boilerplate code for argument passing into / out of the function. It is likely to be safe to ignore this boilerplate code (leave well alone!) and instead work on the definitions described by Amaria in the % commented out code below... What will come out of out1 and out2 will (I think) be defined by the .trainParam and/or .performParam settings you apply. The mean squared error function itself is implemented in this sub-function of mse.m (mymse.m):
function [perfy,n] = performance_y(net,t,y,ew,param)
e = gsubtract(t,y);
e = adjust_error(net,e,ew,param);
[perfy,n] = meansqr(e);
perfy = perfy * (1-param.regularization);
...so I am guessing you need to replace meansqr(e) with your own function? You may also need to modify:
function d = dperf_dy(net,t,y,ew,perf,param)
...to take care of the gradient descent?
Here is an example followed by instructions on the (sub)functions (within mymse.m) that you may want to modify...
copyfile([matlabroot '\toolbox\nnet\nnet\nnperformance\mse.m'], 'mymse.m');
[x,t] = simplefit_dataset;
net = fitnet(10,'trainscg');
net.performFcn = 'mymse';
net = train(net,x,t);
Note: the bits you will probably need to modify start below the BOILERPLATE_END marker around line 518 in version 2012a. The bits above that mainly deal with argument handling and probably need to be left as standard.
Also Amaria has added the following notes:
The best way to create a custom performance function is to use MSE.M and the +MSE package of functions as a template. [i.e. as above] Here are descriptions of the required package functions:
newfcn.m - Same as mse.m
+newfcn/apply.m - The main performance calculation
function perfs = apply(t,y,e,param)
+newfcn/backprop.m - Backprop derivatives
function dy = backprop(t,y,e,param)
+newfcn/forwardprop.m - Forward propagate derivatives
function dperf = forwardprop(dy,t,y,e,param) Return dperf/dwb given
+newfcn/type.m - Indicate that "newfcn" is a performance function.
function t = type
function name = name()
function flag = normalize
function param = parameterInfo
+newfcn/perfwb.m - Regularization performance used to minimize weights
function perf = perfwb(wb,param)
+newfcn/dperf_dwb.m - Regularization derivatives
function dperf = dperf_dwb(wb,param)
net = train(net,x,t,nn7);
y = net(x,nn7);
I hope this helps?