Fromhttp://en.wikipedia.org/wiki/Otsu%27s_method, the algorithm assumes that the image to be thresholded contains two classes of pixels or bi-modal histogram (e.g. foreground and background) then calculates the optimum threshold separating those two classes so that their combined spread (intra-class variance) is minimal.
MULTITHRESH can accept another argument that decides how many such classes of pixels are needed, so that you can segment the image over multiple levels.
Specifically, look at section III.B for an explanation.
The basic idea is to find the two thresholds that maximize between class variance over the gray levels in the image. A search-based optimization is used (fminsearch) to determine the two thresholds. The best way to understand is by looking at lines 141-155 in multithresh.