How to crop uneven image (with random black margins/spaces) automatically?

I'm making a code to stitch a set of images. When I transform the second image to fit the first using imwarp(), the output pixels outside the input image boundaries show as a consistent black or white color, so I get the stitched image which looks like this. When I have to stitch another image to this new panorama it causes a lot of problems.

Is there any way to crop this black outer portion to the axis I have shown automatically? By automatically I mean can I write a code that'll work for any image with such black outer regions.

imclearborder() just gives me an all black image.

I did find a similiar question here - http://www.mathworks.com/matlabcentral/answers/127836-how-to-crop-out-of-black-background

But the answers makes no sense to me.

[Image Analyst]

Can we always assume that the image on the left will be the smaller one and aligned with the axes?

[Sarita8_8]

The image on the right is warped and then stitched, so yes we can assume the one on the left is always aligned. But I don't think it will always be the smaller one...

Thanks!