Problem 447. swap sign sum & multiply castles

It is an easy problem, if you know the answer.
Given a square matrix of NxN ordinary numbers.
Initially place N identical indistinguishable castles or rooks (chess pieces) on the main diagonal.
Then keep swapping any two rows or columns to exhaustively enumerate all possible unique patterns of castle formation.
Not a single castle in any of these formations should be under threat of any other castle,
only one castle watches over an otherwise empty row and column.
For each pattern, find the product of all numbers covered by the castles.
If this pattern was obtained after even number (0,2,4,...) of swaps,
then add the product to an initially empty accumulator,
otherwise subtract the product from the accumulator.
Give the final expected value of the accumulator,
does not matter whether by hook or by crook,
but please give a general solution,
the test suite may be modified soon.