The Spatial Contact Force block is designed to support custom friction and normal force laws. You set this up in the dialog by choosing Normal Force->Method->Provided by Input and/or Frictional Force->Method->Provided by Input, which expose input ports on the block. The normal force input expects a scalar normal force magnitude signal. The frictional force input expects a 2x1 friction force vector signal (x and y components of friction with respect to the contact frame). So taken together, the normal and frictional force inputs fully define the total 3D contact force. The outputs you can sense from the Spatial Contact Force block should likely be sufficient for creating custom normal and frictional forces. A common approach is to feed the outputs of the Spatial Contact Force block into a MATLAB Function block which computes the forces, and then feed those forces back into the Spatial Contact Force block. For instance, simple friction forces like Coulomb friction are usually based on tangential velocity (i.e., sliding velocity) and normal force, both of which can be sensed by the block; simple normal forces are usually based on penetration depth and penetration velocity ("Relative Normal Velocity"), both of which can also be sensed by the block.
For reference, this doc page attempts to illustrate the contact plane and contact frame.
