You made a good first effort, so I will help you out.
First, rand produces a uniform distribution, not integers, so you need to adjust your x, y, and z formulas to turn them into integers with equal probabilities:
x = floor(1 + 6*rand(n,1));
Or look at randi as Walter has suggested.
As you have coded it, this condition only covers two of the possibilities:
if x(k)== y(k)|| y(k)==z(k)
But what about if the first value matches the third value? That is not covered by your test, so you should modify it to account for this possibility.
Also, this condition you have coded isn't quite correct:
if x(k) < z(k) && z(k) > y(k)
Plug in some sample values and you will see what I mean. Remember you will need to cover the case where x(k) is less than y(k) and also the case where x(k) is greater than y(k).
As to your general question of how to calculate probabilities, you simply count the number of successes and divide that by the number of trials. E.g., start two counters, one for the first test and one for the second test, at the front of your code before you enter the loops:
Then, inside the first if test do P = P + 1, and inside the second if test do Q = Q + 1;
At the end of your code you can then do this to get the probabilities: