Controlling state transition using transition probability matrix in Markov chain?

Hello all, I am working on Markov chain and in that I would like control the state transition using transition probability matrix (TPM).

The TPM in my case is 6x6 and is given as TPM = [0.6,0.4,0,0,0,0;0.3,0.4,0.3,0,0,0;0,0.3,0.4,0.3,0,0;0,0,0.3,0.4,0.3,0;0,0,0,0.3,0.4,0.3;0,0,0,0,0.4,0.6]. For clarity, let us denote the state at time t by the row and state at time t+1 by the column of TPM.

I understood that if my current state is 2 and if transition probability is 0.4 then from TPM my next state will also be 2.

But my query is how this condition of transition probability of value 0.4 is generated ?

Any help in this regard will be highly appreciated.

Answer 1

[Torsten]

I understood that if my current state is 2 and if transition probability is 0.4 then from TPM my next state will also be 2.

That's wrong. If you are in state 2, the probability to change in state 1 is 0.3, to remain in state 2 is 0.4 and to change in state 3 is 0.3.

But my query is how this condition of transition probability of value 0.4 is generated ?

The transition probabilities are not generated, but they are fixed and calculated in advance depending on what your Markov chain tries to model.

[chaaru datta]

Thank you so much sir for your response.

1) I think meaning of statement 1 and statement 2 is same, where statement 1 is "I understood that if my current state is 2 and if transition probability is 0.4 then from TPM my next state will also be 2. " and statement 2 is " If you are in state 2, the probability to change in state 1 is 0.3, to remain in state 2 is 0.4 and to change in state 3 is 0.3. "

2) If the Markov chain in my case is as shown in the image attached then can you please tell, if my current state is 2 then what will be my next state ?

[Torsten]

ad 1)

Your statement

I understood that if my current state is 2 and if transition probability is 0.4 then from TPM my next state will also be 2. "

is wrong. The transition probablities are given and fixed. So "if transition probability is 0.4 then from TPM my next state will also be 2." makes no sense. Given you are in state 2, your next state will be 2 if a random experiment with three outcomes (1,2 or 3) where the probability of outcome 1 is 0.3, the probability of outcome 2 is 0.4 and the probability of outcome 3 is 0.3, gives outcome 2 as result.

ad 2)

Take a dice with ten sides and roll it.

If the dice shows 1,2 or 3, your next state will be S1. If it shows 4,5,6 or 7, your state will remain S2. If it shows 8,9 or 10, your next state will be S3.

[chaaru datta]

Ok sir. It means as per the given Markov chain, if current state is 2 then next state is again 2 with probability 0.4, next state is 1 with probability 0.3 or next state is 3 with probability 0.3.

Is my understanding correct sir ?

[Torsten]

Yes, that's the correct interpretation.

[chaaru datta]

Ok Thank you sir. But that again creates little confusion to me about Q-learning with Markov chain.

[chaaru datta]

My query is I am not getting about what value of next state should we put in Bellman equation (used in q-learning) if current state is , where Bellman equation is: