Let $n = 7$.
I try something like this:
$$ n =p = 7 \rightarrow \phi(n) = 6 $$ I chose $d = 17$ and $m = 3$. $$ e = d^{-1} \text{ mod } \phi(n) = 17^{-1} \text{ mod } 6 = 5 $$ $$ 17x \text{ mod } 6 = 1 $$ $$ x = 5 $$ What I should doing further ?
Let $n = 7$.
I try something like this:
$$ n =p = 7 \rightarrow \phi(n) = 6 $$ I chose $d = 17$ and $m = 3$. $$ e = d^{-1} \text{ mod } \phi(n) = 17^{-1} \text{ mod } 6 = 5 $$ $$ 17x \text{ mod } 6 = 1 $$ $$ x = 5 $$ What I should doing further ?
For the Pohlig-Hellman cipher, let's say you want to encrypt $M=3$ with $(e,d,p) = (5,17,7)$ as stated in your question. Then you do:
$$ C = M^e \text{ mod } n = 3^5 \text{ mod } 7 = 5$$
If you want to decrypt, then do:
$$ M = C^d \text{ mod } n = 5^{17} \text{ mod } 7 = 3$$,
As expected.