Let the matrix have rows $a, b$ and $c, d$.
If the matrix is invertible mod $27$, then no row or column can be a vector multiple of $3$. There are two cases:
Case 1: If $3\mid a$ then neither $b$ or $c$ can be a multiple of $3.$ There are $9$ choices for $a$ and $18$ for each of $b$ and $c$.
For a given choice of $a, b,$ and $c$, we need to exclude any values for $d$ that makes $ad-bc$ divisible by $3$. But $bc$ is not divisible by $3$ while $a$ is. So the congruence $ad\equiv bc \pmod{3}$ has no solutions. Therefore $d$ can be any of the $27$ numbers. We have $9\cdot 18\cdot 18\cdot 27 = 78732$ possibilities for Case 1.
Case 2: If $3\nmid a$, then $b$ and $c$ can be anything. So we have $18$ choices for $a$ and $27$ each for $b$ and $c$. Since $3\nmid a$, the congruence $ax=cd \pmod{3}$ has a unique solution which lifts to $9$ solutions $\pmod{27}.$ These must be excluded, so there are $18$ choices for $d$. We have $18\cdot 27 \cdot 27\cdot18 = 236196$ possibilities for Case 2.
We add the two cases and get $314928$ for the final answer.