$$\begin{bmatrix}1 &1 &6\\4& 3& 2\\5 &2& 2\\5& 3& 4\\4& 2& 4\end{bmatrix}\begin{bmatrix}4\\5\\6\end{bmatrix} = \begin{bmatrix}3\\5\\4\\3\\2\end{bmatrix}. $$ I am not getting that how come this result is possible ?
[Editor's comment #1: The question makes sense, but the asker forgot to explain their notation - possibly because they have not been exposed to any alternatives (happens regrettably often when programmers and/or telecommunication majors are introduced to finite fields). Below please find an elaboration of my educated guess, JL.]
Here the notation is using a common way of writing polynomials with binary coefficients as integers. We are working over the field $GF(2^3)$ aka $\Bbb{F}_8$ defined as $\Bbb{F}_2[\alpha]$, where $\alpha$ is a zero of an irreducible cubic. We then compactly represent an arbitrary element $$ z=a_0+a_1\alpha+a_2\alpha^2\in GF(8), \ a_0,a_1,a_2\in GF(2), $$ as the sequence of bits $$ z=a_2a_1a_0 $$ that is then (internally to the computer program) stored as the integer $$ i(z)=(a_2a_1a_0)_2 $$ in base two.
For example, the element $\alpha^2+\alpha$ is converted to $6$, because $$ \alpha^2+\alpha=1\cdot\alpha^2+1\cdot\alpha^1+0\cdot\alpha^0=110_2=6. $$
[Editor's comment #2: A popular choice is that $\alpha$ is a zero of the polynomial $x^3+x+1$. In other words, we have the equation $$ \alpha^3+\alpha+1=0. $$ Unless I made a mistake, the first component of the matrix product above matches with this minimal polynomial in the sense that the calculation $$ 1\cdot4+1\cdot5+6\cdot6=1\cdot\alpha^2+1\cdot(1+\alpha^2)+(\alpha+\alpha^2)^2=\alpha+1=3,$$ is true when $\alpha^3+\alpha+1=0$, JL.]
$\begin{bmatrix} 1&1&6\\4&3&2\\5&2&2\end{bmatrix}$
for example outputs $\begin{bmatrix} 1&1&6\4&3&2\5&2&2\end{bmatrix}$ – JMoravitz Nov 11 '16 at 05:16