Assume $P_\mathbb R$ is the set of all polynomials which have only real coefficients and only real roots. Define $0$ as a polynomial with infinitely many real roots and all other constant polynomials as having no roots, thus no complex roots either.
Say we want to find an operation $\star$ such that $(P_\mathbb R,\star)$ is a group.
Let's start looking with some obvious ones. If we select $\star = +,$ ie. addition of polynomials, we immediately see that $(P_\mathbb R,+)$ is not a group. Take as an example $p_1 = 1+x$ and $p_2 = x^2.$ Then clearly $p_i \in P_\mathbb R$ but $p_1+p_2 = 1+ x+ x^2$ has no real roots. However, construction $(P_\mathbb R,+)$ has an identity element $0$ and every member has an inverse, namely inverse of $p$ is $-p$. It also has associativity. Thus $(P_\mathbb R,+)$ is a groupoid.
If we select $\star = \times$, meaning multiplication of polynomials, then clearly $(P_\mathbb R,\times)$ is closed. If $p_1$ has roots $r_1, r_2 \dots r_n$ and $p_2$ has roots $R_1, R_2 \dots R_k$, then $p_1 p_2$ has roots $r_1, r_2 \dots r_n, R_1, R_2 \dots R_k$. These obviously need not be distinct.
So multiplication seems promising. We also have an identity element, namely constant polynomial $1$. $$1\times p = p \times 1 = p.$$ Multiplication has associativity.
But finding an inverse of $p \in P_\mathbb R$ seems difficult, if not impossible. Lets say we want to invert $1+x$. We need a polynomial $p \in P_\mathbb R$ such that $(1+x)p = 1.$ Solving $p$, we get $p = \frac{1}{x+1},$ which is not a polynomial.
Or is it? We have the Taylor series of $\frac{1}{x+1}$ around $0$ given by $\sum_{k=0}^\infty (-1)^k x^k,$ and, indeed, $$(x+1)\times\sum_{k=0}^\infty (-1)^k x^k = 1$$ as long as $-1<x<1.$ But I have serious doubts that $$\sum_{k=0}^\infty (-1)^k x^k \in P_\mathbb R.$$
Looking at above, it is doubtful $(P_\mathbb R, \times)$ can be a group, no matter how the inverse is defined. If not, we get that $(P_\mathbb R, \times)$ is a monoid.
Is there any way to interpret the inverse of a polynomial in the multiplicative case, that also preserves the group structure? The answer seems to be no, as if we look at the polynomial ring over field of real numbers under multiplication and addition, the only elements with multiplicative inverses are constant polynomials with exception of $0$, which has no multiplicative inverse. As this preserves the properties of multiplication in that ring, multiplication doesn't give a group, even if $0$ is removed.
Does there exists a subring of the polynomial ring under the addition and multiplication defined in the field of real numbers, restricted to some $P^* \subset P_\mathbb R$? In other words, is there a restriction to some subset of polynomials with real coefficients and real roots such that this set forms a group under addition and a monoid under multiplication.