This is the contrapositive of what is sometimes referred to as Gauss' Lemma. The lemma says that if $f(x)$ factors in $F[x]$, then $f(x)$ factors in $R[x]$. Thus if $f(x)$ is irreducible in $R[x]$, then $f(x)$ is irreducible in $F[x]$. For the sake of completeness, we give a short (almost complete) proof. The following lemmas will be required and should be proven:
Lemma 1. If $R$ is a UFD and $f, g \in R[x]$, then $c(f \cdot g) = c(f) \cdot c(g)$ where $c$ is the content.
Lemma 2. In a UFD, if $N$ divides $z_1 \cdot z_2$, then $N = n_1 \cdot n_2$ for some $n_1 \mid z_1$ and $n_2 \mid z_2$.
Suppose $f(x) \in F[x]$ factors as $f(x)=g(x) \cdot h(x)$ where $g(x),h(x) \in F[x]$. The coefficients of $g(x)$ and $h(x)$ are of the form $a/b$ where $a, b \in R$ and $b \ne 0$. Multiply through by $N = \text{lcm}(\text{all denominators})^2$ to obtain
$$
N \cdot f(x) = k(x) \cdot l(x)
$$
where $k(x), l(x) \in R[x]$.By Lemma 1, taking the content of each side gives $N \cdot c(f) = c(k) \cdot c(l)$. Then $N$ divides $c(k) \cdot c(l)$, so by Lemma 2, $N = n_1 \cdot n_2$ where $n_1 \mid c(k)$ and $n_2 \mid c(l)$. Hence
\begin{align}
n_1 \cdot n_2 \cdot f(x) & = k(x) \cdot l(x) \\
& = c(k) \cdot \tilde{k}(x) \cdot c(l) \cdot \tilde{l}(x) \\
& = n_1 \cdot d_1 \cdot \tilde{k}(x) \cdot n_2 \cdot d_2 \cdot \tilde{l}(x),
\end{align}
where $\tilde{k}, \tilde{l} \in R[x]$ are primitive and $n_1 d_1 = c(k)$, $n_2 d_2 = c(l)$. Canceling $n_1$ and $n_2$ gives
$$
f(x) = \underbrace{d_1 \cdot \tilde{k}(x)}_{\text{in $R[x]$}} \cdot \underbrace{d_2 \cdot \tilde{l}(x)}_{\text{in $R[x]$}} \in R[x].
$$
