After some thought, this just follows from a little Hodge theory, and actually the argument that I will give from now on is quite familiar to symplectic geometers.
First notice that we can assume WLOG that $\omega_t$ vanishes near the boundary (by subtracting $\omega_0$ from $\omega_t$). Now, you can take the double of your manifold, and by our assumption, the family $\omega_t$ extends to the double. Thus we can use the corresponding argument applicable to closed manifolds. The argument is as follows:
First choose any Riemannian metric on your closed manifold so that you obtain a Hodge structure. We have assumed that the family $\omega_t$ is exact, which implies it lies on the orthogonal complement of the kernel of Hodge Laplacian $\Delta$. Thus we obtain a smooth family of $d$-forms $\Delta^{-1}(\omega_t)$, and you can now see that $\delta\Delta^{-1}(\omega_t)$ is a desired form (where $\delta$ is the codifferential):
By definition, $\omega_t=\Delta(\Delta^{-1}(\omega_t))=d\delta\Delta^{-1}(\omega_t)+\delta d\Delta^{-1}(\omega_t)$ holds, meaning it suffices to show that $\delta d\Delta^{-1}(\omega_t)$ always vanishes. Indeed,
$\left<\delta d\Delta^{-1}(\omega_t),\delta d\Delta^{-1}(\omega_t)\right>=\left<\omega_t-d\delta \Delta^{-1}(\omega_t),\delta d\Delta^{-1}(\omega_t)\right>=\left<d\omega_t-d^2\delta \Delta^{-1}(\omega_t),d\Delta^{-1}(\omega_t)\right>=\left<0,\delta d\Delta^{-1}(\omega_t)\right>=0$.
Edit. First let me state some well-known facts about the Hodge Laplacian $\Delta$. You may find proofs in introductory textbooks about the Hodge theory.
The Hodge Laplacian is defined as $\Delta=d\delta+\delta d$, which allows a local expression in terms of standard basis $\left\{e_1\wedge \cdots \wedge e_k\right\}$. This is a self-adjoint elliptic operator. Its kernel is the space of harmonic forms, which is finite-dimensional, and $\Delta$ is invertible (in sense of continuus operators) on the orthogonal complement of $\mathrm{ker}\Delta$, which is called Hodge theorem.
Now $\Delta^{-1}\omega_t$ is smooth as $\Delta^{-1}$ is a continuous linear operator. Of course the principle that the OP mentioned in the comment plays a role under the water, but it is a bit hard to explicate where it was used. If you know the elliptic regularity, then it would be automatic as $\Delta$ is elliptic. Next, $\Delta^{-1}$ is local as $\Delta$ is.