This is more than anything a complement/continuation of hm2020's answer, especially the second part denoted by Answer in hm2020's post.
The point of how to think of $\mathrm{GL}(V)$ for infinite-dimensional $V$ is kind of subtle, because already the question of how to think of a "usual" $k$-vector space $V$ as some kind of $k$-scheme itself becomes subtle.
The two flavors of "geometric" vector spaces
There are two common ways of associating a $k$-scheme to a $k$-vector space $V$:
- $X_V = \mathrm{Spec}(\mathrm{Sym}^*_k(V^\vee))$
- $Y_V = \mathrm{Spec}(\mathrm{Sym}^*_k(V))$
For instance, (if I am not mistaken,) Hartshorne and the EGA use 1, while Fulton's Intersection Theory uses 2 (all of them in the more general context of vector bundles, i.e. over a base scheme that isn't necessarily $\mathrm{Spec}(k)$). In either case, once the convention is understood, the "geometric vector space" would usually just be denoted $V$. But we are trying to draw attention to the distinction between them, hence the awkward notion.
The advantage of 1 is that, for a commutative $k$-algebra $A$, the $A$-points of the "geometric" vector space associated to $V$ are $$X_V(A) = \mathrm{Hom}_k(V^\vee, A).$$When $V$ is finite-dimensional, this is further equal to $X_V(A) = V\otimes_k A,$ which is surely what one wants to do from a "functor of points" perspective!
The disadvantage of 1 is that $X_V$ only knows about the $k$-linear dual $V^\vee$, and not of the vector space $V$ itself. Indeed, if you define the notion of a linear map of "geometric" $k$-vector space schemes correctly (doable but annoying), you will discover that $$\mathrm{Hom}_{\mathrm{Vect}(\mathrm{Sch}_k)}(X_V, X_{V'})\simeq \mathrm{Hom}_k(V'^\vee, V^\vee).$$When $V$ and $V'$ are finite-dimensional, this is equal to $\mathrm{Hom}_k(V, V')$, but not otherwise. Hence for non-finite-dimensional vector spaces, the embedding $V\mapsto X_V$ of "algebraic" into "geometric" vector spaces is not fully faithful.
This is where option 2 above is better. With it, we will have
$$\mathrm{Hom}_{\mathrm{Vect}(\mathrm{Sch}_k)}(Y_V, Y_{V'})\simeq \mathrm{Hom}_k(V', V)$$
which, while contravariant, is at least fully faithful. But now "algberaic = usual" $k$-vector spaces, and "geometric" $k$-vector space schemes are anti-equivalent, not equivalent categories as we might desire!
Also, the set of $A$-points of the $k$-scheme associated to $V$ under 2 is
$$
Y_V(A) = \mathrm{Hom}_k(V, A)
$$
which will never be (canonically) equal to $V\otimes_k A$. For finite-dimensional vector spaces $V$, you will get it to be $V^\vee\otimes_k A$, but for infinite-dimensional ones not even that.
Back to the example in question & take-away
In the example from the original question, one might imagine that the $\mathbf C$-vector space $V=\mathbf C(t)$ might be the set of $\mathbf C$-points of some natural associated $\mathbf C$-scheme. But as the above discussion tried to demonstrate, it is not clear how to do so. Our best bet would be to use 2 and get the $\mathbf C$-scheme $Y_{\mathbf C(t)}$ with $\mathbf C$-points $$Y_{\mathbf C(t)}(\mathbf C) = \mathrm{Hom}_{\mathbf C}(\mathbf C(t), \mathbf C)= \mathbf C(t)^\vee.$$
Thus I think that when people make assertions such as "The category of representations of a group scheme $G$ is equivalent to the category of comodules of the Hopf algebra $\mathcal O(G)$", they must probably have either
- only have the finite-dimensional case in mind, in which case there are no issues, or
- allow for arbitrary dimensionality of representations, but suppress the fact that there is really an anti-equivalence, with the arrows in the "geometric" representations going the other way than in the comodules.
In particular, in the regime 2 (which is, so far as I understand, the only one possible if we want to speak about infinite-dimensional reps), a $G$-rep will be equivalently
- A $k$-vector space $V$, together with a $G$-action
$G\times_{\mathrm{Spec}(k)}Y_V\to Y_V$ on the associated "geometric"
vector space $Y_G$.
- A $\mathcal O(G)$-comodule structure on $V$.
In the former description, you can try to get some mileage by thinking of $Y_V$ as cut out by polynomial equations inside some very big affine space, if you so wish. But on the level of $k$-points, you will get a map $$G(k)\times \mathrm{Hom}_k(V, k)\to \mathrm{Hom}_k(V, k)$$or, if you prefer to write it in terms of dual vector spaces,$$G(k)\times V^\vee\to V^\vee.$$
If you wish, you can try to find a vector space $V$ such that there is a natural identification $V^\vee =\mathbf C(t)$, but I don't know how to formulate an action that might look like the $f(t)\mapsto f(t+a)$ for $a\in k=\mathbf G_a(k)$ that was asked about in the original question.
Edit: The edit to the original post clarified that we are following Milne's "Algebraic Groups" book.
Vector spaces as functors, or, the secret third flavor
For a (usual = algebraic) $k$-vector space $V$, we thus define the functor $GL_V:\mathrm{CAlg}_R\to\mathrm{Grp}$ by $\mathrm{GL}_V(R) := \mathrm{Aut}_R(V\otimes_k R)$. This is equivalent to adopting a third perspective on how to incarnate a (possibly infinite-dimensional) vector space $V$ as an object of algebraic geometry:
- As the functor $V_\mathfrak{a}:\mathrm{CAlg}_k\to\mathrm{Vect}_k$,given by $V_\mathfrak{a}(R):= V\otimes_k R$.
Then $\mathrm{GL}_V$ is quite literally automorphisms of $V_\mathfrak{a}$, and a map $G\to\mathrm{GL}_V$ in $\mathrm{Fun}(\mathrm{CAlg}_k, \mathrm{Grp})$ is equivalent to a map of functors $G\times V_\mathfrak{a}\to V_\mathfrak{a}$, satisfying all the usual requirements.
But that is a highly functor-of-points perspective, that is for infinite-dimensional vector spaces $V$ increasingly far from varieties. Indeed, $V_\mathfrak{a}$ is not an algebraic variety if $\dim(V) =\infty$. It might or might not be representable by a scheme (I don't know), but if it is, it isn't by the expected usual friendly "geometric vector space" scheme $X_V=\mathrm{Spec}(\mathrm{Sym}^*_k(V^\vee))$.
What is contained in a representation
But with this definition, it isn't too hard to explain (though satisfaction may vary) what extra data is encoded in saying "$V$ is a $G$-rep". Namely, the map $\rho_k:G(k)\times V\to V$ is required to extend functorially to $R$-linear maps
$$\rho_R : G(R)\times (V\otimes_k R)\to V\otimes_k R$$
for all commutative $k$-algebras $R$.
The (key) functoriality requirement means that, for any $k$-algebra map $\varphi:R\to R'$, the corresponding $R'$-linear map
$$
G(R)\times (V\otimes_k R)\xrightarrow{G(\varphi)\times (\mathrm{id}_V\otimes_k \varphi)}G(R')\times (V\otimes_k R')\xrightarrow{\rho_{R'}} V\otimes_k R'
$$
agrees with
$$
G(R)\times (V\otimes_k R)\xrightarrow{\rho_R} V\otimes_k R\xrightarrow{\mathrm{id}_V\otimes_k \varphi} V\otimes_k R'.
$$
The 'counterexample' in question
Let us try to make the counterexample from the original post work in this context! Let us first pursue this naively, leading to a contradiction, and only afterward point out the issue.
We have $k=\mathbf C$ (though this shouldn't matter), $V=\mathbf C(t)$, and $G=\mathbf G_a$. Thus for any $R\in\mathrm{CAlg}_k$ we have
$$
V_\mathfrak{a}(R) = R(t), \quad \mathbf G_a(R) = R
$$
and the action $\rho_R: R\times R(t)\to R(t)$ should presumably be defined as
$$
\rho_R(a, f(t)) = f(t+a)
$$
for a scalar $a\in R$ and rational function $f(t)\in R(t)$. This clearly is a group homomorphism, and the functoriality condition boils down to the manifestly true requirement that wd have for any $k$-algebra map $\varphi: R\to R'$, $a\in R$, and $f(t)\in R(t)$
$$\varphi(f(t+a)) = \varphi(f)(t+\varphi(a)).$$
This would imply that we have a legitimate representation on our hands, hence that it corresponds to a comodule, and consequently that it is rational (every element is contained inside a finite-dimensional subrep), which we know to be violated, as explained in the original question.
So where did we go wrong?
Answer: in asserting that $V_{\mathfrak{a}}(R) = R(t)$. This sure feels right, and would certainly hold with polynomials in place of rational functions, but isn't true here. Indeed, we instead have by definition $V_\mathfrak{a}(R) = \mathbf C(t)\otimes_{\mathbf C}R$, and as is elegantly demonstrated here, this tensor product is instead
$$
\mathbf C(t)\otimes_{\mathbf C}R = (\mathbf C[t]-0)^{-1}R[t].
$$
That is to say, we are only allowed to divide by polynomials with coefficients in the base field $\mathbf C$.
This seriously hinders the above-sketched program for defining the representation $\rho$ by $\rho_R(a, f(t)) = f(t+a)$ for $a\in R$ and $f\in V_{\mathfrak{a}}(R)$. Indeed, if we write this out, we have $f(t)=\frac{p(t)}{q(t)}$ for $p(t)\in R[t]$ and $q(t)\in\mathbf C[t]-0$. An element $a\in R=\mathbf G_a(R)$ would therefore want to act by
$$
f(t)=\frac{p(t)}{q(t)}\mapsto f(t+a) = \frac{p(t+a)}{q(t+a)},
$$
which is naturally a problem on account of the polynomial $q(t+a)$ now having coefficients in $R$, and so it is unclear if dividing by it is even allowed in $\mathbf C(t)\otimes_{\mathbf C}R$.
In short: it is unclear how one would go about extending the $\mathbf C$-point level representation $\mathbf C\times \mathbf C(t)\to\mathbf C(t)$ to a representation on the level of functors.