Everything above the horizontal line is written in the setting of Hartshorne chapter I section 1.
To compute $I(Y)$, we need to write down all the relations between $x,y,z$. Clearly we have the relations $x^5=z^3$, $x^4=y^3$, and $y^5=z^4$, and by some clever considerations of various products of $x,y,z$ we see that we also get some additional relations: $xz=y^2$, $x^3=yz$, and $x^2y=z^2$. It turns out that by elimination theory (or a computation demonstrating that any element $f\in I(Y)$ can be written as $p(x)+yq(x)+y^2r(x)+\lambda z+ s(x,y,z)$ for polynomials $p,q,r$ and $s\in I(Y)$) these relations are enough to generate $I(Y)$.
To show $Y=V(I(Y))$ which implies $Y$ is closed, we first note that $Y\subset V(I(Y))$ is automatic, so we just need the reverse containment. If any point $(a,b,c)\in V(I(Y))$ has any of $a$, $b$, or $c$ equal to zero and is in $V(I(Y))$, then $a=b=c=0$ by considering our equations $x^5=z^3$, $x^4=y^3$, and $y^5=z^4$ in $I(Y)$. For points which have all three coordinates nonzero, we claim $(a,b,c)$ is the image of $\frac{b}{a}\in \Bbb A^1$ under the map $\Bbb A^1\to\Bbb A^3$. In order to verify this, we need to show that $(\frac{b}{a})^3=a$, $(\frac{b}{a})^4=b$, and $(\frac{b}{a})^5=c$. The first two are straightforwards applications of our relations: $(\frac{b}{a})^3=a$ and $(\frac{b}{a})^4=b$ are both equivalent to $b^3=a^4$. The third requires an extra step. As $ac=b^2$, $b^5=a^5c=a^4b^2$, which is equivalent to $b^3=a^4$, a relation we already know is satisfied. So we've shown that every point in $V(I(Y))$ is in $Y$, or that $Y=V(I(Y))$ and thus $Y$ is closed.
Your idea to show that $I(Y)$ is prime is exactly correct: compute the quotient and show it's the integral domain $k[t^3,t^4,t^5]$. Once you've done this, you have that $Y$ is a closed subset and $I(Y)$ is prime, so $Y$ is in fact closed and irreducible and of dimension 1, so $I(Y)$ is of height 2.
To finish the problem, we'll spend a little time playing around with a nonstandard grading on $A$. Make $A$ a graded algebra over $k$ by declaring the degree of $x$ to be $3$, the degree of $y$ to be $4$, and the degree of $z$ to be $5$. We see that $I(Y)$ is a homogeneous ideal under this grading by inspecting its generating set above.
Now suppose we had two generators for our ideal: this would mean we have a surjection $A^2\to I(Y)\to 0$. Tensoring with $A/(x,y,z)$, we would have that $k^2\to I(Y)/(x,y,z)I$ is again a surjection of $k$-vector spaces, as tensor products are right-exact. This would imply that $I(Y)/(x,y,z)I(Y)$ is of dimension at most two as a $k$-vector space. Next, note that $(x,y,z)I(Y)$ is a graded submodule of the graded module $I(Y)$: this means the quotient $I(Y)/(x,y,z)I(Y)$ is also graded, and as it is of dimension two as a $k$-vector space, it can have at most two nonzero graded pieces. Since graded maps of graded modules can be analyzed on each graded piece, we see that the quotient map $I(Y)\to I(Y)/(x,y,z)I(Y)$ must send all but possibly two graded pieces of $I(Y)$ to zero - in particular, there can be at most two integers $d$ so that $((x,y,z)I(Y))_d=0$ but $I(Y)_d\neq 0$.
Let us look at the terms of minimal degree in $I(Y)$: we have $\deg(xz-y^2)=8$, $\deg(x^3-yz)=9$, and $\deg(x^2y-z^2)=10$, and there are no nonzero elements $f\in I(Y)$ with $\deg(f)<8$ by our computation of the generating set at the start of the post. So we have three graded degrees where $I(Y)_d\neq 0$ but $((x,y,z)I(Y))_d=0$, contradicting our work from the previous paragraph. Thus $I(Y)$ cannot be generated by two elements.
To deal with the questions you've raised during your attempted solution (which are good questions!), we'll need more technology from later in the book. In particular, we need to know what a morphism is. This is introduced in section 3 of chapter I, and the important results are that every morphism of varieties is continuous in the Zariski topology and that morphisms of affine varieties uniquely correspond to morphisms of their coordinate rings.
This provides you another proof of $I(Y)$ prime if you know a few more facts about irreducibility. In particular, the image of an irreducible subset under a continuous mapping is irreducible, so $Y$ is irreducible, and the closure of an irreducible subspace is irreducible, so $\overline{Y}$ is irreducible and thus $I(Y)$ is prime.
Your guess that every map of varieties over $k$ which has source $\Bbb A^1_k$ is closed is correct, though the most straightforwards proof also requires the notion of a finite morphism, which is introduced in Hartshorne chapter II section 3. The proof is that any morphism $\Bbb A_k^1\to X$ of affine varieties over $k$ corresponds to a map of coordinate algebras $k[X]\to k[t]$, and either $k[t]$ is a finite $k[X]$-module if any element of $k[X]$ is mapped to a polynomial of positive degree $d$ in $k[t]$ (because then we get a basis $1,t,t^2,\cdots,t^{d-1}$ as a module) or the mapping $\Bbb A^1_k\to X$ is constant if no element of $k[X]$ maps to a polynomial of positive degree in $k[t]$. If the map is constant, the conclusion is obvious, and otherwise, finite maps are closed.
