Here is a proof of the lower bound. Let us consider the chosen points in each row as a subset of $S=\{1,\dots,n\}$, so we are selecting $n$ subsets $A_1,\dots,A_n$ of $S$, each of size $k$, so that each element of $S$ is in $k$ of the $A_i$'s. Avoiding rectangles means that $|A_i\cap A_j|\leq 1$ if $i\neq j$.
Now consider the $k$ different $A_i$ such that $1\in A_i$. All of the elements of these sets must be distinct except for $1$, so each one has $k-1$ elements that are not in any of the others. This gives $k(k-1)+1=k^2-k+1$ different elements of $S$ (the $k-1$ unique elements of the $k$ sets, plus the element $1$). Thus $n\geq k^2-k+1$.
The claim that $n=k^2-k+1$ is possible in general is incorrect, however. Indeed, observe that when $n=k^2-k+1$, the argument above shows that for every $j\in S\setminus\{1\}$, there is exactly one $A_i$ which contains both $1$ and $j$. Similarly, every two-element subset of $S$ is contained in exactly one $A_i$. That is, in terms of the original grid formulation, for every pair of columns there is exactly one row with a point in both columns. Dually, for every pair of rows there must be exactly one column with a point in both rows. This means that such a configuration is exactly the incidence relation of a finite projective plane, with the rows representing points and the columns representing lines. (Thanks to antkam for suggesting this connection in a comment!)
Given a finite projective plane with $n=k^2-k+1$ points, the number $k-1$ is known as the order of the projective plane. Not every number can be the order of a projective plane; for instance, it is known that there is no projective plane of order $6$, so the statement you quoted is wrong for $k=7$. It is known that projective planes exist for any prime power order and it is conjectured (but not known!) that these are the only possible orders.
Here is how you can construct an example when $k-1$ is a prime power. The key fact we use is that then there exists a field $F$ with $k-1$ elements. Let us then think of our $(k^2-k+1)$-element set $S$ as $$\{0\}\cup((F\cup\{\infty\})\times F).$$ Our first $k$ sets $A_1,\dots,A_k$ are just $\{0\}\cup\{x\}\times F$ for each $x\in F\cup\{\infty\}$. The remaining sets will then all be subsets of $(F\cup\{\infty\})\times F$ which contain one element from each $\{x\}\times F$, so they are just (graphs of) functions $F\cup\{\infty\}\to F$. So, we need to pick $n-k=k^2-2k+1=(k-1)^2$ functions $F\cup\{\infty\}\to F$ such that every possible ordered pair is in exactly $k-1$ of the functions and no two functions have more than one point in common. To do this, for each $a,b\in F$ define $f_{a,b}(x)=ax+b$ for $x\in F$ and $f_{a,b}(\infty)=a$. These functions $f_{a,b}$ have the desired properties because $F$ is a field (so that any two points of $F^2$ with different first coordinates can be interpolated by a unique linear function; similarly given any point in $F^2$ and a choice of slope there is a unique linear function through that line with that slope).
(This construction is known as the projective plane over $F$ and can be described in many other ways. One particularly elegant way is to say that the rows of the grid correspond to 1-dimensional subspaces of the vector space $F^3$, the columns correspond to 2-dimensional subspaces of $F^3$, and you draw a point whenever the 1-dimensional subspace corresponding to a row is contained in the 2-dimensional subspace corresponding to a column.)
