Let's consider second-order logic in full semantics. I read in a paper of Jouko Vaanen that such logic relies entirely on informal reasoning, or to put it in other words: the rules of inference cannot be formalised (in contrast to first-order logic, where the rules of inference are formalised).
Please, can somebody explain me in simple words, what does it mean that the rules of inference cannot be formalised?
For what it's worth, I think the Stanford Encyclopedia article on second-order logic is quite good, and this section is especially relevant.
Additionally, the picture I paint of second-order logic is predominantly negative. However, it's worth noting that it has its defenders - see e.g. Shapiro's book.
This answer has been substantially edited in response to the OP's comments below; see the edit history for the original version.
There are several senses in which SOL with the standard semantics (which is what Vaanaanen is referring to) is fundamentally more complicated than FOL. Whether these differences justify the strong term "unformalizable" is of course a subjective issue, but this is what Vaanaanen is referring to (and incidentally I'd agree with him):
First, we have the failure of compactness for SOL. This immediately implies that there is no proof system as we usually understand such which is sound and complete for SOL: naively, we want proofs to only be able to invoke finitely many axioms, and when compactness fails that doesn't work.
One reasonable response to this is that maybe things get nicer when we restrict attention to finite sets of hypotheses. This doesn't work either. First of all we have the computational complexity aspect: while the set of first-order validities $$Val_1:=\{\varphi\in FOL: \emptyset\models\varphi\}$$ is relatively simple (namely, computably enumerable), the set of second-order validities $$Val_2:=\{\varphi\in SOL: \emptyset\models\varphi\}$$ is vastly worse: it's not even hyperarithmetic!
And things get even worse (this gets to Quine's famous remark about second-order logic being set-theory in sheep's clothing): the set $Val_1$ is set theoretically absolute in a way that $Val_2$ is not. For example, there is a second-order sentence $\varphi$ such that $\varphi\in Val_2$ iff the continuum hypothesis is true, and there is a different second-order sentence $\psi$ such that $\psi\in Val_2$ iff the continuum hypothesis is false; and this is provable in ZFC (indeed much less). As a direct consequence, ZFC proves the statement "There is a second-order validity which ZFC does not prove is a second-order validity (unless ZFC is inconsistent)" while by contrast ZFC also proves "every first-order validity is ZFC-provably a first-order validity" (see e.g. the discussion here).
A bit more technically, the above observations about CH lead to serious differences in the model theory. If $N$ is a forcing extension of $M$ we're guaranteed to have $Val_1^M=Val_1^N$ but we can have $Val_2^M\not=Val_2^N$. (Here "$Val_i^X$" is the thing the model $X$ thinks is $Val_i$; keep in mind that both $Val_1$ and $Val_2$ have obvious definitions in the language of set theory. So the distinction we're making here is between "stable" and "unstable" definable sets in the sense of model theory.)
There are also other even more technical senses in which SOL is "fundamentally set-theoretic" in a way in which FOL is not, e.g. with respect to some associated cardinal invariants. But that's going a bit far afield. The point is that the more we look at SOL the more we see that basic questions about it are "set-theoretically contingent" in a way first-order logic does not exhibit.
As I said above, different people may disagree over whether this justifies Vaanaanen's claim. But this is indeed what he is referring to: the fundamental complexity, both computational and set-theoretic, of SOL (with the standard semantics) as compared to FOL.
Meanwhile, it's also frequently said (e.g. by me in the previous round of this answer) that any "tame fragment" of SOL (= sound, effective, non-complete proof systems for SOL) is essentially just FOL in disguise. While diving fully into detail would take us a bit far afield (after all the original question was just what Vaanaanen meant), let me say a little about it:
The basic idea is the following. Any sound, effective, non-complete proof system $\gamma$ for SOL in the language $\Sigma$ has a corresponding proof system for two-sorted first-order logic in the language $\Sigma\sqcup\{\in\}$. This corresponding proof system, being effective, amounts to adding a set $\Xi$ of axioms to whatever theory we're considering: that is, we have $\Gamma\models_\gamma\theta$ in SOL iff $\Gamma'\cup \Xi\models\theta'$ in two-sorted first-order logic, where the "$'$" refers to the syntactic switch from SOL to two-sorted FOL.
But two-sorted first-order logic is in turn reducible to first-order logic: just replace the sorts with unary predicate symbols (it's only when we look at infinitely many sorts that any meaningful difference arises, and even then it's quite minor). In particular, here's a precise claim:
Let $\gamma$ be a sound, effective, non-complete proof system for FOL and let $Val_\gamma$ be the set of SOL sentences $\gamma$-provable from $\emptyset$. Then there is a set $\Xi$ of first-order sentences in the expanded FOL language (= original language + two unary predicates + "$\in$") such that for every SOL sentence $\varphi$ we have $\varphi\in Val_\gamma$ iff the translation of $\varphi$ to the extended FOL language is a validity in the usual first-order sense.
The Hilbert-Ackermann system is a good specific example to work through (and the SEP article linked above treats it); alternately, if I recall correctly the general topic is discussed in Manzano's book.
Indeed, in a precise sense first-order logic is maximally powerful amongst all logics which satisfy a couple basic concreteness properties (e.g. having a c.e. set of validities and not distinguishing between countable and uncountable infinite sets) - this is Lindstrom's theorem. For more on this side of things, including some interesting points on SOL in particular, see the book Model-theoretic logics.
