You have two different contexts of $A$-algebra in the literature. Both give an external definition of the situation $A\subset_{subring}B$. All depends on the context.
A) The first one (the most spread and your case) is that $B$ is a $A$-module with commutation of the multiplications i.e. for all $a\in A$ and $b_i\in B$, one has the identities (associativity w.r.t. to scaling)
$$
a(b_1b_2))=(ab_1)b_2=b_1(ab_2)
$$
Here, "finitely generated as a $A$ algebra" means that there exists a finite set $F\subset B$ such that the smallest $B_1$ for which
$$
A\cup F\subset_{subring}B_1\subset_{subring}B
$$
is precisely $B$.
From this, you see that $B$ is finitely generated as a $A$-module (FGM) means that it exists $F=\{f_1,f_2,\cdots f_n\}\subset B$, such that for all $b\in B$ we have a decomposition
$$
b=\sum_{i=1}^n a_if_i
$$
with $a_i\in A$.
And $B$ is finitely generated as a $A$-algebra (FGA) means that it exists
$F=\{f_1,f_2,\cdots f_n\}\subset B$, such that for all $b\in B$ we have a decomposition
$$
b=\sum_{\alpha\in \mathbb{N}^F} a_\alpha F^\alpha
$$
where $\alpha$ is a (weight) mapping $F\to \mathbb{N}$ i.e.
$\alpha(f_i)=\alpha_i$ and $F^\alpha=f_1^{\alpha_1}\cdots f_n^{\alpha_n}$ (multiindex notation) and $a_\alpha\in A$.
So (FGM) implies (FGA).
For the converse, you need to extend $F$ with the products of powers of the $f_i$, but remaining finite. There the condition that $B$ is integral has to be used. In view of [1], for all $i\in I$, one can write
$$
f_i^{d_i}=\sum_{k=0}^{d_i-1}a_k\,f_i^k
$$
this proves that every $F^\alpha$ can be written as a $A$-linear combination of the $F^\beta$ with $\beta_i< d_i$ for all $i$. But those $F^\beta$ are in finite number. So $B$ is (FGM).
B) In the second one, we have $B$ is a $A$-bimodule and one has, still for all $a\in A$ and for all $b_i\in B$ (associativity w.r.t. scalings)
$$
a(b_1b_2)=(ab_1)b_2\ ;\ b_1(ab_2)=(b_1a)b_2\ ;\ (b_1b_2)a=b_1(b_2a)
$$
[1] wikipedia page https://en.wikipedia.org/wiki/Integral_element