As a slight generalization to the second example of tkf, for any locally compact (Hausdorff) group $H$ one can consider the group $\operatorname{Aff}(H)=H\rtimes \operatorname{Aut}(H)$ of affine automorphisms of it. By definition, any element of $\operatorname{Aff}(H)$ is of the form $(g,\phi)$, where $g\in H$ and $\phi:H\to H$ is a topological group automorphism. A $(g,\phi)\in \operatorname{Aff}(H)$ acts on $H$ like so:
$$(g,\phi):H\to H, x\mapsto g\phi(x)=l_g\circ \phi(x),$$
where $l_g$ is the left translation by $g$. Note that the group structure of $\operatorname{Aff}(H)$ is given by: $(g,\phi)(h,\psi)=(g\phi(h), \phi\circ \psi)$. In other words, what you call "affine maps" of $H$ assemble into a group that acts via affine maps on $H$.
In particular, if $H$ is a compact group, $G=\operatorname{Aff}(H)$ is a locally compact group, and $G\curvearrowright H$ is an action by affine automorphisms where most elements are not automorphisms (i.e. most elements don't fix the identity element $e_H$ of $H$). One can also take subgroups of $\operatorname{Aff}(H)$ (that has non-trivial translation part) to get further examples (in fact, any (faithful) affine action on $H$ is of this latter form, by definition).
For example if $H=\mathbb{T}^d$ ($d\in\mathbb{Z}_{\geq1}$), then $G=\operatorname{Aff}(\mathbb{T}^d)= \mathbb{T}^d\rtimes \operatorname{Aut}(\mathbb{T}^d)\cong \mathbb{T}^d\rtimes \operatorname{GL}(d,\mathbb{Z})$ (e.g. by https://math.stackexchange.com/a/4180086/169085). Taking the subgroup $\mathbb{T}^d\rtimes 1\leq G$, for $d=1$ we recover tkf's second example.
Above I skipped one detail: since the action $\operatorname{Aff}(H)\curvearrowright H$ is supposed to be topological, we need to specify a topology on $\operatorname{Aff}(H)$. $\operatorname{Aff}(H)$ is a semidirect product, so as a set it is a cartesian product. Accordingly we may define the topology of $\operatorname{Aff}(H)$ as the product topology (see e.g. How to construct a topological group from a semidirect product of two subgroups). So specifying the topology of $\operatorname{Aff}(H)$ reduces to specifying a topology on $\operatorname{Aut}(H)$. The standard way of doing this is to consider $\operatorname{Aut}(H)$ as a subgroup of the group $\operatorname{Homeo}(H)$ of homeomorphisms of $H$, for which there is a variety of natural topologies, e.g. the compact-open topology (= topology of convergence on compact subsets) (see https://mathoverflow.net/q/413361/66883). In the example of the affine actions on the torus there are more options as there is a differentiable structure, and one can consider $\operatorname{Aut}(\mathbb{T}^d)$ as a subgroup of the group $\operatorname{Diff}^r(\mathbb{T}^d)$ of $C^r$ diffeomorphisms of the torus (or the group of volume preserving diffeomorphisms, ...). In any event, the continuity of the affine action will be pointwise continuity, which is straightforward to guarantee.