The Core Question
How would one define and justify the tensor notation for vectors and operators on a separable Hilbert space?
Motivation
Whenever I work with finite-dimensional linear algebra, I find it very illuminating to think about vectors, linear maps, bilinear forms etc. collectively as just various types of tensors. I like how the tensor formalism puts emphasis on representation-free formulations and how even a very complicated multi-linear map can be understood in terms of its individual „indices“.
When I started to learn Quantum Mechanics, I immediately tried to use the tensor notation. At a surface level it seemed to work just fine. One can formally rewrite the bra-ket expressions on left with a „tensor-like“ notation on the right: $$ \begin{aligned} \left< \psi \mid \varphi \right> \qquad &\leftrightsquigarrow \qquad \overline{\psi^{\,\mu}} \; g_{\overline{\mu} \nu} \; \varphi^\nu \\ \left| \psi \right> = \hat A \hat B \left| \varphi \right> \qquad &\leftrightsquigarrow \qquad \psi^{\,\mu} = A^\mu_\nu \; B^\nu_\kappa \; \varphi^\kappa \\ W = \left| \psi \right>\left< \psi\right|, \; \operatorname{Tr} W = 1 \qquad &\leftrightsquigarrow \qquad W^\mu_\nu = \psi^{\,\mu} \; g_{\overline{\kappa}\nu} \; \overline{\psi^\kappa}, \;\; W^\mu_\mu = 1 \end{aligned} $$ Furthermore, the tensor notation feels more natural when dealing with composite systems, where the state is a tensor product of states of the underlying systems. However, when I tried to put the notation on a more rigorous ground, I learned that the topic is much more difficult and nuanced than I thought.
The Problem
In finite-dimensional spaces, the tensor notation is made possible by several circumstances: \begin{gather} V^* \otimes V \simeq \operatorname{Hom}(V, V) \label{homomorphism} \tag{1} \\[5pt] T(\mathbf{v}) = T( \; \sum_k v^k \mathbf{e}_k \;) = \sum_k v^k \; T(\mathbf{e}_k) \label{continuity} \tag{2} \\ \big| \operatorname{Tr}(T) \, \big| < \infty \label{trace} \tag{3} \end{gather} While \eqref{homomorphism} essentially makes sure that all $k$-covariant $l$-contravariant tensors are from the same space, \eqref{continuity} lets us describe all tensors using their coefficients wrt. some basis and \eqref{trace} lets us express all operations on tensors using just tensor product and contraction.
In a sense, neither of these is true for the separable Hilbert space $\mathcal H$ where QM is done. By a basis on a Hilbert space one usually means the Schauder basis, which describes vectors with an infinite series of coefficients. This, combined with the fact that most interesting operators in QM aren't continuous, gives us: $$ T(\mathbf{v}) = T( \; \sum_{k=1}^\infty v^k \mathbf{e}_k \;) = T( \; \lim_{N\to\infty}\sum_{k=1}^N v^k \mathbf{e}_k \;) \neq \lim_{N\to\infty} T( \; \sum_{k=1}^N v^k \mathbf{e}_k \;) = \sum_{k=1}^\infty v^k \; T(\mathbf{e}_k) $$ The sequence on the RHS might either diverge, or even converge to a different value than LHS (although this is pathological and usually not considered in practise). Condition \eqref{trace} only holds for the so-called trace-class operators (not even the identity is trace-class) and the space $\mathcal H^* \widehat{\otimes} \mathcal H$ (tensor product of Hilbert spaces) is isomorphic to finite-rank operators, a proper subset of $\operatorname{Hom}(\mathcal H, \mathcal H)$.
A Possible Solution? (and more questions)
These unfortunate circumstances mean, that if it's even possible to justify the tensor notation in infinite-dimensional spaces, it can't be by a simple generalization of the finite-dimensional case. So, is it possible to make it work?
The condition \eqref{continuity} seems relatively easy to fix – one just needs to replace the Schauder basis with a Hamel basis. A serious downside to this would be that a typical operator would have an uncountable number of coefficients.
For \eqref{homomorphism} it would be great if one could find a space $\mathcal G$, such that the Hilbert space $\mathcal H$ is embedded in it $\mathcal H \subset \mathcal G$, with the property $\mathcal G \otimes \mathcal G \simeq \operatorname{Hom}(\mathcal H^*, \mathcal H)$. Vectors of the bigger space could then be used to construct operators and bilinear forms. Here, one immediately thinks about the Gelfand triple $\Phi \subset \mathcal H \subset \Phi^*$ – could it be that $\mathcal G = \Phi^*$ for an appropriate choice of rigging? Sadly it appears that no, as the Schwartz kernel theorem says that $\Phi^* \otimes \Phi^* \simeq \operatorname{Hom}(\Phi, \Phi^*)$. Is such a space $\mathcal G$ even possible? If yes, would this approach generalize to tensors of higher degree?
Finally, the problem with \eqref{trace} seems to be there to stay. There is no reasonable way to define the trace of identity, for example. What I'm interested in is whether, given a complicated expression, one can tell a priori which indices are contractible and which will inevitably diverge.
Are these proposed “solutions” any good? Is there any literature that would investigate this topic? Or am I on the wrong path and should I stop wasting my time with tensors in the infinite dimension?