When you say "transactions", I assume you mean "transaction output".
It is literally not possible to have a transaction output that is larger than the transaction size limit. Otherwise the transaction containing that output would not be in the blockchain.
Rather I think what you are looking for is that the output script is larger than the maximum script size. Such scripts are provably unspendable and can be excluded from the UTXO set.
Bitcoin Core has a function to determine whether outputs are unspendable. It is not an exhaustive list and is just the two that you mention: Scripts beginning with OP_RETURN, and scripts larger than the max script size.
However there are other scripts that are provably unspendable, just that they require a little more work to prove. For example scripts that use invalid or disabled opcodes are provably unspendable. The script interpreter will fail if any such opcode is found in the script when it is verifying the spend.
But even that has a catch. Unknown opcodes that are not executed (e.g. in an unexecuted IF branch), the script can still pass validation. However if the script contains any of the disabled opcodes anywhere (there are a few exceptions to this as some "disabled" opcodes were actually just outright removed), it is invalid.
There are also potentially scripts that require something larger than the maximum stack item size to be pushed to the stack. Or something that manipulates a stack item so that it becomes a negative locktime when used with OP_CHECKLOCKTIMEVERIFY or OP_CHECKSEQUENCEVERIFY.
Then there are output scripts that are programmed invalidly such as having just an OP_ELSE or OP_ENDIF with no OP_IF/OP_NOTIF coming before it.
All of these things are hard to determine and really require inspecting the individual scripts. They also require having a consensus conforming script interpreter, which itself is hard to do because the exact semantics of scripts are actually unclear.