I'm new to the topic guys, but as there are literally too many encryption methods out there, too that I could even come up with many ideas of them, how to we actually test/calculate "how secure" an encryption algorithm is?
Do we a standard or a formula that we can use to test security level of encryption algorithm?
And how strong would a monoalphabetic cipher would be?
There's no single formula to quantify the security of an encryption algorithm. However, we do estimate a measure, the "security level", for various cryptographic functions. Security level is measured in "bits", where $n$-bit security means that an attacker would have to perform $2^n$ operations to break the system in question. Sometimes we'll further quantify different sorts of operations, or how much memory is needed. At least about $2^{112}$ bits of security is commonly considered the minimum for any system to be "secure" against practical attackers.
"Security" is a rather broad concept. There are several different notions of security. For encryption, we use the IND- definitions, which define what capabilities an attacker has. We usually require the strongest of those definitions, IND-CCA3 (AKA AE-security) to be met for a system to be considered secure, though individual components of that system may only meet lesser security definitions.
As for your question, no, we don't have "a formula". What we do is try to provide a mathematical proof that goes as follows: if you break my cryptosystem, then you are capable of solving a super-hard (public) problem that nobody (publicly) knows the solution for. Or, if you break my cryptosystem, then you also know how to break that other cryptosystem as well.
– zugzwang Oct 24 '22 at 11:38