I'm currently using SSL with AES-CBC and HMAC for a file transfer containing string M. Now suppose Alice already knows SHA1(M) (and the adversary does not), and she downloads M from Bob using only AES-CBC without a MAC.
Normally this would be ill-advised, but since she can check SHA1(M) after the download completes, isn't this still secure?
The short answer: No. It is not secure.
Details. To answer the question properly, we first have to decide what we mean by "secure". In this case, I assume security means confidentiality plus integrity. So let's talk about each separately.
Integrity: yes, this provides integrity, under your assumptions. @poncho explained why.
Confidentiality: no, this does not provide strong confidentiality protection. There are chosen-ciphertext attacks that may allow an active attacker to learn partial information about M.
The details and feasibility of such an attack will depend upon things like exactly what padding scheme you use, whether known plaintext is available, and so on. However, the basic concept is: the attacker replaces Bob's transmission with a carefully constructed ciphertext, and watches Alice's reaction. In many systems, Alice's reaction will reveal whether the decrypted message matched Alice's hash. Thus, the attacker learns 1 bit of information from Alice's reaction. And, in certain situations, an attacker can leverage this to learn partial information about M, by choosing his ciphertext cleverly.
Read about padding oracle attacks and that sort of thing to learn more.
Summary. Use a message authentication code in encrypt-then-authenticate mode, or use authenticated encryption. This stuff is tricky and subtle. Don't try to take shortcuts. It is too easy to open up a non-obvious security weakness.
More details: (added 7/29) I see some folks don't believe that the confidentiality attacks are possible. The misplaced confidence in the impossibility of such attacks is an interesting comment on how people's convictions about what is secure are not necessarily correlated to what is actually secure, but I guess I should give further references to demonstrate the possibility of such attacks, as they are non-obvious, counter-intuitive, and quite surprising. To learn more, see, e.g.,
Attacking the IPsec Standards in Encryption-only Configurations, Jean Paul Degabriele and Kenneth G. Paterson, IEEE Security and Privacy 2007.
Problem areas for the IP security protocols, Steven M. Bellovin, Usenix Security 1996, Section 3.8.
Password Interception in a SSL/TLS Channel, Brice Canvel et al, CRYPTO 2003.
Intercepting Mobile Communications: The Insecurity of 802.11, Nikita Borisov et al, MOBICOM 2001, Sections 4.4.2, 4.5.
Pro tip: you might want to make sure you understand the attacks in those papers before feeling too confident that the scheme proposed here can definitely resist those kinds of attacks.
An example attack: For those who have read the papers, here's a sketch of an example attack against the scheme proposed here.
Let's make some assumptions to keep the example simple. Suppose that, before encryption, the message M is padded out to a multiple of 16 bytes by appending arbitrary bytes and then a single byte indicating how many bytes of padding should be removed. Suppose for simplicity of exposition that the message M is always exactly 1 byte long. This means M will be padded before encryption out to 16 bytes by appending 14 random bytes and then 1 byte with the value 0x0F. Suppose that the attacker has previously captured plenty of known plaintext and its corresponding encryption, and now the attacker has intercepted the encryption of M and wants to learn something about M.
Now, suppose the attacker has a guess G at the value of the message (G is a single byte), and the attacker wants to confirm whether M=G or not. Here's how the attacker can confirm his guess.
Let $IV$ denote the IV used when encrypting M; note that $IV$ is known to the attacker. The attacker looks through his collection of pairs $(P_i,C_i)$ of known input-output pairs for AES (each one is a block width); these can be derived from the known plaintext and ciphertext for a CBC-mode encryption. The attacker looks for such a pair $(P_i,C_i)$ such that $IV \oplus P_i$ has G as its first byte and has 0xF as its last byte. The attacker replaces the ciphertext that Bob sent by the ciphertext $(IV, C_i)$ and waits to see Alice's reaction. If Alice rejects this message, then the attacker learns that $M\ne G$. If Alice accepts this message, the attacker learns that $M=G$.
Therefore, the possibility of this attack means that the scheme proposed in this question does not provide semantic security: it is not IND-CCA2 secure. Put in non-technical terms, it does not provide the level of confidentiality protection that we have come to expect from an encryption scheme. In practice, such a vulnerability might be quite severe and might be readily exploitable by bad guys.
If you've read the research literature that I cited above, none of this is anything especially novel or new. Of course, if you're not familiar with that research literature, this stuff is thoroughly surprising and unexpected. So, take this as a lesson about the dangers of departing from accepted practice, even if everything seems fine to you. If you're not an expert in this area, you probably aren't qualified to judge whether your scheme is secure -- and if you are an expert, I suspect you'll probably be especially leary of departing from accepted practice, given the number of surprising attacks that have been discovered in the past.
It is easy to see that this secure, in the sense that the attacker cannot cause Alice to accept any download except for the file that Bob originally sent.
This remains true even if the attacker knows the encryption (CBC) key (alternatively, Alice and Bob doesn't bother to encrypt the message at all), and if the attacker also knows the correct $SHA1(M)$ value, as long as the attacker cannot choose the file $M$.
This is because if the attacker can cause Alice can accept another message $M' \ne M$, then that's because $SHA1(M) = SHA1(M')$, and so the attacker has effectively found a second preimage to the message $M$. As far as we know, this is infeasible to do with SHA1.
Personally, I would suggest using (say) SHA256 rather than SHA1; that way, we don't have to even make the assumption that the attacker cannot influence $M$.
Now, you might ask "if this works, why do we bother with MACs"? Well, I believe that a large part of it is that it is fairly unusual that Alice knows the value of $SHA1(M)$ independently of the download; in most cases, the receiver has no apriori information of the message contents.
You can be sure that the attacker did not manipulate the file. It's preferable to use a hash function that's collision resistant, even if it doesn't seem to be strictly necessary in your application.
But confidentiality against an active attack is problematic. Since the authentication happens inside encryption (MAC-then-encrypt). A careless implementation that leaks if the padding was valid will allow an attacker to recover the whole plaintext via a padding oracle. A careful implementation can reduce the leak, but that's pretty tricky to do and might still allows some attacks. I don't see any attack that works, but why would you take such an unnecessary risk? If you want/need to rely on hash verification, use CTR or OFB mode which are far easier to reason about than CBC or CFB.
Finally with your method Alice can only reject or accept the data one the download has completed. This requires a full re-download on error, and prevents streaming use. This can be avoided with tree-hashing or packet wise MACs.
M? What padding do you use? Is the message split in several blocks, or just a single continuous stream with an IV at the beginning, and the padding in the end? Do you send multiple messages through the same channel? – CodesInChaos Jul 29 '12 at 20:50