If neither of the 'big two' of TLS Handshake and IKE are appropriate in a given situation, what alternative Authenticated Key Exchange (AKE) standards exist and are recommended?
Many protocols have been proposed in the literature over the past 20+ years, though only a small subset have made it into standards of some form that are also still considered secure.
Since you do not describe why TLS Handshake and IKE are appropriate in your situation, and as long as you don't describe your situation, it's hard to really help you. Also, you haven't stated if it's only IKE that's not appropriate, or if that also includes IKEv2 (which improved the IKE protocol). Therefore, I'll simply assume you meant both.
As an alternative to IKE, the first thing that comes to mind is KINK (Kerberized Internet Negotiation of Keys). It's a protocol defined in RFC 4430. It is used to set up an IPsec security association (SA), similar to Internet Key Exchange (IKE), utilizing the Kerberos protocol to allow trusted third parties to handle authentication of peers and management of security policies in a centralized fashion.
As said — as long as you don't describe your situation, it's hard to pinpoint what might suit it best.
It could well be that your situation might be solved with RFC 6539, as IBAKE provides an Identity-Based Authenticated Key Exchange.
Or maybe "augmented PAKE" is a valid option for your situation. That's a variation applicable to client/server scenarios, in which the server does not store password-equivalent data. This means that an attacker that stole the server data still cannot masquerade as the client unless they first perform a brute force search for the password. Examples include: AMP, Augmented-EKE, B-SPEKE, PAK-Z, SRP, and AugPAKE (RFC 6628).
It's hard to tell when not knowing your situation. But I would check out KINK first… as that should be a close-to-perfect match, being an alternative to IKE.
In addition to the earlier remarks about the missing background of your question please also consider that TLS and IKEv2 are actually not just a single authentication and key exchange protocol but rather a framework that supports many different AKA protocols.
Let us use TLS as an example. In TLS you have the concept of ciphersuites and they allow you to add new AKA algorithms to TLS. To pick one ciphersuite, RFC 4279, defines pre-shared secret ciphersuites with TLS. This may be useful for you if you do not want to use certificates but rather want to rely on shared secrets instead.
There lots of different ciphersuites defined for TLS, as you can see from the IANA registry.
There may be the case that you want to use asymmetric cryptopraphy but you do not like the overhead that comes with X.509 certificates. Then, you have the choice of using raw public keys or PGP keys instead (search for draft-ietf-tls-oob-pubkey).
I would argue that it is rather likely to find your favorite authentication and key exchange protocol already standardized for TLS. With more information about the requirements it would be possible to point you to the appropriate one.
http://csrc.nist.gov/groups/ST/toolkit/documents/SP800-56Arev1_3-8-07.pdf
– Brownbat Aug 24 '13 at 01:40