Subject: "too many auth failures"?

From: der Mouse <[email-masked]>
Date: Tue, 26 Oct 2010 14:20:27 -0400 (EDT)

I recently ran into some ssh behaviour that strikes me as broken and
thought I'd see what the list thought.

This is a semi-embedded commercial implementation; I'm not naming the
vendor here because this hasn't been reported to them yet, so it's
hardly fair to do anything even vaguely like name-&-shame now.

The symptom was that ssh to the device worked fine once, but when I
tried later, form a slightly different environment, I would
consistently get a "protocol error" failure.  Upon turning on cleartext
packet tracing, it turned out that my client offered six public keys
(four in the agent and two from files), none of which were accepted.
When my client then tried to go keyboard-interactive in response to the
sixth key failure, the remote end sent an SSH_MSG_DISCONNECT with
reason code 2, SSH_DISCONNECT_PROTOCOL_ERROR, and a description string
saying there had been too many authentication failures for that user.

Disabling publickey auth in the client (which the client I was using
can do on the command line) got me a password prompt and I was able to
log in normally (thus indicating this did not mean "too many failures,
account locked").  The first attempts, the ones which worked without
any fuss, came from a different login, one which had only one key in
the agent and two in files.

This strikes me as wrong in two respects: (1) it's (apparently)
counting each failed public key as a separate "authentication" (okay,
three respects; that should be "authorization") failure, and (2) it's
claiming a protocol error for an error condition which is not in fact a
protocol error.  It seems to me that it either should use a better
disconnet code (SERVICE_NOT_AVAILABLE, maybe?), or it should fail in a
userauth way - perhaps SSH_MSG_USERAUTH_FAILURE with an empty list of
authentication methods.  (Hm, that's a botch in the RFCs I missed when
we were hammering them out; the "Authentication Protocol" is actually
more of an authorization protocol than an authentication protocol.)

So, thoughts?  Am I missing something, or is this really as ill-behaved
as I think?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 26 Oct 2010 13:35:12 -0500

> thought I'd see what the list thought.

Yes, I've seen this a lot.

Some implementations treat all userauth method failures as contributing
equally to a max-failures counter.  This is really, really wrong.  Only
"password" and "keyboard-interactive" failures should count for the
traditional max-failures counter, with a separate counter for GSS,
pubkey, and hostbased userauth.

What's really galling is that "none" failures also count -- there's too
many clients that start by attempting "none" userauth.  And, of course,
clients try the non-interactive/ password methods first, as you'd
expect.

So you might have 1 "none" failure, 2 "hostbased" failures (if the
client has DSS and RSA keys), N "publickey" failures, and never get to
"keyboard-interactive" nor "password" userauth.  If the server has a 6
failures max counter your client will get disconnected when it tries the
third publickey that is rejected.

Whenever I run into such a server I have to resort to enabling only the
userauth method I expect to see succeed.  I hate having to do that.

I also hate the fact that keyex is not re-tryiable.

I'm all for publishing an RFC saying "don't do such stupid things".

Nico
--

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Oct 2010 15:05:07 +1300

der Mouse <[email-masked]> writes:

>So, thoughts?  Am I missing something, or is this really as ill-behaved as I
>think?

I don't think it is.  If you've set the server up to allow three tries at auth
then you get three tries (I'm assuming it's set up to allow six here, which is
a bit non-traditional, I would have expected three).  Stepping back a bit, why
are you sending *six* keys to the server?  Shouldn't the client know which key
it's supposed to use?  It seems more like the client is broken than the
server.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Oct 2010 15:06:35 +1300

>I also hate the fact that keyex is not re-tryiable.

Wouldn't that defeat having a retry counter?

Peter.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 00:21:08 -0500

> Wouldn't that defeat having a retry counter?

The two issues are unrelated.  The SSHv2 initial key exchange cannot be
retried if it fails.  It can only really fail if you use the SSHv2 w/
GSS-API keyex, and it does happen, often because of Kerberos
configuration issues.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 00:23:04 -0500

> it's supposed to use?  It seems more like the client is broken than the

My view is that servers should have two failure counters: one for
password and keyboard-interactive, another one for all others.

Nico
--

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Oct 2010 18:40:09 +1300

Nicolas Williams <[email-masked]> writes:

>My view is that servers should have two failure counters: one for password
>and keyboard-interactive, another one for all others.

Yeah, I'd thought about that too, but where do you stop?  Which counter type
would a ZKP use?  Or EKE?  Or IBE?

(I still think the real problem isn't the server but the client, how's it
managing to try six different keys to get to one server?).

Peter.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 00:54:12 -0500

> would a ZKP use?  Or EKE?  Or IBE?

The distinction relates to password guessing.  ZKP -> same retry counter
as password and keyboard-interactive; same for EKE.  IBE (ID based
encryption?) is not like a password-based method at all.

Methods that might result in MITMs getting material suitable for
off-line dictionary attacks are not special here as no counter on the
server side can help prevent such MITM attacks, but because such methods
are subject to password guessing, they should be subject to the same
counter as password and keyboard-interactive.

> managing to try six different keys to get to one server?).

Why not?  SSH user public keys are effectively pseudonyms, so you can
see why users might have many of them.  Nothing wrong with that...  On
the server side you don't want a client sucking up resources testing
whether some pubkey or another might work...  so the server says all
might work and the client has to actually sign with those keys, and the
server also has to implement resource controls pre-authentication
(including a timer to disconnect if the client takes too long to
authenticate).

Nico
--

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Oct 2010 19:43:18 +1300

Simon Josefsson <[email-masked]> writes:

>Isn't this another problem?  Often private keys are protected by a password
>or requires a PIN to unlock a smartcard, and iterating to sign with all keys
>becomes a user interface issue quickly.  This is how libssh2 works, and I
>couldn't think of any way to avoid it.

That's been a major issue for S/MIME deployments in the past, as soon as you 
enable message signing/encryption you get bombarded with requests to insert 
smart cards, enter PINs and passwords, and generally end up being a crypto 
bookkeeper for your mail app.  The usual solution is to use S/MIME gateways, 
where the organisational mail gateway holds your key and signs/decrypts on 
your behalf, more or less defeating the point of personal message security 
since now you're just doing a STARTTLS for SMTP-equivalent in a really awkward 
way.

(The lesson there is that the app should do all it can to minimise the load on
the user, otherwise security gets sacrificed in exchange for functionality).

Peter.

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 27 Oct 2010 08:23:19 +0200

> (including a timer to disconnect if the client takes too long to

Isn't this another problem?  Often private keys are protected by a
password or requires a PIN to unlock a smartcard, and iterating to sign
with all keys becomes a user interface issue quickly.  This is how
libssh2 works, and I couldn't think of any way to avoid it.

/Simon

---

From: der Mouse <[email-masked]>
Date: Wed, 27 Oct 2010 09:58:46 -0400 (EDT)

>> So, thoughts?  Am I missing something, or is this really as
>> ill-behaved as I think?
> I don't think it is.  If you've set the server up to allow three
> tries at auth then you get three tries (I'm assuming it's set up to
> allow six here, which is a bit non-traditional, I would have expected
> three).

I didn't set it up; this is, as far as I know, out-of-the-box vendor
defaults for the system in question.

> Stepping back a bit, why are you sending *six* keys to the server?
> Shouldn't the client know which key it's supposed to use?

How should it know that?  It's got four keys in the agent; when they're
exhausted, it tries the two it finds in files.  I could have configured
it differently, but, since I didn't give it any particular
configuration, it has no reason to think any key is more likely to work
than any other.

As for retries, I conjecture that the retry limit is something more
normal like three or five, but, while the implementation counts
publickey failures (or, more precisely, keys offered which the server
finds unacceptable) against the limit, it doesn't actually *test* the
limit until the client tries to use keyboard-interactive.

But that's just conjecture.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 27 Oct 2010 10:15:12 -0400 (EDT)

> Yeah, I'd thought about that too, but where do you stop?  Which
> counter type would a ZKP use?  Or EKE?  Or IBE?

The "all others".

Think about the point of those failure counters: they're designed to
slow down password-guessing attacks.  publickey, ZKP, etc, don't have
anything like password-guessing risks, so it's arguably inappropriate
to do failure counters for them.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 27 Oct 2010 10:34:35 -0400 (EDT)

>> [trying six publickey keys]
> On the server side you don't want a client sucking up resources
> testing whether some pubkey or another might work...

That's actually a relatively cheap operation on most servers, and ssh
specifically supports it: see the first form of
SSH_MSG_USERAUTH_REQUEST (RFC4252, section 7).

> so the server says all might work and the client has to actually sign
> with those keys,

This was not the behaviour I saw.  My packet traces show the server
refusing the keys without asking the client to perform PK operations,
and my agent has an interface that, among other things, prints
something every time it actually performs a publickey operation, which
didn't show anything during my failed login attempts.

> and the server also has to implement resource controls
> pre-authentication (including a timer to disconnect if the client
> takes too long to authenticate).

The delay from my attempting the login to my getting the failure was
less than any reasonable timeout - less than the time it would take for
me to type a password I didn't know by heart, certainly.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 27 Oct 2010 10:36:18 -0400 (EDT)

> password or requires a PIN to unlock a smartcard, and iterating to
> sign with all keys becomes a user interface issue quickly.

Indeed.  If you are faced with such an implementation, you probably
should tell your client which key to use (possibly directly, possibly
via some sort of configuration).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Joseph Galbraith <[email-masked]>
Date: Wed, 27 Oct 2010 10:51:46 -0600

> it's supposed to use?  It seems more like the client is broken than the

I don't think the SSH server should count unsigned publickey attempts as
failures, since they aren't really an attempt.  There are many
clients that simply try all the keys they know about as unsigned
attempts in an attempt to get connected with less user configuration.

A signed publickey attempt with a bad signature probably should be
counted as a failure though.

Thanks,

Joseph

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 14:07:18 -0500

> libssh2 works, and I couldn't think of any way to avoid it.

That's a client-side issue.  We're talking about server-side max try
counters.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 14:08:21 -0500

On Wed, Oct 27, 2010 at 06:54:47AM +0000, Love Hörnquist Åstrand wrote:
> Maybe I'm missing something, but you don't need to iterate over all
> keys since you can ask the ssh server if the public key is ok to use.

Yes, but servers can be configured to answer "yes" to all such probes.

>                               To avoid unnecessary processing and user
>    interaction, the following message is provided for querying whether
>    authentication using the "publickey" method would be acceptable.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 14:18:39 -0500

> to do failure counters for them.

EKE is password-based, therefore it has a password-guessing problem.  I
assume ZKP here means zero-knowledge proof, and if it's a proof of a
password, then that too falls into the password-based method list that
should be subject to a small retry counter.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 14:22:13 -0500

> That's a client-side issue.  We're talking about server-side max try

Also, you could refuse to enter a PIN to get the key skipped...  You
could specify what key to use...  You could specify to not use any keys.
You could use an agent that gets the PIN.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 27 Oct 2010 14:32:15 -0400

--On Wednesday, October 27, 2010 10:51:46 AM -0600 Joseph Galbraith 
<[email-masked]> wrote:

>> I don't think it is.  If you've set the server up to allow three tries
>> at auth then you get three tries (I'm assuming it's set up to allow six
>> here, which is a bit non-traditional, I would have expected three).
>> Shouldn't the client know which key it's supposed to use?  It seems more
>> like the client is broken than the server.
>

Agree.  In fact, I think userauth methods should distinguish between 
authentication failures and authorization failures (in which authentication 
succeeded or was not tried), and only count the former.

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 15:12:16 -0500

On Wed, Oct 27, 2010 at 09:53:39PM +0200, [email-masked] wrote:
> > Often private keys are protected by a password or requires a PIN to
> > unlock a smartcard, and iterating to sign with all keys becomes a user
> > interface issue quickly.
> 
> As far as I understand, the possibility to send a
> SSH_MSG_USERAUTH_REQUEST without a signature in it is intended to solve
> precisely this problem.

But some servers always answer that query positively.

A better solution is to have ssh-agent-like facilities that can use your
smartcard/token, and to which you give your PIN.  Certainly there's no
way to prevent the advent of such agents.

> The client can store the *public* keys somewhere where you don't need
> any user interaction to retrieve them, send a bunch of userauth requests
> with these keys and no signatures. When you get a SSH_MSG_USERAUTH_PK_OK
> response from the server, you ask the user to unlock the corresponding
> private key and send a single SSH_MSG_USERAUTH_REQUEST with both key and
> signature. This shouldn't have to cost more than a single network
> roundtrip, independent of the number of public keys.

Even so users won't like having to interact even that much.

> (Since I'd consider that client behaviour perfectly normal, I also think
> that the server should *not* increment the failure counter when sending
> a SSH_MSG_USERAUTH_PK_OK).

Yes.

---

From: [email-masked] (Niels Möller)
Date: Wed, 27 Oct 2010 21:53:39 +0200

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

---

From: [email-masked] (Niels Möller)
Date: Wed, 27 Oct 2010 23:56:12 +0200

> But some servers always answer that query positively.

Which servers do that? That seems like very broken behaviour to me. It
would be somewhat understandable if there were some library function
which checks the signature and the authorization as an (to the caller)
atomic step. But I'd rather expect an ssh implementation to do something
like

  if (key_authorized(...))
    {
      if (signature_provided)
        {
          if (signature_correct(...))
            send SSH_MSG_USERAUTH_SUCCESS, and start service
          else
            send SSH_MSG_USERAUTH_FAILURE
        }
      else
	send SSH_MSG_USERAUTH_PK_OK
    }
  else
    send SSH_MSG_USERAUTH_FAILURE

where signature verification and authorization are independent functions, possibly
even in different libraries.

Regards,
/Niels

---

From: Peter Gutmann <[email-masked]>
Date: Thu, 28 Oct 2010 13:46:12 +1300

[Combining several replies to save space]

der Mouse <[email-masked]> writes:

>How should it know that? 

By keeping track of which key belongs to which server.  When I sign in to
gmail I (meaning Firefox acting on my behalf) don't try forty different
passwords in the hope that one fits, I send the exact password that's used for
gmail.  Why is the SSH client unable to do the same thing for a public key?

Simon Josefsson <[email-masked]> writes:

>Also, you could refuse to enter a PIN to get the key skipped...  You could
>specify what key to use...  You could specify to not use any keys. You could
>use an agent that gets the PIN.

You forgot the one that actually happens, based on extensive real-life
experience with S/MIME and SSL:

- The user bypasses the annoying security mechanisms in order to get on with
  their job and avoid all the crap that their (from their perspective) broken
  software is throwing at them.

James Cloos <[email-masked]> writes:

>Ssh(1) will try the key specificed on the command line or in the config files
>for the specified remote host, if any.  Should that fail, it then tries every
>key in the agent in the order they were ssh-add(1)ed.

That sounds like really, really broken behaviour.  You're saying the client is
totally unable to keep track of which key is used to authenticate to which
server, and just tries them at random in the hope that one fits?  That would
be like Firefox (see above) trying every stored password it knows for every
site in the hope that one works.  There are an awful lot of password managers
around for browsers with a huge number of users (the Roboforms one, the last
time I checked, claimed 28 million downloads), and none of them resort to this
kind of brokenness.

This could even be a security problem, because a malicious server can get a
client to sign an auth message using any key the client holds, which it can
then forward to another server and impersonate the client.  The only thing
preventing this is the exchange hash, which requires that either the malicious
server can create collisions with SHA-1 or that the target server is using a
poor RNG *cough*Debian*cough* so you can keep the exchange hash constant.
Still, a mechanism that allows the server to have the client act as an
arbitrary signing oracle really doesn't seem like a good idea.

>From the above it certainly looks to me like the client is badly broken.  Fix
the client, don't require all sorts of kludgery at the server.

Peter.

---

From: der Mouse <[email-masked]>
Date: Wed, 27 Oct 2010 22:48:12 -0400 (EDT)

> By keeping track of which key belongs to which server.  When I sign
> in to gmail I (meaning Firefox acting on my behalf) don't try forty
> different passwords in the hope that one fits, I send the exact
> password that's used for gmail.  Why is the SSH client unable to do
> the same thing for a public key?

Well, in the most naïve form, the answer is "because I never
implemented it" (mutatis mutandis for those using other
implementations).

But it's worth thinking about why the difference arises.  The biggest
difference, I think, is simple: the damage potentially done by trying
inappropriate keys is substantially less than that done by trying
inappropriate passwords.  If you reveal to me the public key for the
key pair you to ssh-authenticate yourself to (say) your employer, I
don't learn enough to do anything significant.  But if you send me your
gmail password then I can trivially impersonate you to gmail.  This
makes it far more reasonable for your ssh client to not bother keeping
track of which keys work for which servers - and, at the same time, far
more reasonable for you to use the same key for disparate servers.

Also relevant: there reasonably can be (and often are) many keys which
work for a single login.  There rarely are multiple passwords which
work for a single login.

That said, the ability to for the client to be told, or perhaps even
autolearn, what keys work for what servers is probably a desirable
thing, even if not particularly valuable in many cases.  It does,
however, appear to be comon for clients to not do it.

>> [...client just tries keys it has available...]
> That sounds like really, really broken behaviour.  You're saying the
> client is totally unable to keep track of which key is used to
> authenticate to which server, and just tries them at random in the
> hope that one fits?

Generally, yes.  (Well, for weak values of "random".)

> That would be like Firefox (see above) trying every stored password
> it knows for every site in the hope that one works.

Except significantly less bad; see above.

> There are an awful lot of password managers around for browsers with
> a huge number of users (the Roboforms one, the last time I checked,
> claimed 28 million downloads), and none of them resort to this kind
> of brokenness.

Because with passwords, you can't really do that; the exposure is too
high.  One of the nice things about public-key auth is that you _can_
do that without significant ill effects.

> This could even be a security problem, because a malicious server can
> get a client to sign an auth message using any key the client holds,
> which it can then forward to another server and impersonate the
> client.

This is called a man-in-the-middle attack.  Passwords are vulnerable to
it (among many other active attacks).  ssh is carefully designed to
_not_ be vulnerable to it - well, except for first connect in cases
where the user incorrectly believes it's safe to trust the
peer-presented host key; those in situations where such risks are part
of the threat model should take care to not do that.  There might even
be a place for clients that don't support doing it at all.

> Still, a mechanism that allows the server to have the client act as
> an arbitrary signing oracle really doesn't seem like a good idea.

Then perhaps it's just as well ssh takes care to make sure it can't be
abused that way.  This is (part of) why client-chosen entropy and
server-chosen entropy are both mixed into the data signed during PK
auth: the server does not get to have the client sign arbitrary data.

> Fix the client, don't require all sorts of kludgery at the server.

Fortunately or unfortunately, that ship has sailed: ssh requires that
the server include support for this sort of behaviour by the client.
Of course, servers may choose to refuse service to clients that take
advantage of it, or can go out of their way to be obnoxious to clients
that try to use it (by lying, claiming that keys that cannot work might
work); as this thread demonstrates, such server behaviour is widely
considered problematic, even broken - and, I believe, with good cause.
I certainly know I'd file a bug report with the applicable vendor upon
encountering such behaviour (as I intend to with the vendor whose
server started this off - quite likely would have today except that
other issues blew up).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 Oct 2010 23:00:04 -0500

> >use an agent that gets the PIN.

I wrote that, not Simon :)

>   software is throwing at them.

How would they do that from the client side?

Anyways, I suspect that SSH keys are rarely stored on smartcards, so
this isn't a big problem _yet_.  I agree that it will be, ...

...but the most reasonable solution is to hand the smartcard and PIN to
an ssh-agent.  If the user's not willing even to type a PIN _once_, then
smartcards won't succeed.

Nico
--

---

From: Love Hörnquist Åstrand <[email-masked]>
Date: Wed, 27 Oct 2010 06:54:47 +0000

> couldn't think of any way to avoid it.

Maybe I'm missing something, but you don't need to iterate over all keys since you can ask the ssh server if the public key is ok to use.

      byte      SSH_MSG_USERAUTH_REQUEST
      string    user name in ISO-10646 UTF-8 encoding [RFC3629]
      string    service name in US-ASCII
      string    "publickey"
      boolean   FALSE
      string    public key algorithm name
      string    public key blob

Love

---

From: James Cloos <[email-masked]>
Date: Wed, 27 Oct 2010 17:18:36 -0400

>>>>> "PG" == Peter Gutmann <[email-masked]> writes:

PG> I don't think it is.  If you've set the server up to allow three
PG> tries at auth then you get three tries (I'm assuming it's set up to
PG> allow six here, which is a bit non-traditional, I would have
PG> expected three).  Stepping back a bit, why are you sending *six*
PG> keys to the server?  Shouldn't the client know which key it's
PG> supposed to use?  It seems more like the client is broken than the
PG> server.

I only use openssh clients.  I can’t speak for other implementations.

If you have speparate key pairs for specific remote sites, it is very
easy to end up with several keys loaded into ssh-agent(1).

Ssh(1) will try the key specificed on the command line or in the config
files for the specified remote host, if any.  Should that fail, it then
tries every key in the agent in the order they were ssh-add(1)ed.

It seems clear that few if any of the early implementers used more than
one rsa, one dsa and one ssh1 key per account back when the v2 protocol
worked its way through this working group.  Else this issue would have
been on the radar back then.

One implmentation fix would be a way to say not only ‘use this private
key for that set of server(s)’, but also ‘use this private key for ONLY
that set of server(s)’.  I’m not aware of a way to do that in openssh’s
config files.

-JimC
-- 
James Cloos <[email-masked]>         OpenPGP: 1024D/ED7DAEA6

---

From: Damien Miller <[email-masked]>
Date: Sun, 31 Oct 2010 22:03:49 +1100 (EST)

On Wed, 27 Oct 2010, James Cloos wrote:

> One implmentation fix would be a way to say not only ?use this private
> key for that set of server(s)?, but also ?use this private key for ONLY
> that set of server(s)?.  I?m not aware of a way to do that in openssh?s
> config files.

~/.ssh/config's IdentityFile and IdentitiesOnly options do what you want.

-d