Is it the same account and non-SSL works ok, but SSL is slow? If it's not the same account it's possible you have specified a hostmask (i.e.
foo@xyz.com) and the identd response or the nameserver lookup is what is really slowing things down as these must be validated before the user can login.
Also, where does the slow part happen... Before the account prompt, after the password is entered, etc. You gotta be more specific.