Search This Blog

10.24.2014

Postfix, Dovecot SASL, and a broken cert

Just spent roughly 16-18 hours (Monday around 2 or 3 to Wednesday around 4) fixing a problem (well, 4 major problems and a minor problem) with the webmail server.  It only manifested as, mail can be received from the webmail server via IMAP, webmail server can receive mail from the servers that forward it mail, but mail cannot be sent through it.  It runs Postfix SMTP, and Dovecot for IMAP and SASL.  It's been broken for a few months, but it was going to be decommissioned for a while (that plan fell by the wayside), and it generally wasn't an issue until someone hit 'send read receipt' in Outlook instead of 'don't send'.  Outlook on this user's computer didn't have an Outbox for the webmail accounts, so there was no way to stop it from trying to send the message over and over except to fix the problem with the server.

First, I looked over the Dovecot config for glaring issues, since I remember the breakage happening around the time the server was upgraded from Debian Squeeze to Wheezy (Dovecot 1.x to 2.x).  I saw none.  I abandoned that route at that point because I didn't realize Dovecot was serving up SASL yet.

Next, I looked over Postfix's main.cf to see if somehow TLS had been taken out of the config.  Everything looked fine there, key was specified, socket for SASL was specified, I noted that it was using Dovecot for SASL support.  Next, I checked to make sure Postfix was running on the right ports (25 and 587)... yep.  Netstat -tuap for port names, netstat -tuapn for port numbers.  So, on to telnet:

webmail:/etc/postfix# telnet 127.0.0.1 587
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)
EHLO TESTING
250-webmail.all-spec.com
250-PIPELINING
250-SIZE 20240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
QUIT
221 2.0.0 Bye
Connection closed by foreign host.


Oops, I don't speak TLS... but hey, at least it's listening and willing to talk there.  After googling for a few minutes, I come across openssl s_client, which can talk SSL for me, and check out the certificate. (From http://www.schwarzvogel.de/writings/ssl_tls_testing.html)

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -connect 127.0.0.1:587
CONNECTED(00000003)
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 2113 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 60EA2BD828D4EAC4D11D574148A390B5272216A38EA0AFD5F1D55DE8EEC9029B
    Session-ID-ctx: 
    Master-Key: 6838FE9CD4CD46FB92CAA3CB6CF2298BE1A51D1BB47D159B51983AACBA8FF4AEAECD1BD37DF3413FA7352B9C459312CB
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 75 00 f3 e3 46 ec 43 06-23 13 73 c7 db 4d 3b d6   u...F.C.#.s..M;.
    0010 - ef c0 f9 4c bc 1a 31 d8-c4 02 23 0b 82 10 4f 7f   ...L..1...#...O.
    0020 - 47 94 67 15 07 68 36 46-d6 62 cd 5f 10 17 57 d2   G.g..h6F.b._..W.
    0030 - e1 27 8f c0 89 22 99 16-be 23 d9 19 38 fd 80 b9   .'..."...#..8...
    0040 - 62 6b 70 b4 05 8a 7a 9d-c7 0b 2a a1 4e 40 81 d2   bkp...z...*.N@..
    0050 - 79 1f d2 ae 88 69 b6 7f-ba e1 62 7c b6 f7 de 0e   y....i....b|....
    0060 - 71 77 81 1d 40 c2 77 b2-b9 c1 37 df 40 35 79 c0   qw..@.w...7.@5y.
    0070 - b1 29 3e 6e c0 94 cc d0-87 a5 f0 1f 38 6a d7 a2   .)>n........8j..
    0080 - f9 5d 66 0d 9b f8 69 6a-d7 4d dd 2a 35 3e e4 1c   .]f...ij.M.*5>..
    0090 - 0f d7 b2 0f b5 33 e9 b6-d6 19 c0 8c 28 5e 95 0b   .....3......(^..
    00a0 - 39 41 6d d2 d2 ae eb 69-f2 4e 67 31 83 55 6f 5f   9Am....i.Ng1.Uo_

    Compression: 1 (zlib compression)
    Start Time: 1413832877
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
250 DSN
read:errno=0

For gnutls-cli, the Debian package is gnutls-bin.  I installed it, and ran it:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587
Resolving 'webmail.all-spec.com'...
Connecting to '192.168.0.4:587'...
^C

Wrong IP.  That was the server's old IP, before the network migration in May.  Ifconfig showed the right IP, as did dig.  Checked the /etc/hosts file... old IP.  Fix the hosts file, all is well.  So, once again, run gnutls-cli, and have a second terminal logged in (as root; sudo would work just as well, but we run as root here) to issue `killall -ALRM gnutls-cli` after telling the server STARTTLS, which causes the server to think handshaking has started and so it dumps out its certificate to gnutls-cli, which analyzes it and shows the results:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587
Resolving 'webmail.all-spec.com'...
Connecting to '172.16.150.47:587'...

- Simple Client Mode:

220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)
EHLO test
250-webmail.all-spec.com
250-PIPELINING
250-SIZE 20240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
STARTTLS
220 2.0.0 Ready to start TLS
*** Starting TLS handshake
- Ephemeral Diffie-Hellman parameters
 - Using prime: 1024 bits
 - Secret key: 1019 bits
 - Peer's public key: 1023 bits
- Certificate type: X.509
 - Got a certificate list of 1 certificates.
 - Certificate[0] info:
  - subject `C=US,ST=North Carolina,L=Wilmington,O=All-Spec Industries Inc,CN=webmail.all-spec.com', issuer `C=US,O=Thawte\, Inc.,CN=Thawte SSL CA', RSA key 2048 bits, signed using RSA-SHA1, activated `2014-01-22 00:00:00 UTC', expires `2016-02-19 23:59:59 UTC', SHA-1 fingerprint `8b72a2af8a317d70e97bf44b6e8a14e00bdf6a12'
- The hostname in the certificate matches 'webmail.all-spec.com'.
- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted
- Version: TLS1.2
- Key Exchange: DHE-RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
*** Fatal error: A TLS packet with unexpected length was received.
*** Server has terminated the connection abnormally.

Hmm... issuer unknown, although it says it right in the certificate, and the certificate is not trusted.  It didn't click in that there should be more than 1 certificate listed (hence, 'chain of trust' for certificates), but I moved on to looking at logs.

webmail:/etc/postfix# cat /var/log/mail.err
Oct 20 09:12:35 webmail postfix/smtpd[21508]: fatal: no SASL authentication mechanisms

... And that same error, repeated ad infinitum.  I'm concerned with the broken cert, so this doesn't seem helpful.  I run gnutls-cli and openssl s_client again, get no insight there, and start digging into Postfix.  To save time digging through main.cf and all the comments, I ran postconf -n, which gives the content of main.cf that postfix itself will actually read in, not the whole file.

webmail:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
home_mailbox = mail/
inet_interfaces = all
mailbox_command = /usr/lib/dovecot/deliver
mailbox_size_limit = 0
message_size_limit = 20240000
mydestination = webmail.all-spec.com, localhost.all-spec.com, , localhost
myhostname = webmail.all-spec.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = all-spec.com
readme_directory = no
recipient_delimiter = +
relayhost = [172.16.150.48]
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_recipient_restrictions = reject_invalid_hostname, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_rbl_client zen.dnsbl, permit
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_tls_cert_file = /etc/ssl/webmail.all-spec.com.cert
smtpd_tls_key_file = /etc/ssl/webmail.all-spec.com.key
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
transport_maps = hash:/etc/postfix/transport

TLS is turned on and pointing to the cert and key, and SASL is pointing to dovecot for the mechanism and /var/spool/postfix/private/auth for the socket.  So nothing's wrong there.  Maybe it's not looking in the right path for everything?

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -CApath /etc/ssl -connect webmail.all-spec.com:587
CONNECTED(00000003)
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 2113 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 62CF0930C4A751B8493D8B9CD0112BDFBF6EF8C945019313606F939174EEE16E
    Session-ID-ctx: 
    Master-Key: B6D8EDCF03939E39C6E3896CE82C9E4B12E46662B0AC1E64410D114B9E0BD18016E4EAB7AAF6E475C8FD77E1B42E3DB5
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 92 59 c2 c7 6f 0d a0 68-d9 f9 2f 46 5a 7c 42 c1   .Y..o..h../FZ|B.
    0010 - b9 55 0d 8f a7 98 76 68-5c 30 93 d3 d1 d1 ea 7e   .U....vh\0.....~
    0020 - 06 d2 59 7e 80 fc 60 9c-39 71 5a 84 09 e3 ed 48   ..Y~..`.9qZ....H
    0030 - 44 45 cb 8e e5 26 e3 fe-45 e4 f5 82 49 26 cb 0b   DE...&..E...I&..
    0040 - cc 79 20 46 99 41 70 bb-b3 b9 d1 a3 60 c3 84 67   .y F.Ap.....`..g
    0050 - 69 6b 1f 87 fe c4 56 56-47 d7 a2 99 5c 49 bd 10   ik....VVG...\I..
    0060 - dd 85 db 38 11 94 fb 76-77 15 1c 10 8e cf bb a3   ...8...vw.......
    0070 - 9a e2 a7 63 d3 a8 95 9e-72 dd 02 e4 b8 3a a6 01   ...c....r....:..
    0080 - c6 0d b0 3e aa 29 50 f5-4d 64 53 35 55 66 f5 de   ...>.)P.MdS5Uf..
    0090 - 09 24 52 9a cf 9a a1 94-0a ea 21 3f aa 6c af b3   .$R.......!?.l..
    00a0 - ab dc 9c 66 72 9f f9 72-ef 83 16 86 59 22 6e c1   ...fr..r....Y"n.

    Compression: 1 (zlib compression)
    Start Time: 1413901802
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
250 DSN
read:errno=0

Nope, same output as before.  Double-check the networking, since there had been issues with that earlier:

webmail:/etc/postfix# netstat -tuap | grep smtp
tcp        0      0 *:smtp                  *:*                     LISTEN      4400/master     
tcp6       0      0 [::]:smtp               [::]:*                  LISTEN      4400/master     
webmail:/etc/postfix# netstat -tuapn | grep 25
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      4400/master     
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      4400/master     
tcp        0      0 172.16.150.47:22        172.16.150.112:55579    ESTABLISHED 25596/1         
tcp6       0      0 :::25                   :::*                    LISTEN      4400/master     
webmail:/etc/postfix# netstat -tuapn | grep 587
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      4400/master     
tcp6       0      0 :::587                  :::*                    LISTEN      4400/master     
webmail:/etc/postfix# dig @8.8.8.8 webmail.all-spec.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @8.8.8.8 webmail.all-spec.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- 64663="" font="" id:="" noerror="" opcode:="" query="" status:="">
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;webmail.all-spec.com. IN A

;; ANSWER SECTION:
webmail.all-spec.com. 7199 IN A 97.66.29.211

;; Query time: 365 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Oct 21 11:29:09 2014
;; MSG SIZE  rcvd: 54

Perfect.  Ok, so what is in /etc/ssl?  There should be multiple certs there somewhere to verify the server key, right?
webmail:/etc/postfix# ls /etc/ssl
2009-2011     cabundle.crt.old private       webmail.all-spec.com.csr
2011-2013     certs webmail.all-spec.com.cert      webmail.all-spec.com.key
cabundle.crt  openssl.cnf webmail.all-spec.com.cert.old
webmail:/etc/postfix# ls /etc/ssl/certs/
ACEDICOM_Root.pem
AC_Raíz_Certicámara_S.A..pem
Actalis_Authentication_Root_CA.pem
AddTrust_External_Root.pem
AddTrust_Low-Value_Services_Root.pem
AddTrust_Public_Services_Root.pem
AddTrust_Qualified_Certificates_Root.pem
AffirmTrust_Commercial.pem
AffirmTrust_Networking.pem
AffirmTrust_Premium_ECC.pem
AffirmTrust_Premium.pem
America_Online_Root_Certification_Authority_1.pem
America_Online_Root_Certification_Authority_2.pem
ApplicationCA_-_Japanese_Government.pem
A-Trust-nQual-03.pem
Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
Baltimore_CyberTrust_Root.pem
ca-certificates.crt
cacert.org.pem
CA_Disig.pem
Camerfirma_Chambers_of_Commerce_Root.pem
Camerfirma_Global_Chambersign_Root.pem
ca.pem
Certigna.pem
Certinomis_-_Autorité_Racine.pem
Certplus_Class_2_Primary_CA.pem
certSIGN_ROOT_CA.pem
Certum_Root_CA.pem
Certum_Trusted_Network_CA.pem
Chambers_of_Commerce_Root_-_2008.pem
CNNIC_ROOT.pem
Comodo_AAA_Services_root.pem
COMODO_Certification_Authority.pem
COMODO_ECC_Certification_Authority.pem
Comodo_Secure_Services_root.pem
Comodo_Trusted_Services_root.pem
ComSign_CA.pem
ComSign_Secured_CA.pem
Cybertrust_Global_Root.pem
Deutsche_Telekom_Root_CA_2.pem
DigiCert_Assured_ID_Root_CA.pem
DigiCert_Global_Root_CA.pem
DigiCert_High_Assurance_EV_Root_CA.pem
Digital_Signature_Trust_Co._Global_CA_1.pem
Digital_Signature_Trust_Co._Global_CA_3.pem
dovecot.pem
DST_ACES_CA_X6.pem
DST_Root_CA_X3.pem
EBG_Elektronik_Sertifika_Hizmet_Sağlayıcısı.pem
EE_Certification_Centre_Root_CA.pem
E-Guven_Kok_Elektronik_Sertifika_Hizmet_Saglayicisi.pem
Entrust.net_Premium_2048_Secure_Server_CA.pem
Entrust.net_Secure_Server_CA.pem
Entrust_Root_Certification_Authority.pem
ePKI_Root_Certification_Authority.pem
Equifax_Secure_CA.pem
Equifax_Secure_eBusiness_CA_1.pem
Equifax_Secure_eBusiness_CA_2.pem
Equifax_Secure_Global_eBusiness_CA.pem
Firmaprofesional_Root_CA.pem
GeoTrust_Global_CA_2.pem
GeoTrust_Global_CA.pem
GeoTrust_Primary_Certification_Authority_-_G2.pem
GeoTrust_Primary_Certification_Authority_-_G3.pem
GeoTrust_Primary_Certification_Authority.pem
GeoTrust_Universal_CA_2.pem
GeoTrust_Universal_CA.pem
Global_Chambersign_Root_-_2008.pem
GlobalSign_Root_CA.pem
GlobalSign_Root_CA_-_R2.pem
GlobalSign_Root_CA_-_R3.pem
Go_Daddy_Class_2_CA.pem
Go_Daddy_Root_Certificate_Authority_-_G2.pem
GTE_CyberTrust_Global_Root.pem
Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
Hongkong_Post_Root_CA_1.pem
IGC_A.pem
Izenpe.com.pem
Juur-SK.pem
Microsec_e-Szigno_Root_CA_2009.pem
Microsec_e-Szigno_Root_CA.pem
NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
NetLock_Business_=Class_B=_Root.pem
NetLock_Express_=Class_C=_Root.pem
NetLock_Notary_=Class_A=_Root.pem
NetLock_Qualified_=Class_QA=_Root.pem
Network_Solutions_Certificate_Authority.pem
OISTE_WISeKey_Global_Root_GA_CA.pem
QuoVadis_Root_CA_2.pem
QuoVadis_Root_CA_3.pem
QuoVadis_Root_CA.pem
Root_CA_Generalitat_Valenciana.pem
RSA_Root_Certificate_1.pem
RSA_Security_2048_v3.pem
Secure_Global_CA.pem
SecureSign_RootCA11.pem
SecureTrust_CA.pem
Security_Communication_EV_RootCA1.pem
Security_Communication_RootCA2.pem
Security_Communication_Root_CA.pem
Sonera_Class_1_Root_CA.pem
Sonera_Class_2_Root_CA.pem
spi-ca-2003.pem
spi-cacert-2008.pem
ssl-cert-snakeoil.pem
Staat_der_Nederlanden_Root_CA_-_G2.pem
Staat_der_Nederlanden_Root_CA.pem
Starfield_Class_2_CA.pem
Starfield_Root_Certificate_Authority_-_G2.pem
Starfield_Services_Root_Certificate_Authority_-_G2.pem
StartCom_Certification_Authority_G2.pem
StartCom_Certification_Authority.pem
S-TRUST_Authentication_and_Encryption_Root_CA_2005_PN.pem
Swisscom_Root_CA_1.pem
SwissSign_Gold_CA_-_G2.pem
SwissSign_Platinum_CA_-_G2.pem
SwissSign_Silver_CA_-_G2.pem
Taiwan_GRCA.pem
TC_TrustCenter_Class_2_CA_II.pem
TC_TrustCenter_Class_3_CA_II.pem
TC_TrustCenter_Universal_CA_III.pem
TC_TrustCenter_Universal_CA_I.pem
TDC_Internet_Root_CA.pem
TDC_OCES_Root_CA.pem
Thawte_Premium_Server_CA.pem
thawte_Primary_Root_CA_-_G2.pem
thawte_Primary_Root_CA_-_G3.pem
thawte_Primary_Root_CA.pem
Thawte_Server_CA.pem
Trustis_FPS_Root_CA.pem
T-TeleSec_GlobalRoot_Class_3.pem
TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem
TURKTRUST_Certificate_Services_Provider_Root_1.pem
TURKTRUST_Certificate_Services_Provider_Root_2.pem
TWCA_Root_Certification_Authority.pem
UTN_DATACorp_SGC_Root_CA.pem
UTN_USERFirst_Email_Root_CA.pem
UTN_USERFirst_Hardware_Root_CA.pem
ValiCert_Class_1_VA.pem
ValiCert_Class_2_VA.pem
Verisign_Class_1_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem
Verisign_Class_1_Public_Primary_Certification_Authority.pem
Verisign_Class_2_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem
Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
Verisign_Class_3_Public_Primary_Certification_Authority.pem
Verisign_Class_4_Public_Primary_Certification_Authority_-_G3.pem
VeriSign_Universal_Root_Certification_Authority.pem
Visa_eCommerce_Root.pem
Wells_Fargo_Root_CA.pem
WellsSecure_Public_Root_Certificate_Authority.pem
XRamp_Global_CA_Root.pem

Good lord.  That's with the hashed certs snipped out, BTW.  But yes, looking through the list, there's Thawte certs there that should be the root cert(s) to verify the key.  So why is the key not verifying, and SASL not working?  I verified the Thawte root certs, and they all verify.  The server key still doesn't.

So I focused on the 'why is SASL not working' side of it, since the key not working shouldn't affect that.  Going by the last answer here: http://serverfault.com/questions/541662/postfix-returning-no-sasl-authentication-mechanisms-despite-mysql-auth-config I checked over my dovecot config for the socket setting:

webmail:~# dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-686-bigmem i686 Debian 7.6 
disable_plaintext_auth = no
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_location = maildir:~/mail
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
passdb {
  driver = pam
}
plugin {
  quota = maildir
  quota_rule = *:storage=50G
  quota_rule2 = .AllSpec:ignore
  quota_rule3 = .WebOrders:ignore
  quota_rule4 = .Trash:ignore
  quota_rule5 = .Sales:ignore
  quota_rule6 = .Orders:ignore
}
protocols = imap
ssl_ca =
ssl_cert =
ssl_key =
userdb {
  driver = passwd
}
protocol imap {
  mail_plugins = quota imap_quota
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
  mail_plugin_dir = /usr/lib/dovecot/modules/
  mail_plugins = sieve
  postmaster_address = postmaster@all-spec.com
  sendmail_path = /usr/lib/sendmail
}

No socket config.  Added it to 10-master.conf.  After several attempts to restart Dovecot to force the socket to appear, I finally added the socket settings to dovecot.conf... and discovered that it completely broke Dovecot and that I hadn't properly ported dovecot.conf from the 1.x syntax to 2.x way back when I'd upgraded the server.  So, after putting in a vanilla config file and reconfiguring it (and note, there's include lines in the new dovecot.conf to pull in files in conf.d), the socket was being generated.  Still no working SASL, however.  So I made sure everything from here (http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL): 

submission inet n - - - - smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_sasl_security_options=noanonymous
  -o smtpd_sasl_local_domain=$myhostname
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_login_maps=hash:/etc/postfix/virtual
  -o smtpd_sender_restrictions=reject_sender_login_mismatch
  -o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject

was included in master.cf.  Postfix restarted faster than it had been, and the 'no SASL mechanism' errors were gone by the time I got done reconfiguring Dovecot and Postfix.

So, now that the servers were ok, all that was left was to get the key to verify.  I took a while researching this, because I desperately did not want it to be the keys themselves, in case we had to buy new ones.  I tried moving the certificates down into /etc/ssl/certs to see if it was problem with finding the certs to verify against with the server certificate in the parent directory... nope.  After reading http://www.howtoforge.com/forums/showthread.php?t=66000http://www.thawte.nl/en/support/manuals/dovecot/dovecot+imap+server/install+certificate/http://openssl.6102.n7.nabble.com/Intermediate-root-CA-s-lost-and-confused-td8733.html, and http://www.fordfrog.com/2012/01/28/ssl-in-postfix-with-intermediate-certificate/, it seemed that the intermediate key was the problem.  Or a lack thereof, rather.  I'd seen some references in my earlier research to concatenating the certs together so there would be a definite cert chain there.  I looked at the cert I was working with, and lo and behold, there's only 1 cert listed in the file.  I asked my boss if there were any intermediate certs that were supposed to be in with the server cert (he handles buying certs), and he passed along the link Thawte.  No purchase necessary, since the cert was still valid.  I downloaded the bundle and looked at the certs... and the intermediate cert was not in /etc/ssl/certs or /etc/ssl.  So, I concatenated the intermediate cert and server cert together, and it worked:

webmail:/etc/ssl/certs# openssl s_client -starttls smtp -connect webmail.all-spec.com:587
CONNECTED(00000003)
depth=3 C = ZA, ST = Western Cape, L = Cape Town, O = Thawte Consulting cc, OU = Certification Services Division, CN = Thawte Premium Server CA, emailAddress = premium-server@thawte.com
verify return:1
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA
verify return:1
depth=1 C = US, O = "Thawte, Inc.", CN = Thawte SSL CA
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 3252 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0EE2B05739518E66B0E82490D51DBE23DDC481456FF447E7382288586ACBC0AF
    Session-ID-ctx: 
    Master-Key: F475FB98D8D4B80FAE196785BCAF77537327B4F8936A2E8CD1EF13343A930B9EA3B8B1D5AB63F761BC06DCEDD1CCE218
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - d1 d6 0d c2 2c a3 49 da-c0 5b f1 21 ef b4 21 dd   ....,.I..[.!..!.
    0010 - f0 0f fa 34 08 30 db 0e-a2 2b 84 3e 84 ee ca 70   ...4.0...+.>...p
    0020 - 05 f1 dc 15 9e bb f7 ec-f4 54 6d 56 6b 3b 00 7e   .........TmVk;.~
    0030 - 47 9f bd fb df 24 92 2e-9d 9c e7 f8 db 52 2d ab   G....$.......R-.
    0040 - c2 38 7f 77 86 b6 42 d7-77 9f 18 d0 cc be 6a 00   .8.w..B.w.....j.
    0050 - d2 b2 47 5c 34 aa 6b 37-1b 70 d2 f9 d2 38 40 fe   ..G\4.k7.p...8@.
    0060 - b9 38 c1 63 18 fe 1e 39-68 b1 98 cc 53 bd 1c 11   .8.c...9h...S...
    0070 - 09 2f 3c 1a 20 c9 bd d5-a7 bd e9 06 d5 8f f8 2a   ./<. ..........*
    0080 - 71 f2 8b b0 9b 06 59 ae-63 20 56 42 ad 7d 6b 94   q.....Y.c VB.}k.
    0090 - 39 7c 20 c9 26 1b 33 6f-da 95 05 6e 29 80 33 56   9| .&.3o...n).3V
    00a0 - 99 1f f8 e1 5b 82 04 26-db 7e bd c2 c0 75 c7 1e   ....[..&.~...u..

    Compression: 1 (zlib compression)
    Start Time: 1414009179
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
250 DSN
^C

A final note: if you run a webmail interface, you do have to restart Apache to get the webmail back working entirely correctly.  Specifically, Roundcube freaks out with all the above going on in the background, and decides it doesn't want to display new mail anymore.