Portswigger Academy SQL injection login bypass walkthrough

 This is the second of the labs for SQL Injection, the first section of the apprentice track in Portswigger Academy.  Like the first lab, it doesn't require Burp Suite, although you can use it.  I chose not to.

The lab website is a sample store.  Click the My Account link at the top of the page.

This takes you to a login page. 

In the Username box, type administrator'-- and in the password box, enter anything (the password will not get passed to the database).  That's it, you're now logged in as administrator, and the banner pops up congratulating you on completing the lab.

Copperhead, Job Hunting

 I'm trying to decide whether to even bother writing Copperhead.  It's supposed to run nmap -sV scans against hosts, then check exploitdb for known exploits.  The problem is, it takes a ton of text parsing and massaging to get something vague enough to search exploitdb without being too vague.  Oh, and it'd have to require the offline version of exploitdb and its tools, because I can't find a way to search exploitdb's website.

It almost seems like a solution in search of a problem.  So at the moment, it's on the back burner.

I'm also getting discouraged in my job hunt.  I'm not sure if I'm not putting myself out there well enough (too shy and stuttery, bad body language, etc.) or if my lack of experience is my problem.  I do have about 3 years of experience in security... as a sysadmin.  But because I didn't have "security" in my job title, that doesn't seem to count.  Plus that was 6 years ago.  I'm re-teaching myself everything I've forgotten about security over the years plus adding new skills, but... I don't know.  Job hunting is feeling futile at the moment.

Miscellany

 I've been doing little things here and there, but nothing worthy of a full blog post.  I'm trying to get in the habit of writing at least weekly, though.

I'm doing little things here and there on TryHackMe, but I'm running into more and more rooms that are premium-only.  Rooms that you need to complete to finish training paths, or even worse, to complete free rooms.  Looking over Hack The Box, it's similarly restricted for free users, plus it has time limits and no way around the time limits.  At least THM only limits your use of their in-browser attack box for free users, but VPN access is unlimited.  The $10/mo. doesn't sound like much for THM, but it adds up and I really can't justify it right now.

I'm still slowly working my way through "Violent Python", and it's inspired Copperhead and Pocketnuke over on my github.  Pocketnuke is a more refined version of the C runstub I mentioned previously in this blog.

I had been studying again for my CCNA, but that's fallen by the wayside for right now.  Instead, I'm slowly working on studying for the eJPT.  Or at least, doing INE's PTS courses through their free "starter pass".  I'm not sure I want to do pentesting, or jump right into Red Team, but at least in my opinion, you can't defend against the enemy properly unless you know how they think and operate.  So, like when I was in high school and hung out in 2600 IRC to learn how they would attack my server so I could protect it, I'm learning pentesting.

Book Reviews - "Defensive Security Handbook" and "Practical Cloud Security"

 To brush back up on security and learn new things that have become the standard in the past five years or so, I've bought a stack of books I'm working my way through.  The first two I decided to start with are "Defensive Security Handbook" by Brotherston and Berlin and "Practical Cloud Security" by Dotson.

"Defensive Security Handbook" built nicely upon what I already knew from doing security as a sysadmin and what was taught in the Security+ study materials.  It's basically a crash-course in how to set up good security policies and procedures if you're a sysadmin or something similar who has been handed a whole environment and been told "oh, by the way, we think this was secure, but you need to make sure and you need to bring the security up to date.".  Needless to say, I wish I'd had this book when I was at All-Spec, but it came out long after I'd left there.  If you find  yourself in a similar situation, or just want to make sure your security is right, or are trying to make the move from the infrastructure side of IT to the security side, I highly recommend this book.  I finished it in a week or so, and enjoyed it.  The writing is engaging as well as informative.  The book isn't meant to be a comprehensive security book, just a starting point with suggestions on where to go beyond what's covered in the book.  5/5

"Practical Cloud Security", conversely, was a slog.  The problem might be the fact that I have very little hands-on experience with cloud, but I kept finding myself getting bored by this book, and I'm not sure I retained much of the information I read.  It's definitely geared for someone facing a move to a cloud environment, or starting a position with a company that has a cloud environment, who's going "now what?".  I do like that for any specific cloud security features, the name of each feature in each cloud environment was listed.  I'll reserve judgement on the book for now, and just give it a 3/5.

Next up on the menu will be "Violent Python" and "Applied Incident Response".  Happy hacking.

Changing users with SUID and setuid/setgid

 Doing the Linux privilege escalation room on TryHackMe reminded me of a trick I've used in the past.  I haven't used it for nefarious purposes, only to get around a web script being run as the woefully unprivileged apache user, when it needed to run as a more privileged user.  It could easily be used as a good pentesting exploit.  I DO NOT endorse using this for evil.  Only use this for good, people.

It uses what I call a run stub.  It's a C wrapper that launches the script, but only after it changes the UID/GID to the desired higher-privileged user.  The run stub has to have SUID permissions and be owned by root.  So, chmod u+s runstub after compiling runstub.c into runstub.  The permissions listing should say: -rwsr-xr-x root root.

Below is the code: 

#include <stdio.h> 
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <string.h>
int uid;
int gid;
char runprog[512];
int main(int argc, char *argv[])
{
setgid(<gid>); /* <gid> is whatever GID you want to switch to */
setuid(<uid>);  /* <uid> is whatever UID you want to switch to */
snprintf(runprog, 512, "/path/to/script %s", argv[1]);  /* runprog is the buffer, 512 is the size of the buffer, the rest is the script you're trying to run and any arguments you want to pass to it, specified on the run stub's command line */
system(runprog);
return 0;
}

The two magic parts here are setuid()/setgid(), which *nix uses to switch users and groups in things like sudo and su, and system().  The snprintf() line formats the command into a string that can be passed to system(), which executes commands.

So, use your powers for good, and happy hacking.

It's been a while

 So, yeah, it's been a while since I posted.  I either haven't had anything interesting I've been working on, or things at work have been interesting to me, but not something I think is interesting to others or can be sufficiently sanitized.  I've also been busy with non-computer hobbies like gardening and writing fiction.

Now I'm trying to get (back) into cybersecurity, so I'll have interesting projects I'm doing on the side.  I got my Security+ cert about a month ago.  I bought a small library of books with my latest stimulus check that I'm slowly working my way through.  I've been doing some TryHackMe rooms, so I might do some writeups of those.  And I have a tool I'm starting on, an automated banner-grabber that looks up exploits in exploitdb.

Postfix, Dovecot SASL, and a broken cert

Just spent roughly 16-18 hours (Monday around 2 or 3 to Wednesday around 4) fixing a problem (well, 4 major problems and a minor problem) with the webmail server.  It only manifested as, mail can be received from the webmail server via IMAP, webmail server can receive mail from the servers that forward it mail, but mail cannot be sent through it.  It runs Postfix SMTP, and Dovecot for IMAP and SASL.  It's been broken for a few months, but it was going to be decommissioned for a while (that plan fell by the wayside), and it generally wasn't an issue until someone hit 'send read receipt' in Outlook instead of 'don't send'.  Outlook on this user's computer didn't have an Outbox for the webmail accounts, so there was no way to stop it from trying to send the message over and over except to fix the problem with the server.

First, I looked over the Dovecot config for glaring issues, since I remember the breakage happening around the time the server was upgraded from Debian Squeeze to Wheezy (Dovecot 1.x to 2.x).  I saw none.  I abandoned that route at that point because I didn't realize Dovecot was serving up SASL yet.

Next, I looked over Postfix's main.cf to see if somehow TLS had been taken out of the config.  Everything looked fine there, key was specified, socket for SASL was specified, I noted that it was using Dovecot for SASL support.  Next, I checked to make sure Postfix was running on the right ports (25 and 587)... yep.  Netstat -tuap for port names, netstat -tuapn for port numbers.  So, on to telnet:

webmail:/etc/postfix# telnet 127.0.0.1 587
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)
EHLO TESTING
250-webmail.all-spec.com
250-PIPELINING
250-SIZE 20240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
QUIT
221 2.0.0 Bye
Connection closed by foreign host.

Oops, I don't speak TLS... but hey, at least it's listening and willing to talk there.  After googling for a few minutes, I come across openssl s_client, which can talk SSL for me, and check out the certificate. (From http://www.schwarzvogel.de/writings/ssl_tls_testing.html)

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -connect 127.0.0.1:587

CONNECTED(00000003)

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=27:certificate not trusted

verify return:1

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=21:unable to verify the first certificate

verify return:1

---

Certificate chain

 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA

---

Server certificate

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA

---

No client certificate CA names sent

---

SSL handshake has read 2113 bytes and written 478 bytes

---

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: zlib compression

Expansion: zlib compression

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: 60EA2BD828D4EAC4D11D574148A390B5272216A38EA0AFD5F1D55DE8EEC9029B

    Session-ID-ctx: 

    Master-Key: 6838FE9CD4CD46FB92CAA3CB6CF2298BE1A51D1BB47D159B51983AACBA8FF4AEAECD1BD37DF3413FA7352B9C459312CB

    Key-Arg   : None

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    TLS session ticket lifetime hint: 3600 (seconds)

    TLS session ticket:

    0000 - 75 00 f3 e3 46 ec 43 06-23 13 73 c7 db 4d 3b d6   u...F.C.#.s..M;.

    0010 - ef c0 f9 4c bc 1a 31 d8-c4 02 23 0b 82 10 4f 7f   ...L..1...#...O.

    0020 - 47 94 67 15 07 68 36 46-d6 62 cd 5f 10 17 57 d2   G.g..h6F.b._..W.

    0030 - e1 27 8f c0 89 22 99 16-be 23 d9 19 38 fd 80 b9   .'..."...#..8...

    0040 - 62 6b 70 b4 05 8a 7a 9d-c7 0b 2a a1 4e 40 81 d2   bkp...z...*.N@..

    0050 - 79 1f d2 ae 88 69 b6 7f-ba e1 62 7c b6 f7 de 0e   y....i....b|....

    0060 - 71 77 81 1d 40 c2 77 b2-b9 c1 37 df 40 35 79 c0   qw..@.w...7.@5y.

    0070 - b1 29 3e 6e c0 94 cc d0-87 a5 f0 1f 38 6a d7 a2   .)>n........8j..

    0080 - f9 5d 66 0d 9b f8 69 6a-d7 4d dd 2a 35 3e e4 1c   .]f...ij.M.*5>..

    0090 - 0f d7 b2 0f b5 33 e9 b6-d6 19 c0 8c 28 5e 95 0b   .....3......(^..

    00a0 - 39 41 6d d2 d2 ae eb 69-f2 4e 67 31 83 55 6f 5f   9Am....i.Ng1.Uo_

    Compression: 1 (zlib compression)

    Start Time: 1413832877

    Timeout   : 300 (sec)

    Verify return code: 21 (unable to verify the first certificate)

---

250 DSN

read:errno=0

For gnutls-cli, the Debian package is gnutls-bin.  I installed it, and ran it:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587

Resolving 'webmail.all-spec.com'...

Connecting to '192.168.0.4:587'...

^C

Wrong IP.  That was the server's old IP, before the network migration in May.  Ifconfig showed the right IP, as did dig.  Checked the /etc/hosts file... old IP.  Fix the hosts file, all is well.  So, once again, run gnutls-cli, and have a second terminal logged in (as root; sudo would work just as well, but we run as root here) to issue `killall -ALRM gnutls-cli` after telling the server STARTTLS, which causes the server to think handshaking has started and so it dumps out its certificate to gnutls-cli, which analyzes it and shows the results:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587

Resolving 'webmail.all-spec.com'...

Connecting to '172.16.150.47:587'...

- Simple Client Mode:

220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)

EHLO test

250-webmail.all-spec.com

250-PIPELINING

250-SIZE 20240000

250-VRFY

250-ETRN

250-STARTTLS

250-ENHANCEDSTATUSCODES

250-8BITMIME

250 DSN

STARTTLS

220 2.0.0 Ready to start TLS

*** Starting TLS handshake

- Ephemeral Diffie-Hellman parameters

 - Using prime: 1024 bits

 - Secret key: 1019 bits

 - Peer's public key: 1023 bits

- Certificate type: X.509

 - Got a certificate list of 1 certificates.

 - Certificate[0] info:

  - subject `C=US,ST=North Carolina,L=Wilmington,O=All-Spec Industries Inc,CN=webmail.all-spec.com', issuer `C=US,O=Thawte\, Inc.,CN=Thawte SSL CA', RSA key 2048 bits, signed using RSA-SHA1, activated `2014-01-22 00:00:00 UTC', expires `2016-02-19 23:59:59 UTC', SHA-1 fingerprint `8b72a2af8a317d70e97bf44b6e8a14e00bdf6a12'

- The hostname in the certificate matches 'webmail.all-spec.com'.

- Peer's certificate issuer is unknown

- Peer's certificate is NOT trusted

- Version: TLS1.2

- Key Exchange: DHE-RSA

- Cipher: AES-128-CBC

- MAC: SHA1

- Compression: NULL

*** Fatal error: A TLS packet with unexpected length was received.

*** Server has terminated the connection abnormally.

Hmm... issuer unknown, although it says it right in the certificate, and the certificate is not trusted.  It didn't click in that there should be more than 1 certificate listed (hence, 'chain of trust' for certificates), but I moved on to looking at logs.

webmail:/etc/postfix# cat /var/log/mail.err

Oct 20 09:12:35 webmail postfix/smtpd[21508]: fatal: no SASL authentication mechanisms

... And that same error, repeated ad infinitum.  I'm concerned with the broken cert, so this doesn't seem helpful.  I run gnutls-cli and openssl s_client again, get no insight there, and start digging into Postfix.  To save time digging through main.cf and all the comments, I ran postconf -n, which gives the content of main.cf that postfix itself will actually read in, not the whole file.

webmail:/etc/postfix# postconf -n

alias_database = hash:/etc/aliases

alias_maps = hash:/etc/aliases

append_dot_mydomain = no

biff = no

config_directory = /etc/postfix

content_filter = smtp-amavis:[127.0.0.1]:10024

home_mailbox = mail/

inet_interfaces = all

mailbox_command = /usr/lib/dovecot/deliver

mailbox_size_limit = 0

message_size_limit = 20240000

mydestination = webmail.all-spec.com, localhost.all-spec.com, , localhost

myhostname = webmail.all-spec.com

mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

myorigin = all-spec.com

readme_directory = no

recipient_delimiter = +

relayhost = [172.16.150.48]

smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)

smtpd_recipient_restrictions = reject_invalid_hostname, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_rbl_client zen.dnsbl, permit

smtpd_sasl_path = private/auth

smtpd_sasl_type = dovecot

smtpd_tls_cert_file = /etc/ssl/webmail.all-spec.com.cert

smtpd_tls_key_file = /etc/ssl/webmail.all-spec.com.key

smtpd_tls_security_level = may

smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

transport_maps = hash:/etc/postfix/transport

TLS is turned on and pointing to the cert and key, and SASL is pointing to dovecot for the mechanism and /var/spool/postfix/private/auth for the socket.  So nothing's wrong there.  Maybe it's not looking in the right path for everything?

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -CApath /etc/ssl -connect webmail.all-spec.com:587

CONNECTED(00000003)

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=27:certificate not trusted

verify return:1

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify error:num=21:unable to verify the first certificate

verify return:1

---

Certificate chain

 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA

---

Server certificate

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA

---

No client certificate CA names sent

---

SSL handshake has read 2113 bytes and written 478 bytes

---

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: zlib compression

Expansion: zlib compression

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: 62CF0930C4A751B8493D8B9CD0112BDFBF6EF8C945019313606F939174EEE16E

    Session-ID-ctx: 

    Master-Key: B6D8EDCF03939E39C6E3896CE82C9E4B12E46662B0AC1E64410D114B9E0BD18016E4EAB7AAF6E475C8FD77E1B42E3DB5

    Key-Arg   : None

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    TLS session ticket lifetime hint: 3600 (seconds)

    TLS session ticket:

    0000 - 92 59 c2 c7 6f 0d a0 68-d9 f9 2f 46 5a 7c 42 c1   .Y..o..h../FZ|B.

    0010 - b9 55 0d 8f a7 98 76 68-5c 30 93 d3 d1 d1 ea 7e   .U....vh\0.....~

    0020 - 06 d2 59 7e 80 fc 60 9c-39 71 5a 84 09 e3 ed 48   ..Y~..`.9qZ....H

    0030 - 44 45 cb 8e e5 26 e3 fe-45 e4 f5 82 49 26 cb 0b   DE...&..E...I&..

    0040 - cc 79 20 46 99 41 70 bb-b3 b9 d1 a3 60 c3 84 67   .y F.Ap.....`..g

    0050 - 69 6b 1f 87 fe c4 56 56-47 d7 a2 99 5c 49 bd 10   ik....VVG...\I..

    0060 - dd 85 db 38 11 94 fb 76-77 15 1c 10 8e cf bb a3   ...8...vw.......

    0070 - 9a e2 a7 63 d3 a8 95 9e-72 dd 02 e4 b8 3a a6 01   ...c....r....:..

    0080 - c6 0d b0 3e aa 29 50 f5-4d 64 53 35 55 66 f5 de   ...>.)P.MdS5Uf..

    0090 - 09 24 52 9a cf 9a a1 94-0a ea 21 3f aa 6c af b3   .$R.......!?.l..

    00a0 - ab dc 9c 66 72 9f f9 72-ef 83 16 86 59 22 6e c1   ...fr..r....Y"n.

    Compression: 1 (zlib compression)

    Start Time: 1413901802

    Timeout   : 300 (sec)

    Verify return code: 21 (unable to verify the first certificate)

---

250 DSN

read:errno=0

Nope, same output as before.  Double-check the networking, since there had been issues with that earlier:

webmail:/etc/postfix# netstat -tuap | grep smtp

tcp        0      0 *:smtp                  *:*                     LISTEN      4400/master     

tcp6       0      0 [::]:smtp               [::]:*                  LISTEN      4400/master     

webmail:/etc/postfix# netstat -tuapn | grep 25

tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      4400/master     

tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      4400/master     

tcp        0      0 172.16.150.47:22        172.16.150.112:55579    ESTABLISHED 25596/1         

tcp6       0      0 :::25                   :::*                    LISTEN      4400/master     

webmail:/etc/postfix# netstat -tuapn | grep 587

tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      4400/master     

tcp6       0      0 :::587                  :::*                    LISTEN      4400/master     

webmail:/etc/postfix# dig @8.8.8.8 webmail.all-spec.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @8.8.8.8 webmail.all-spec.com

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- 64663="" font="" id:="" noerror="" opcode:="" query="" status:="">

;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:

;webmail.all-spec.com. IN A

;; ANSWER SECTION:

webmail.all-spec.com. 7199 IN A 97.66.29.211

;; Query time: 365 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Tue Oct 21 11:29:09 2014

;; MSG SIZE  rcvd: 54

Perfect.  Ok, so what is in /etc/ssl?  There should be multiple certs there somewhere to verify the server key, right?

webmail:/etc/postfix# ls /etc/ssl

2009-2011     cabundle.crt.old private       webmail.all-spec.com.csr

2011-2013     certs webmail.all-spec.com.cert      webmail.all-spec.com.key

cabundle.crt  openssl.cnf webmail.all-spec.com.cert.old

webmail:/etc/postfix# ls /etc/ssl/certs/

ACEDICOM_Root.pem

AC_Raíz_Certicámara_S.A..pem

Actalis_Authentication_Root_CA.pem

AddTrust_External_Root.pem

AddTrust_Low-Value_Services_Root.pem

AddTrust_Public_Services_Root.pem

AddTrust_Qualified_Certificates_Root.pem

AffirmTrust_Commercial.pem

AffirmTrust_Networking.pem

AffirmTrust_Premium_ECC.pem

AffirmTrust_Premium.pem

America_Online_Root_Certification_Authority_1.pem

America_Online_Root_Certification_Authority_2.pem

ApplicationCA_-_Japanese_Government.pem

A-Trust-nQual-03.pem

Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem

Baltimore_CyberTrust_Root.pem

ca-certificates.crt

cacert.org.pem

CA_Disig.pem

Camerfirma_Chambers_of_Commerce_Root.pem

Camerfirma_Global_Chambersign_Root.pem

ca.pem

Certigna.pem

Certinomis_-_Autorité_Racine.pem

Certplus_Class_2_Primary_CA.pem

certSIGN_ROOT_CA.pem

Certum_Root_CA.pem

Certum_Trusted_Network_CA.pem

Chambers_of_Commerce_Root_-_2008.pem

CNNIC_ROOT.pem

Comodo_AAA_Services_root.pem

COMODO_Certification_Authority.pem

COMODO_ECC_Certification_Authority.pem

Comodo_Secure_Services_root.pem

Comodo_Trusted_Services_root.pem

ComSign_CA.pem

ComSign_Secured_CA.pem

Cybertrust_Global_Root.pem

Deutsche_Telekom_Root_CA_2.pem

DigiCert_Assured_ID_Root_CA.pem

DigiCert_Global_Root_CA.pem

DigiCert_High_Assurance_EV_Root_CA.pem

Digital_Signature_Trust_Co._Global_CA_1.pem

Digital_Signature_Trust_Co._Global_CA_3.pem

dovecot.pem

DST_ACES_CA_X6.pem

DST_Root_CA_X3.pem

EBG_Elektronik_Sertifika_Hizmet_Sağlayıcısı.pem

EE_Certification_Centre_Root_CA.pem

E-Guven_Kok_Elektronik_Sertifika_Hizmet_Saglayicisi.pem

Entrust.net_Premium_2048_Secure_Server_CA.pem

Entrust.net_Secure_Server_CA.pem

Entrust_Root_Certification_Authority.pem

ePKI_Root_Certification_Authority.pem

Equifax_Secure_CA.pem

Equifax_Secure_eBusiness_CA_1.pem

Equifax_Secure_eBusiness_CA_2.pem

Equifax_Secure_Global_eBusiness_CA.pem

Firmaprofesional_Root_CA.pem

GeoTrust_Global_CA_2.pem

GeoTrust_Global_CA.pem

GeoTrust_Primary_Certification_Authority_-_G2.pem

GeoTrust_Primary_Certification_Authority_-_G3.pem

GeoTrust_Primary_Certification_Authority.pem

GeoTrust_Universal_CA_2.pem

GeoTrust_Universal_CA.pem

Global_Chambersign_Root_-_2008.pem

GlobalSign_Root_CA.pem

GlobalSign_Root_CA_-_R2.pem

GlobalSign_Root_CA_-_R3.pem

Go_Daddy_Class_2_CA.pem

Go_Daddy_Root_Certificate_Authority_-_G2.pem

GTE_CyberTrust_Global_Root.pem

Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem

Hongkong_Post_Root_CA_1.pem

IGC_A.pem

Izenpe.com.pem

Juur-SK.pem

Microsec_e-Szigno_Root_CA_2009.pem

Microsec_e-Szigno_Root_CA.pem

NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem

NetLock_Business_=Class_B=_Root.pem

NetLock_Express_=Class_C=_Root.pem

NetLock_Notary_=Class_A=_Root.pem

NetLock_Qualified_=Class_QA=_Root.pem

Network_Solutions_Certificate_Authority.pem

OISTE_WISeKey_Global_Root_GA_CA.pem

QuoVadis_Root_CA_2.pem

QuoVadis_Root_CA_3.pem

QuoVadis_Root_CA.pem

Root_CA_Generalitat_Valenciana.pem

RSA_Root_Certificate_1.pem

RSA_Security_2048_v3.pem

Secure_Global_CA.pem

SecureSign_RootCA11.pem

SecureTrust_CA.pem

Security_Communication_EV_RootCA1.pem

Security_Communication_RootCA2.pem

Security_Communication_Root_CA.pem

Sonera_Class_1_Root_CA.pem

Sonera_Class_2_Root_CA.pem

spi-ca-2003.pem

spi-cacert-2008.pem

ssl-cert-snakeoil.pem

Staat_der_Nederlanden_Root_CA_-_G2.pem

Staat_der_Nederlanden_Root_CA.pem

Starfield_Class_2_CA.pem

Starfield_Root_Certificate_Authority_-_G2.pem

Starfield_Services_Root_Certificate_Authority_-_G2.pem

StartCom_Certification_Authority_G2.pem

StartCom_Certification_Authority.pem

S-TRUST_Authentication_and_Encryption_Root_CA_2005_PN.pem

Swisscom_Root_CA_1.pem

SwissSign_Gold_CA_-_G2.pem

SwissSign_Platinum_CA_-_G2.pem

SwissSign_Silver_CA_-_G2.pem

Taiwan_GRCA.pem

TC_TrustCenter_Class_2_CA_II.pem

TC_TrustCenter_Class_3_CA_II.pem

TC_TrustCenter_Universal_CA_III.pem

TC_TrustCenter_Universal_CA_I.pem

TDC_Internet_Root_CA.pem

TDC_OCES_Root_CA.pem

Thawte_Premium_Server_CA.pem

thawte_Primary_Root_CA_-_G2.pem

thawte_Primary_Root_CA_-_G3.pem

thawte_Primary_Root_CA.pem

Thawte_Server_CA.pem

Trustis_FPS_Root_CA.pem

T-TeleSec_GlobalRoot_Class_3.pem

TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem

TURKTRUST_Certificate_Services_Provider_Root_1.pem

TURKTRUST_Certificate_Services_Provider_Root_2.pem

TWCA_Root_Certification_Authority.pem

UTN_DATACorp_SGC_Root_CA.pem

UTN_USERFirst_Email_Root_CA.pem

UTN_USERFirst_Hardware_Root_CA.pem

ValiCert_Class_1_VA.pem

ValiCert_Class_2_VA.pem

Verisign_Class_1_Public_Primary_Certification_Authority_-_G2.pem

Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem

Verisign_Class_1_Public_Primary_Certification_Authority.pem

Verisign_Class_2_Public_Primary_Certification_Authority_-_G2.pem

Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem

Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem

Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem

VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem

VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem

Verisign_Class_3_Public_Primary_Certification_Authority.pem

Verisign_Class_4_Public_Primary_Certification_Authority_-_G3.pem

VeriSign_Universal_Root_Certification_Authority.pem

Visa_eCommerce_Root.pem

Wells_Fargo_Root_CA.pem

WellsSecure_Public_Root_Certificate_Authority.pem

XRamp_Global_CA_Root.pem

Good lord.  That's with the hashed certs snipped out, BTW.  But yes, looking through the list, there's Thawte certs there that should be the root cert(s) to verify the key.  So why is the key not verifying, and SASL not working?  I verified the Thawte root certs, and they all verify.  The server key still doesn't.

So I focused on the 'why is SASL not working' side of it, since the key not working shouldn't affect that.  Going by the last answer here: http://serverfault.com/questions/541662/postfix-returning-no-sasl-authentication-mechanisms-despite-mysql-auth-config I checked over my dovecot config for the socket setting:

webmail:~# dovecot -n

# 2.1.7: /etc/dovecot/dovecot.conf

# OS: Linux 2.6.32-5-686-bigmem i686 Debian 7.6 

disable_plaintext_auth = no

log_timestamp = "%Y-%m-%d %H:%M:%S "

mail_location = maildir:~/mail

mail_privileged_group = mail

managesieve_notify_capability = mailto

managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave

passdb {

  driver = pam

}

plugin {

  quota = maildir

  quota_rule = *:storage=50G

  quota_rule2 = .AllSpec:ignore

  quota_rule3 = .WebOrders:ignore

  quota_rule4 = .Trash:ignore

  quota_rule5 = .Sales:ignore

  quota_rule6 = .Orders:ignore

}

protocols = imap

ssl_ca =

ssl_cert =

ssl_key =

userdb {

  driver = passwd

}

protocol imap {

  mail_plugins = quota imap_quota

}

protocol pop3 {

  pop3_uidl_format = %08Xu%08Xv

}

protocol lda {

  mail_plugin_dir = /usr/lib/dovecot/modules/

  mail_plugins = sieve

  postmaster_address = postmaster@all-spec.com

  sendmail_path = /usr/lib/sendmail

}

No socket config.  Added it to 10-master.conf.  After several attempts to restart Dovecot to force the socket to appear, I finally added the socket settings to dovecot.conf... and discovered that it completely broke Dovecot and that I hadn't properly ported dovecot.conf from the 1.x syntax to 2.x way back when I'd upgraded the server.  So, after putting in a vanilla config file and reconfiguring it (and note, there's include lines in the new dovecot.conf to pull in files in conf.d), the socket was being generated.  Still no working SASL, however.  So I made sure everything from here (http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL): 

submission inet n - - - - smtpd

  -o smtpd_tls_security_level=encrypt

  -o smtpd_sasl_auth_enable=yes

  -o smtpd_sasl_type=dovecot

  -o smtpd_sasl_path=private/auth

  -o smtpd_sasl_security_options=noanonymous

  -o smtpd_sasl_local_domain=$myhostname

  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

  -o smtpd_sender_login_maps=hash:/etc/postfix/virtual

  -o smtpd_sender_restrictions=reject_sender_login_mismatch

  -o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject

was included in master.cf.  Postfix restarted faster than it had been, and the 'no SASL mechanism' errors were gone by the time I got done reconfiguring Dovecot and Postfix.

So, now that the servers were ok, all that was left was to get the key to verify.  I took a while researching this, because I desperately did not want it to be the keys themselves, in case we had to buy new ones.  I tried moving the certificates down into /etc/ssl/certs to see if it was problem with finding the certs to verify against with the server certificate in the parent directory... nope.  After reading http://www.howtoforge.com/forums/showthread.php?t=66000, http://www.thawte.nl/en/support/manuals/dovecot/dovecot+imap+server/install+certificate/, http://openssl.6102.n7.nabble.com/Intermediate-root-CA-s-lost-and-confused-td8733.html, and http://www.fordfrog.com/2012/01/28/ssl-in-postfix-with-intermediate-certificate/, it seemed that the intermediate key was the problem.  Or a lack thereof, rather.  I'd seen some references in my earlier research to concatenating the certs together so there would be a definite cert chain there.  I looked at the cert I was working with, and lo and behold, there's only 1 cert listed in the file.  I asked my boss if there were any intermediate certs that were supposed to be in with the server cert (he handles buying certs), and he passed along the link Thawte.  No purchase necessary, since the cert was still valid.  I downloaded the bundle and looked at the certs... and the intermediate cert was not in /etc/ssl/certs or /etc/ssl.  So, I concatenated the intermediate cert and server cert together, and it worked:

webmail:/etc/ssl/certs# openssl s_client -starttls smtp -connect webmail.all-spec.com:587

CONNECTED(00000003)

depth=3 C = ZA, ST = Western Cape, L = Cape Town, O = Thawte Consulting cc, OU = Certification Services Division, CN = Thawte Premium Server CA, emailAddress = premium-server@thawte.com

verify return:1

depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA

verify return:1

depth=1 C = US, O = "Thawte, Inc.", CN = Thawte SSL CA

verify return:1

depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com

verify return:1

---

Certificate chain

 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

 1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA

   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

---

Server certificate

subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com

issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA

---

No client certificate CA names sent

---

SSL handshake has read 3252 bytes and written 478 bytes

---

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: zlib compression

Expansion: zlib compression

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: 0EE2B05739518E66B0E82490D51DBE23DDC481456FF447E7382288586ACBC0AF

    Session-ID-ctx: 

    Master-Key: F475FB98D8D4B80FAE196785BCAF77537327B4F8936A2E8CD1EF13343A930B9EA3B8B1D5AB63F761BC06DCEDD1CCE218

    Key-Arg   : None

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    TLS session ticket lifetime hint: 3600 (seconds)

    TLS session ticket:

    0000 - d1 d6 0d c2 2c a3 49 da-c0 5b f1 21 ef b4 21 dd   ....,.I..[.!..!.

    0010 - f0 0f fa 34 08 30 db 0e-a2 2b 84 3e 84 ee ca 70   ...4.0...+.>...p

    0020 - 05 f1 dc 15 9e bb f7 ec-f4 54 6d 56 6b 3b 00 7e   .........TmVk;.~

    0030 - 47 9f bd fb df 24 92 2e-9d 9c e7 f8 db 52 2d ab   G....$.......R-.

    0040 - c2 38 7f 77 86 b6 42 d7-77 9f 18 d0 cc be 6a 00   .8.w..B.w.....j.

    0050 - d2 b2 47 5c 34 aa 6b 37-1b 70 d2 f9 d2 38 40 fe   ..G\4.k7.p...8@.

    0060 - b9 38 c1 63 18 fe 1e 39-68 b1 98 cc 53 bd 1c 11   .8.c...9h...S...

    0070 - 09 2f 3c 1a 20 c9 bd d5-a7 bd e9 06 d5 8f f8 2a   ./<. ..........*

    0080 - 71 f2 8b b0 9b 06 59 ae-63 20 56 42 ad 7d 6b 94   q.....Y.c VB.}k.

    0090 - 39 7c 20 c9 26 1b 33 6f-da 95 05 6e 29 80 33 56   9| .&.3o...n).3V

    00a0 - 99 1f f8 e1 5b 82 04 26-db 7e bd c2 c0 75 c7 1e   ....[..&.~...u..

    Compression: 1 (zlib compression)

    Start Time: 1414009179

    Timeout   : 300 (sec)

    Verify return code: 0 (ok)

---

250 DSN

^C

A final note: if you run a webmail interface, you do have to restart Apache to get the webmail back working entirely correctly.  Specifically, Roundcube freaks out with all the above going on in the background, and decides it doesn't want to display new mail anymore.