Search This Blog

Loading...

10.24.2014

Postfix, Dovecot SASL, and a broken cert

Just spent roughly 16-18 hours (Monday around 2 or 3 to Wednesday around 4) fixing a problem (well, 4 major problems and a minor problem) with the webmail server.  It only manifested as, mail can be received from the webmail server via IMAP, webmail server can receive mail from the servers that forward it mail, but mail cannot be sent through it.  It runs Postfix SMTP, and Dovecot for IMAP and SASL.  It's been broken for a few months, but it was going to be decommissioned for a while (that plan fell by the wayside), and it generally wasn't an issue until someone hit 'send read receipt' in Outlook instead of 'don't send'.  Outlook on this user's computer didn't have an Outbox for the webmail accounts, so there was no way to stop it from trying to send the message over and over except to fix the problem with the server.

First, I looked over the Dovecot config for glaring issues, since I remember the breakage happening around the time the server was upgraded from Debian Squeeze to Wheezy (Dovecot 1.x to 2.x).  I saw none.  I abandoned that route at that point because I didn't realize Dovecot was serving up SASL yet.

Next, I looked over Postfix's main.cf to see if somehow TLS had been taken out of the config.  Everything looked fine there, key was specified, socket for SASL was specified, I noted that it was using Dovecot for SASL support.  Next, I checked to make sure Postfix was running on the right ports (25 and 587)... yep.  Netstat -tuap for port names, netstat -tuapn for port numbers.  So, on to telnet:

webmail:/etc/postfix# telnet 127.0.0.1 587
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)
EHLO TESTING
250-webmail.all-spec.com
250-PIPELINING
250-SIZE 20240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
QUIT
221 2.0.0 Bye
Connection closed by foreign host.


Oops, I don't speak TLS... but hey, at least it's listening and willing to talk there.  After googling for a few minutes, I come across openssl s_client, which can talk SSL for me, and check out the certificate. (From http://www.schwarzvogel.de/writings/ssl_tls_testing.html)

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -connect 127.0.0.1:587
CONNECTED(00000003)
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 2113 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 60EA2BD828D4EAC4D11D574148A390B5272216A38EA0AFD5F1D55DE8EEC9029B
    Session-ID-ctx: 
    Master-Key: 6838FE9CD4CD46FB92CAA3CB6CF2298BE1A51D1BB47D159B51983AACBA8FF4AEAECD1BD37DF3413FA7352B9C459312CB
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 75 00 f3 e3 46 ec 43 06-23 13 73 c7 db 4d 3b d6   u...F.C.#.s..M;.
    0010 - ef c0 f9 4c bc 1a 31 d8-c4 02 23 0b 82 10 4f 7f   ...L..1...#...O.
    0020 - 47 94 67 15 07 68 36 46-d6 62 cd 5f 10 17 57 d2   G.g..h6F.b._..W.
    0030 - e1 27 8f c0 89 22 99 16-be 23 d9 19 38 fd 80 b9   .'..."...#..8...
    0040 - 62 6b 70 b4 05 8a 7a 9d-c7 0b 2a a1 4e 40 81 d2   bkp...z...*.N@..
    0050 - 79 1f d2 ae 88 69 b6 7f-ba e1 62 7c b6 f7 de 0e   y....i....b|....
    0060 - 71 77 81 1d 40 c2 77 b2-b9 c1 37 df 40 35 79 c0   qw..@.w...7.@5y.
    0070 - b1 29 3e 6e c0 94 cc d0-87 a5 f0 1f 38 6a d7 a2   .)>n........8j..
    0080 - f9 5d 66 0d 9b f8 69 6a-d7 4d dd 2a 35 3e e4 1c   .]f...ij.M.*5>..
    0090 - 0f d7 b2 0f b5 33 e9 b6-d6 19 c0 8c 28 5e 95 0b   .....3......(^..
    00a0 - 39 41 6d d2 d2 ae eb 69-f2 4e 67 31 83 55 6f 5f   9Am....i.Ng1.Uo_

    Compression: 1 (zlib compression)
    Start Time: 1413832877
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
250 DSN
read:errno=0

For gnutls-cli, the Debian package is gnutls-bin.  I installed it, and ran it:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587
Resolving 'webmail.all-spec.com'...
Connecting to '192.168.0.4:587'...
^C

Wrong IP.  That was the server's old IP, before the network migration in May.  Ifconfig showed the right IP, as did dig.  Checked the /etc/hosts file... old IP.  Fix the hosts file, all is well.  So, once again, run gnutls-cli, and have a second terminal logged in (as root; sudo would work just as well, but we run as root here) to issue `killall -ALRM gnutls-cli` after telling the server STARTTLS, which causes the server to think handshaking has started and so it dumps out its certificate to gnutls-cli, which analyzes it and shows the results:

webmail:/etc/postfix# gnutls-cli -s webmail.all-spec.com -p 587
Resolving 'webmail.all-spec.com'...
Connecting to '172.16.150.47:587'...

- Simple Client Mode:

220 webmail.all-spec.com ESMTP Postfix (Debian/GNU)
EHLO test
250-webmail.all-spec.com
250-PIPELINING
250-SIZE 20240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
STARTTLS
220 2.0.0 Ready to start TLS
*** Starting TLS handshake
- Ephemeral Diffie-Hellman parameters
 - Using prime: 1024 bits
 - Secret key: 1019 bits
 - Peer's public key: 1023 bits
- Certificate type: X.509
 - Got a certificate list of 1 certificates.
 - Certificate[0] info:
  - subject `C=US,ST=North Carolina,L=Wilmington,O=All-Spec Industries Inc,CN=webmail.all-spec.com', issuer `C=US,O=Thawte\, Inc.,CN=Thawte SSL CA', RSA key 2048 bits, signed using RSA-SHA1, activated `2014-01-22 00:00:00 UTC', expires `2016-02-19 23:59:59 UTC', SHA-1 fingerprint `8b72a2af8a317d70e97bf44b6e8a14e00bdf6a12'
- The hostname in the certificate matches 'webmail.all-spec.com'.
- Peer's certificate issuer is unknown
- Peer's certificate is NOT trusted
- Version: TLS1.2
- Key Exchange: DHE-RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
*** Fatal error: A TLS packet with unexpected length was received.
*** Server has terminated the connection abnormally.

Hmm... issuer unknown, although it says it right in the certificate, and the certificate is not trusted.  It didn't click in that there should be more than 1 certificate listed (hence, 'chain of trust' for certificates), but I moved on to looking at logs.

webmail:/etc/postfix# cat /var/log/mail.err
Oct 20 09:12:35 webmail postfix/smtpd[21508]: fatal: no SASL authentication mechanisms

... And that same error, repeated ad infinitum.  I'm concerned with the broken cert, so this doesn't seem helpful.  I run gnutls-cli and openssl s_client again, get no insight there, and start digging into Postfix.  To save time digging through main.cf and all the comments, I ran postconf -n, which gives the content of main.cf that postfix itself will actually read in, not the whole file.

webmail:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
home_mailbox = mail/
inet_interfaces = all
mailbox_command = /usr/lib/dovecot/deliver
mailbox_size_limit = 0
message_size_limit = 20240000
mydestination = webmail.all-spec.com, localhost.all-spec.com, , localhost
myhostname = webmail.all-spec.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = all-spec.com
readme_directory = no
recipient_delimiter = +
relayhost = [172.16.150.48]
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_recipient_restrictions = reject_invalid_hostname, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_rbl_client zen.dnsbl, permit
smtpd_sasl_path = private/auth
smtpd_sasl_type = dovecot
smtpd_tls_cert_file = /etc/ssl/webmail.all-spec.com.cert
smtpd_tls_key_file = /etc/ssl/webmail.all-spec.com.key
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
transport_maps = hash:/etc/postfix/transport

TLS is turned on and pointing to the cert and key, and SASL is pointing to dovecot for the mechanism and /var/spool/postfix/private/auth for the socket.  So nothing's wrong there.  Maybe it's not looking in the right path for everything?

webmail:/etc/postfix# openssl s_client -starttls smtp -crlf -CApath /etc/ssl -connect webmail.all-spec.com:587
CONNECTED(00000003)
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 2113 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 62CF0930C4A751B8493D8B9CD0112BDFBF6EF8C945019313606F939174EEE16E
    Session-ID-ctx: 
    Master-Key: B6D8EDCF03939E39C6E3896CE82C9E4B12E46662B0AC1E64410D114B9E0BD18016E4EAB7AAF6E475C8FD77E1B42E3DB5
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 92 59 c2 c7 6f 0d a0 68-d9 f9 2f 46 5a 7c 42 c1   .Y..o..h../FZ|B.
    0010 - b9 55 0d 8f a7 98 76 68-5c 30 93 d3 d1 d1 ea 7e   .U....vh\0.....~
    0020 - 06 d2 59 7e 80 fc 60 9c-39 71 5a 84 09 e3 ed 48   ..Y~..`.9qZ....H
    0030 - 44 45 cb 8e e5 26 e3 fe-45 e4 f5 82 49 26 cb 0b   DE...&..E...I&..
    0040 - cc 79 20 46 99 41 70 bb-b3 b9 d1 a3 60 c3 84 67   .y F.Ap.....`..g
    0050 - 69 6b 1f 87 fe c4 56 56-47 d7 a2 99 5c 49 bd 10   ik....VVG...\I..
    0060 - dd 85 db 38 11 94 fb 76-77 15 1c 10 8e cf bb a3   ...8...vw.......
    0070 - 9a e2 a7 63 d3 a8 95 9e-72 dd 02 e4 b8 3a a6 01   ...c....r....:..
    0080 - c6 0d b0 3e aa 29 50 f5-4d 64 53 35 55 66 f5 de   ...>.)P.MdS5Uf..
    0090 - 09 24 52 9a cf 9a a1 94-0a ea 21 3f aa 6c af b3   .$R.......!?.l..
    00a0 - ab dc 9c 66 72 9f f9 72-ef 83 16 86 59 22 6e c1   ...fr..r....Y"n.

    Compression: 1 (zlib compression)
    Start Time: 1413901802
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
250 DSN
read:errno=0

Nope, same output as before.  Double-check the networking, since there had been issues with that earlier:

webmail:/etc/postfix# netstat -tuap | grep smtp
tcp        0      0 *:smtp                  *:*                     LISTEN      4400/master     
tcp6       0      0 [::]:smtp               [::]:*                  LISTEN      4400/master     
webmail:/etc/postfix# netstat -tuapn | grep 25
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      4400/master     
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      4400/master     
tcp        0      0 172.16.150.47:22        172.16.150.112:55579    ESTABLISHED 25596/1         
tcp6       0      0 :::25                   :::*                    LISTEN      4400/master     
webmail:/etc/postfix# netstat -tuapn | grep 587
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      4400/master     
tcp6       0      0 :::587                  :::*                    LISTEN      4400/master     
webmail:/etc/postfix# dig @8.8.8.8 webmail.all-spec.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> @8.8.8.8 webmail.all-spec.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- 64663="" font="" id:="" noerror="" opcode:="" query="" status:="">
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;webmail.all-spec.com. IN A

;; ANSWER SECTION:
webmail.all-spec.com. 7199 IN A 97.66.29.211

;; Query time: 365 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Oct 21 11:29:09 2014
;; MSG SIZE  rcvd: 54

Perfect.  Ok, so what is in /etc/ssl?  There should be multiple certs there somewhere to verify the server key, right?
webmail:/etc/postfix# ls /etc/ssl
2009-2011     cabundle.crt.old private       webmail.all-spec.com.csr
2011-2013     certs webmail.all-spec.com.cert      webmail.all-spec.com.key
cabundle.crt  openssl.cnf webmail.all-spec.com.cert.old
webmail:/etc/postfix# ls /etc/ssl/certs/
ACEDICOM_Root.pem
AC_Raíz_Certicámara_S.A..pem
Actalis_Authentication_Root_CA.pem
AddTrust_External_Root.pem
AddTrust_Low-Value_Services_Root.pem
AddTrust_Public_Services_Root.pem
AddTrust_Qualified_Certificates_Root.pem
AffirmTrust_Commercial.pem
AffirmTrust_Networking.pem
AffirmTrust_Premium_ECC.pem
AffirmTrust_Premium.pem
America_Online_Root_Certification_Authority_1.pem
America_Online_Root_Certification_Authority_2.pem
ApplicationCA_-_Japanese_Government.pem
A-Trust-nQual-03.pem
Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
Baltimore_CyberTrust_Root.pem
ca-certificates.crt
cacert.org.pem
CA_Disig.pem
Camerfirma_Chambers_of_Commerce_Root.pem
Camerfirma_Global_Chambersign_Root.pem
ca.pem
Certigna.pem
Certinomis_-_Autorité_Racine.pem
Certplus_Class_2_Primary_CA.pem
certSIGN_ROOT_CA.pem
Certum_Root_CA.pem
Certum_Trusted_Network_CA.pem
Chambers_of_Commerce_Root_-_2008.pem
CNNIC_ROOT.pem
Comodo_AAA_Services_root.pem
COMODO_Certification_Authority.pem
COMODO_ECC_Certification_Authority.pem
Comodo_Secure_Services_root.pem
Comodo_Trusted_Services_root.pem
ComSign_CA.pem
ComSign_Secured_CA.pem
Cybertrust_Global_Root.pem
Deutsche_Telekom_Root_CA_2.pem
DigiCert_Assured_ID_Root_CA.pem
DigiCert_Global_Root_CA.pem
DigiCert_High_Assurance_EV_Root_CA.pem
Digital_Signature_Trust_Co._Global_CA_1.pem
Digital_Signature_Trust_Co._Global_CA_3.pem
dovecot.pem
DST_ACES_CA_X6.pem
DST_Root_CA_X3.pem
EBG_Elektronik_Sertifika_Hizmet_Sağlayıcısı.pem
EE_Certification_Centre_Root_CA.pem
E-Guven_Kok_Elektronik_Sertifika_Hizmet_Saglayicisi.pem
Entrust.net_Premium_2048_Secure_Server_CA.pem
Entrust.net_Secure_Server_CA.pem
Entrust_Root_Certification_Authority.pem
ePKI_Root_Certification_Authority.pem
Equifax_Secure_CA.pem
Equifax_Secure_eBusiness_CA_1.pem
Equifax_Secure_eBusiness_CA_2.pem
Equifax_Secure_Global_eBusiness_CA.pem
Firmaprofesional_Root_CA.pem
GeoTrust_Global_CA_2.pem
GeoTrust_Global_CA.pem
GeoTrust_Primary_Certification_Authority_-_G2.pem
GeoTrust_Primary_Certification_Authority_-_G3.pem
GeoTrust_Primary_Certification_Authority.pem
GeoTrust_Universal_CA_2.pem
GeoTrust_Universal_CA.pem
Global_Chambersign_Root_-_2008.pem
GlobalSign_Root_CA.pem
GlobalSign_Root_CA_-_R2.pem
GlobalSign_Root_CA_-_R3.pem
Go_Daddy_Class_2_CA.pem
Go_Daddy_Root_Certificate_Authority_-_G2.pem
GTE_CyberTrust_Global_Root.pem
Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
Hongkong_Post_Root_CA_1.pem
IGC_A.pem
Izenpe.com.pem
Juur-SK.pem
Microsec_e-Szigno_Root_CA_2009.pem
Microsec_e-Szigno_Root_CA.pem
NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
NetLock_Business_=Class_B=_Root.pem
NetLock_Express_=Class_C=_Root.pem
NetLock_Notary_=Class_A=_Root.pem
NetLock_Qualified_=Class_QA=_Root.pem
Network_Solutions_Certificate_Authority.pem
OISTE_WISeKey_Global_Root_GA_CA.pem
QuoVadis_Root_CA_2.pem
QuoVadis_Root_CA_3.pem
QuoVadis_Root_CA.pem
Root_CA_Generalitat_Valenciana.pem
RSA_Root_Certificate_1.pem
RSA_Security_2048_v3.pem
Secure_Global_CA.pem
SecureSign_RootCA11.pem
SecureTrust_CA.pem
Security_Communication_EV_RootCA1.pem
Security_Communication_RootCA2.pem
Security_Communication_Root_CA.pem
Sonera_Class_1_Root_CA.pem
Sonera_Class_2_Root_CA.pem
spi-ca-2003.pem
spi-cacert-2008.pem
ssl-cert-snakeoil.pem
Staat_der_Nederlanden_Root_CA_-_G2.pem
Staat_der_Nederlanden_Root_CA.pem
Starfield_Class_2_CA.pem
Starfield_Root_Certificate_Authority_-_G2.pem
Starfield_Services_Root_Certificate_Authority_-_G2.pem
StartCom_Certification_Authority_G2.pem
StartCom_Certification_Authority.pem
S-TRUST_Authentication_and_Encryption_Root_CA_2005_PN.pem
Swisscom_Root_CA_1.pem
SwissSign_Gold_CA_-_G2.pem
SwissSign_Platinum_CA_-_G2.pem
SwissSign_Silver_CA_-_G2.pem
Taiwan_GRCA.pem
TC_TrustCenter_Class_2_CA_II.pem
TC_TrustCenter_Class_3_CA_II.pem
TC_TrustCenter_Universal_CA_III.pem
TC_TrustCenter_Universal_CA_I.pem
TDC_Internet_Root_CA.pem
TDC_OCES_Root_CA.pem
Thawte_Premium_Server_CA.pem
thawte_Primary_Root_CA_-_G2.pem
thawte_Primary_Root_CA_-_G3.pem
thawte_Primary_Root_CA.pem
Thawte_Server_CA.pem
Trustis_FPS_Root_CA.pem
T-TeleSec_GlobalRoot_Class_3.pem
TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem
TURKTRUST_Certificate_Services_Provider_Root_1.pem
TURKTRUST_Certificate_Services_Provider_Root_2.pem
TWCA_Root_Certification_Authority.pem
UTN_DATACorp_SGC_Root_CA.pem
UTN_USERFirst_Email_Root_CA.pem
UTN_USERFirst_Hardware_Root_CA.pem
ValiCert_Class_1_VA.pem
ValiCert_Class_2_VA.pem
Verisign_Class_1_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem
Verisign_Class_1_Public_Primary_Certification_Authority.pem
Verisign_Class_2_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem
Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem
Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
Verisign_Class_3_Public_Primary_Certification_Authority.pem
Verisign_Class_4_Public_Primary_Certification_Authority_-_G3.pem
VeriSign_Universal_Root_Certification_Authority.pem
Visa_eCommerce_Root.pem
Wells_Fargo_Root_CA.pem
WellsSecure_Public_Root_Certificate_Authority.pem
XRamp_Global_CA_Root.pem

Good lord.  That's with the hashed certs snipped out, BTW.  But yes, looking through the list, there's Thawte certs there that should be the root cert(s) to verify the key.  So why is the key not verifying, and SASL not working?  I verified the Thawte root certs, and they all verify.  The server key still doesn't.

So I focused on the 'why is SASL not working' side of it, since the key not working shouldn't affect that.  Going by the last answer here: http://serverfault.com/questions/541662/postfix-returning-no-sasl-authentication-mechanisms-despite-mysql-auth-config I checked over my dovecot config for the socket setting:

webmail:~# dovecot -n
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-686-bigmem i686 Debian 7.6 
disable_plaintext_auth = no
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_location = maildir:~/mail
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
passdb {
  driver = pam
}
plugin {
  quota = maildir
  quota_rule = *:storage=50G
  quota_rule2 = .AllSpec:ignore
  quota_rule3 = .WebOrders:ignore
  quota_rule4 = .Trash:ignore
  quota_rule5 = .Sales:ignore
  quota_rule6 = .Orders:ignore
}
protocols = imap
ssl_ca =
ssl_cert =
ssl_key =
userdb {
  driver = passwd
}
protocol imap {
  mail_plugins = quota imap_quota
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
  mail_plugin_dir = /usr/lib/dovecot/modules/
  mail_plugins = sieve
  postmaster_address = postmaster@all-spec.com
  sendmail_path = /usr/lib/sendmail
}

No socket config.  Added it to 10-master.conf.  After several attempts to restart Dovecot to force the socket to appear, I finally added the socket settings to dovecot.conf... and discovered that it completely broke Dovecot and that I hadn't properly ported dovecot.conf from the 1.x syntax to 2.x way back when I'd upgraded the server.  So, after putting in a vanilla config file and reconfiguring it (and note, there's include lines in the new dovecot.conf to pull in files in conf.d), the socket was being generated.  Still no working SASL, however.  So I made sure everything from here (http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL): 

submission inet n - - - - smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_sasl_security_options=noanonymous
  -o smtpd_sasl_local_domain=$myhostname
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_login_maps=hash:/etc/postfix/virtual
  -o smtpd_sender_restrictions=reject_sender_login_mismatch
  -o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject

was included in master.cf.  Postfix restarted faster than it had been, and the 'no SASL mechanism' errors were gone by the time I got done reconfiguring Dovecot and Postfix.

So, now that the servers were ok, all that was left was to get the key to verify.  I took a while researching this, because I desperately did not want it to be the keys themselves, in case we had to buy new ones.  I tried moving the certificates down into /etc/ssl/certs to see if it was problem with finding the certs to verify against with the server certificate in the parent directory... nope.  After reading http://www.howtoforge.com/forums/showthread.php?t=66000http://www.thawte.nl/en/support/manuals/dovecot/dovecot+imap+server/install+certificate/http://openssl.6102.n7.nabble.com/Intermediate-root-CA-s-lost-and-confused-td8733.html, and http://www.fordfrog.com/2012/01/28/ssl-in-postfix-with-intermediate-certificate/, it seemed that the intermediate key was the problem.  Or a lack thereof, rather.  I'd seen some references in my earlier research to concatenating the certs together so there would be a definite cert chain there.  I looked at the cert I was working with, and lo and behold, there's only 1 cert listed in the file.  I asked my boss if there were any intermediate certs that were supposed to be in with the server cert (he handles buying certs), and he passed along the link Thawte.  No purchase necessary, since the cert was still valid.  I downloaded the bundle and looked at the certs... and the intermediate cert was not in /etc/ssl/certs or /etc/ssl.  So, I concatenated the intermediate cert and server cert together, and it worked:

webmail:/etc/ssl/certs# openssl s_client -starttls smtp -connect webmail.all-spec.com:587
CONNECTED(00000003)
depth=3 C = ZA, ST = Western Cape, L = Cape Town, O = Thawte Consulting cc, OU = Certification Services Division, CN = Thawte Premium Server CA, emailAddress = premium-server@thawte.com
verify return:1
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA
verify return:1
depth=1 C = US, O = "Thawte, Inc.", CN = Thawte SSL CA
verify return:1
depth=0 C = US, ST = North Carolina, L = Wilmington, O = All-Spec Industries Inc, CN = webmail.all-spec.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
   i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
 1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=North Carolina/L=Wilmington/O=All-Spec Industries Inc/CN=webmail.all-spec.com
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 3252 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0EE2B05739518E66B0E82490D51DBE23DDC481456FF447E7382288586ACBC0AF
    Session-ID-ctx: 
    Master-Key: F475FB98D8D4B80FAE196785BCAF77537327B4F8936A2E8CD1EF13343A930B9EA3B8B1D5AB63F761BC06DCEDD1CCE218
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - d1 d6 0d c2 2c a3 49 da-c0 5b f1 21 ef b4 21 dd   ....,.I..[.!..!.
    0010 - f0 0f fa 34 08 30 db 0e-a2 2b 84 3e 84 ee ca 70   ...4.0...+.>...p
    0020 - 05 f1 dc 15 9e bb f7 ec-f4 54 6d 56 6b 3b 00 7e   .........TmVk;.~
    0030 - 47 9f bd fb df 24 92 2e-9d 9c e7 f8 db 52 2d ab   G....$.......R-.
    0040 - c2 38 7f 77 86 b6 42 d7-77 9f 18 d0 cc be 6a 00   .8.w..B.w.....j.
    0050 - d2 b2 47 5c 34 aa 6b 37-1b 70 d2 f9 d2 38 40 fe   ..G\4.k7.p...8@.
    0060 - b9 38 c1 63 18 fe 1e 39-68 b1 98 cc 53 bd 1c 11   .8.c...9h...S...
    0070 - 09 2f 3c 1a 20 c9 bd d5-a7 bd e9 06 d5 8f f8 2a   ./<. ..........*
    0080 - 71 f2 8b b0 9b 06 59 ae-63 20 56 42 ad 7d 6b 94   q.....Y.c VB.}k.
    0090 - 39 7c 20 c9 26 1b 33 6f-da 95 05 6e 29 80 33 56   9| .&.3o...n).3V
    00a0 - 99 1f f8 e1 5b 82 04 26-db 7e bd c2 c0 75 c7 1e   ....[..&.~...u..

    Compression: 1 (zlib compression)
    Start Time: 1414009179
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
250 DSN
^C

A final note: if you run a webmail interface, you do have to restart Apache to get the webmail back working entirely correctly.  Specifically, Roundcube freaks out with all the above going on in the background, and decides it doesn't want to display new mail anymore.


10.28.2012

The Beauty and Wonder of Programming Specs

Bit of a work rant. I'm not trying to use this as a whine-fest, although it'll sound like one at points. Take this more of a case study or illustrative, anecdotal example of why it's proper practice to write out a formal specification for your developers.

There's a project I've been working on for 9 months now; I'm not sure if I've mentioned it, but if I have it's the pricing update program. Anyway, this was originally slated to be a 6mo project. Well, obviously, it hasn't been. Part of the problem has been, I'm learning Flashbasic (Pick/Basic, whatever) as I'm writing it. So that's obviously going to make things take longer than if someone experienced in the language is working on it.

 However, I'm getting ready to put it into testing for the 3rd time here in the next week or so. Why? I had no specification to work off of. All I've had to go by is a couple meetings with the heads of the involved departments, and streams of emails back and forth. Plus the input during the previous 2 demos/tests. Neither meeting was that enlightening-- the first one got bogged down in minutiae and was more confusing to me than explanatory.  I took down some notes, and ended up with something that got completely rejected when I called the second meeting.  However, that was all I got out of the second meeting: what you're doing won't work, it's too unlike what we're doing now and generally wrong anyway, but the current way of doing this is still terrible.  I cobbled together something based on what I could glean from the second meeting and from what I was learning on my own from the way pricing currently works.  Then I demoed it.  It had a lot of flaws, was missing a lot of features, there were a lot of miscommunicated aspects.  I took copious notes, learned that the department head I hadn't consulted with so much a) had taken on the duty of updating pricing and therefore needed the project more than the other department did, and b) he was more analytical and demanding, and therefore easier to hammer out the program with.  Cue the storm of emails back and forth on what was needed where.  Got a mostly workable version, demoed it again, except for a major missing feature it was great.  Now I'm finishing that feature and reworking the program flow to shoehorn this whole extra step in, and hoping it'll fly (it's necessary still to edit the spreadsheet, just not to the extent it was the last time I showed them the program-- they want to do 0 editing.  This is not possible).

Note all the miscommunication, poorly stated needs and goals, and tendency to leave how to write the program (and therefore alter the business processes surrounding it) to an inexperienced programmer who has never had anything to do with that aspect of the business, and never will on a daily basis.  Note how much time was wasted simply trying to get information and input from the people who will be using it every day for the rest of their career with the company (theoretically).  This could have been avoided with one thing-- a program specification.

Even though my classes in college were theoretically all intro classes (although you could get more out of them if you put more in, as I did), we were taught to work with specs.  In Linux Administration class,  Netware Administration, C Programming, Java Programming, Security, Forensics, all the electronics classes (DC/AC Circuit Analysis, Semiconductor Applications, Digital Electronics, Intro to Microprocessors), even the first semester classes like Computer Repair and Survey of Operating Systems-- they all dealt with both writing specs, and working from one.  With a proper spec, there is no question of what needs to be done.  Everything is laid out, from the current process, to what is broken about the current process, to what needs to remain the same with the new process, to the new process.  Even if the new process is a proposal, it still gives the programmer, who does not and will not work in the department that needs the program they're tasked to write, a starting point to work from.  And in the meeting about the proposed spec, there can be discussion over what is possible, what the needs are, etc.  Rather than everyone just throwing up their hands and asking "well, what do you think it should do?".  I was, in fact, asked by the less-analytical dept. head that exact question.  That was the point I threw up my hands and called the second meeting, so her boss and my boss could provide us some guidance.  Had there been a spec prior to the first meeting, much of that back-and-forth wouldn't have happened, and much of my time (and everyone else's) wouldn't have been wasted.

Unfortunately, and this is where it gets a little whiny again, I've brought up the suggestion (during the first demo) of writing a spec from now on.  It was met with total resistance.  So right now I'm basically... how shall I put this politely?.... whistling in the wind, as far as applying this to myself.

But maybe this can be an example to someone else, and save them the heartache I'm relegated to.  However, this experience has taught me a lot about my coworkers, and working with them on designing programs.  And hopefully taught them a thing or two about working with the programmer who hasn't basically built everything from the ground up, or rebuilt it for the company's needs, for the past 15 years.

I'm just really looking forward to the end of this project, and to move on to other, more interesting projects I have lined up.

7.04.2012

SolusOS

I have a new hobby. For the past couple months, I've been working on SolusOS. For those who don't know, in a nutshell it's a desktop distro based on Debian Stable, with some of our own packages thrown in, and Gnome3 Fallback mode hacked up to look like Gnome2. I'd worked with Justin and Ikey on Mint, and left shortly after they did. Around the time they left Mint, they came up with the idea for SolusOS, which is what LMDE was supposed to be but never was. I love it (obviously)-- not only do I squash bugs and maintain a couple packages for it, but it's running on pretty much all my systems.

I'm basically head of bugsquashing. It's not the most glamorous job, but I'm good at it, and want to see it done right. Right now I maintain the non-PAE i686 kernel package, and I'm getting ready to package up inxi, smxi, and sgfxi. Those scripts are already included in the distro by default, but they need to be packaged for upgrade and ISO-rolling purposes.

I'm still figuring out balancing working and hacking, but I discovered last weekend I can bring work home and not let it kill my weekend. And I've also figured out to hack while my fiance's playing video games, since that's great 'me' time. So slowly but surely, I'm taking on more stuff with SolusOS.

So yeah, if you're reading this and you haven't tried SolusOS, go try it. It's awesome, I promise.

1.29.2012

Work Hacks

Man, it's been a while since I've blogged. I've been busy working, and finding a new balance between working and.. well.. not working. With everything that's been going on in my personal life, I wouldn't say 'relaxing'. And I haven't really been hacking on my days off.

I have been hacking at work a lot. And learning. A lot. Didn't realize how incredibly green I was when I took the job-- thank the gods I mellowed down my arrogance years ago. :)

Anyhoo, besides brushing up on my linux admin skills and getting back my programming chops, I've had to learn a system that used to be a mainframe OS back in the day, and is now a giant, sprawling database with its own scripting language that still sometimes thinks it's an OS. That system is D3(formerly known as Pick). Interesting stuff. I've gotten fairly decent at programming in pick/basic now (which is the lovechild of Dartmouth Basic and Fortran), and I'm really starting to get the hang of admining it.

My first project I did myself for it was to try to bring modern syslog-style system and debug logs to it. I've been fairly successful, although my needs are now outstripping the capabilities of my system of scripts. But, as my boss pointed out, the original/current production code is a proof-of-concept. It's written in a mix of pick/basic and python. The pick/basic subroutine takes in the error/info message as an argument (the string is freeform to allow as little or as much content as you want included), appends user, port number, timestamp, and a bunch of other data, then writes it out to a file in a queue in the underlying linux system. From there, a python TFTP client picks it up, hands it off to a python TFTP server on another system, which puts it in a queue for a script that writes the entry out to syslog and various log files. It needs to pass the logs off to another system because space is at a premium on the D3 server, but not so much on other servers. Plus then I can move it over to a Debian server, which is much easier to hack on than RHEL. RHEL 4 doesn't even have git, and still has python2.4.

Sounds good so far. But right now, it does everything from log debug messages from various subsystems in our D3 frontend, to collect web hits. Which means it's running up to 10 times a second at peak times. The scripts themselves are fine with this (actually, I have to keep a sleep() in the TFTP client so it doesn't start cycling too fast and maxing the CPUs when there's nothing for it to pick up), that's just a massive amount of logs. Plus the web stuff shouldn't be treated the same as debug messages. So I'm forking the logger to handle web hits separately. I'm also going to re-write the regular logger to use inotify rather than looping to read the contents of the queue to see if there's anything new to upload-- this should both cut down on overhead and keep it from freaking out when the queue is empty. The web logger will do the same thing, but instead of logging to syslog, it'll store info in an SQL database since that's more efficient for storing long-term data. Not to mention easier to analyze.

There's some other projects I'm working on, but I think this blog is long enough for now. I should be back with more follow-up on the web logger, and will probably have interesting war stories about talking to FedEx's web services with python to do on-demand tracking, and doing PDF parsing for uploading price data.

7.27.2011

New Job

Sudden random real life post... I got a job a couple weeks ago as an assistant sysadmin. :) No more retail hell. I still have 1 idi^H^H^Hluser to deal with, but hey. Better than being surrounded by hundreds. Not sure how much hacking I'm going to do, apparently it's even true for me-- once you work on computers for a living, you're not too inclined to mess with them for fun. :\ I'm stuck on whiskey, I have to learn pyqt to make any more headway. Trash80 is trucking along on rewriting coreutils in python, and I'm sitting back giving advice and feedback. And now my fiance is asking me to make a one-off distro for him.

On a job-related note, I'm compiling a list of "Sysadmin Rules to Live By". I have 3 so far.

1. Everyone is an idiot.
Seriously, everyone. I have my stupid moments too, I won't lie.
2. The user didn't do anything.
If I had a dollar for every time I've heard "I didn't do anything, it just broke!" I could buy a curry chicken lunch combo... after just 3.5 weeks on the job.
3. It's impossible, but it has to be done anyway.
Had an example of that this morning: clean out the IT detritus in the back of customer service... even though there's no room in the server room for more crap. And storing it in an empty cubicle is not an option. Oy.

But, owing to my massive masochistic streak, I do enjoy my work now. Maybe after I settle in I'll find my way back to hacking.

1.02.2011

Whiskey is on launchpad!

Whiskey is now on launchpad, in 2 places since I can't figure out how to link habanero (trash80's and my project name) to my personal account. https://code.launchpad.net/~compgrokker/+junk/whiskey and https://code.launchpad.net/~habanero/+junk/whiskey.

Update on the gui: since pyglet will never upgrade to python 3.0, I'm using pyjamas. PIL hasn't upgraded yet either, but it's tempting to grab the bits I need (a small subset of the whole module), fork it, and upgrade it. I haven't decided on that yet, but for right now whiskey still depends on python2.6 (arch's stable python, and the highest version I know of that PIL will work with), although I'm writing my code to be upwards compatible with 3.0.

Although whiskey is up on launchpad, it doesn't have a PPA yet simply because it's not particularly useful yet. Hence why it's still in experimental. Arch has the debian buildtools though, so once whiskey is close to what I'm aiming for (or at least close enough that I feel comfortable opening it up to a wider audience), look for a PPA.

12.29.2010

whiskey db backend

After playing around with it and reading the specs, yaml is not the right tool for the job. It's too hard to have multiple attributes for the same key (ie, colours: grey, blue, green) and even harder to parse that back out and search it. Some sort of nosql db seems to be what I'm looking for, and mongodb seems to be the friendlier of the python-supported nosql dbs out there. So... scratch the yaml stuff (until it comes time for config files, that's still a good idea), mongodb is going to be the db backend. And I should have some working, prealpha code going up on launchpad soon. No promises though, my work hours have expanded (yay money, boo exhaustion) so it's tough to get anything done but work, chugging coffee, and sleeping. But I'm trying. :)