I got an email from a customer the other day about mitigating POODLE with IBM's Lotus Protector for Mail Security (LPMS). There is a technote for this, 1687838. At the top there is an interesting warning:

IMPORTANT: disabling SSLv3 for XMail may cause severe incompatibility problems with other MTAs that do not support TLS 1.x

I was asked if this was an issue. My response:

It depends on who you are STARTTLS emailing to....

This only affects domains that you have set as requiring TLS between your server and theirs. So I'd check with them before you do it

Now, in reality I'd most likely leave SSLv3 enabled in my SMTP environment (I'm not talking about clients connecting here like Outlook, Thunderbird, just edge SMTP relay servers). My rationale (to date) here is laid out below:

Scenario 1 : Both SMTP servers can do TLS1.0+
1.        I send an email to blah.com
2.        My server and the blah.com are both enabled for STARTTLS
3.        The negotiate TLS1.0, the delivery transaction is encrypted between the servers and email flows

Scenario 2 : One server can (but is not mandated to) do STARTTLS, the other can't
1.        I send an email to blah.com
2.        My server sees that blah.com cannot do STARTTLS
3.        The email is sent in plain text and the email flows

Scenario 3 : Both servers can do STARTTLS, but one will only do TLS1.0+ and the other will do SSLv3.
1.        I send an email to blah.com
2.        My server and the blah.com are both enabled for STARTTLS
3.        The servers cannot negotiate a protocol, so no encryption takes place
4.        The email is sent in plain text and the email flows

Scenario 3 is the interesting one here. You tried  to send an email down an encrypted tunnel but you can't. Even though both servers could. Now POODLE is bad an all, but really, if two SMTP servers are trying to send email to one another via an encrypted tunnel and they fail back to plain text to avoid POODLE what good does that do? POODLE is a  man-in-the-middle attack vector, and you could argue that you have no idea where your SMTP transaction is going, but that's really all I can come up with for causing "fail back to plain text". It's not like you have your SMTP edge gateway servers sitting in a Starbucks hanging off their public wifi (if you are, then you really should disable SSLv3).

As President Obama used to say about same sex marriage, my thoughts on this are still evolving, but it sure feels like disabling SSLv3 on an SMTP server may lead to some unexpected results.

If your SMTP edge server is Domino, then you may have issues leaving STARTTLS enabled (as outlined by Frank Paolino) until IBM release the multi-protocol fix (which I think maybe in the 9.0+ fix, it's starting to get confusing about what fix is going where). But this goes back to my main point....in Frank's case ProofPoint have disabled SSLv3, so now Frank has to send plain text email to ProofPoint.

So, I'll throw this out to the world in general....what are you doing for STARTTLS?
Darren Duke   |   October 23 2014 11:47:17 AM   |    domino    |   Comments [1]

Behold, the silence has ended.....the crescendo that is IBM has finally lifted the veil on some fixes for some very large security holes....AFAIK these are native Domino fixes for all platforms. I'm unsure of the protocols supported, but my understanding is all of them, but only time will tell.

These are not available yet, but should be in "weeks"...

First up, fix POODLE outlined in Technote 1687167. This is coming to:
  • 9.0.1 FP2
  • 9.0
  • 8.5.3 FP6
  • 8.5.2 FP4
  • 8.5.1 FP5

I think that is every supported Domino platform.  

Second is SHA2 support and TLS 1.2 support, as outlined in Technote 1418982. This is coming to
  • 9.x

This is great news, however if you need TLS 1.2 or SHA2 on 8.5.x you are out of luck....but you can still use my reverse proxy for that scenario.
Darren Duke   |   October 21 2014 10:53:44 AM   |    domino    |   Comments [2]

In an effort to help Domino customers mitigate the disaster that is the SSLv3 Poodle bug, I am providing the virtual machine linked at the bottom of this post. Note, you can also use the IBM HTTP Server bundled with R9 if you are on a Windows server....if that is the case, stop reading.

YOU USE THIS POST AT YOUR OWN RISK. For professional services related to this contact STS Sales.

Take backup copies of any files you change, including the Domino Directory. That way if you screw up......

Read this in it's entirety before you start.....it is not for the faint of heart. I take no responsibility for you screwing up your environment. None.

This VM is an Ubuntu 14 LTS server (patched as of Oct 15th 2014) with Apache installed in a way to allow easy integration as a reverse proxy for a Domino server. This allows the user to disable SSLv3 and utilize TLS 1.0, 1.1 and 1.2  thus mitigating Poodle. The apache server will use the best cipher for the client connecting to it, so it will prefer TLS 1.2 if the browser can support it.

No warranty is implied or provided. You use this VM at your own risk. There is no guarantee this will fix any and all security problems. It is suggested that after install you check your installation here https://www.ssllabs.com/ssltest/index.html (although at the time of writing the test site didn't indicate SSLv3 as an issue....IT IS).

OK, so what do you have to do to get this thing working.....

1) You need to be able to install OVF virtual machine templates. If you don't have a virtual infrastructure then this may not help
2) You have Domino working as a web server, or iNotes, or Quickr, or Traveler
3) You want to fix the Poodle bug and you can't or won't wait for IBM to address this properly
4) You don't need Windows XP with Internet Explorer support (this VM uses SNI, XP with IE can't do SNI although I believe Firefox and Chrome on XP can....). If you need XP support I may create another VM. You never know.
5) You don't mind changing the HTTP settings of your domino servers, including adding new DNS records to your internal DNS servers.
6) You want to address Poodle, SHA2 and/or add TLS to Domino.

If all of these are a check marked, continue reading....

The VM contains one Apache site capable of handling three different scenarios, iNotes, Quickr and Traveler.

1) Go download the VM here (there is no warranty, implied or given by use of this VM)
2) Install the VM on your virtual hardware
3) Power up and log in (default is root/root)
4) Change the default password using the passwd command
5) Change the IP address assigned to the machine with vi /etc/network/interfaces command (change all of the settings here to match your network). If you don't know vi then google it.
6) Reboot
7) Get an Apache compatible SSL certificate from your provider. If you need to create a new CSR do not use Domino to do this, but rather use OpenSSL (installed on this VM if you don't have an installation). Your SSL vendors site will have instructions on how to do this, here are GoDaddy's instuctions. When you have the key file and the signed certificate for your site, sites or wildcard copy them to the /etc/apache2/ssl folder (your provider will also give you a "bundle" certificate, copy that over too).
8) Use WinSCP to log into the VM and navigate to /etc/apache2/sites-enabled and double click on the combined.conf

Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle
9) The first two virtual hosts (signified by the tag) are iNotes, the second two are Quickr, the third pair is Traveler. If you don't need a particular host (you don't use Quickr for example), simply delete everything between the two corresponding and tags (including the tags themselves). TAKE A BACKUP FIRST....you might do this wrong.

iNotes: Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle

Quickr: Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle

Traveler : Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle

10) Edit the file changing at least anything with an IP in it, anything with a domain name in it, anything with a server name in it and anything with an SSL certificate in it. Here is what needs to be changed for iNotes:

a) Take a backup of the Domino Directory before you change anything.....I'm not going to outline the Domino part, I figured if you're reading this you know that part.
b) Our Domino server was called webmail.yourdomain.com. We are now moving this name to Apache and have changed the Domino HTTP server to domino1.yourdomain.com.  (if you don't know how to do this, stop and hire me via the link above)
c) Our Domino server was also using HTTPS, but now we've turned this off for Domino and only HTTP is in use on Domino.
d) There is also a new internal DNS entry pointing domino1.yourdomain.com to the Domino server IP (this is not an external DNS entry, only internal).
e) Externally, webmail.yourdomain.com points to Apache (in this case
f) Make sure you can ping the new domino1.yourdomain.com address from both the Apache server and the Domino server.

Remember, there are two Apache virtual hosts per Domino server....one that maps to HTTP that in turn redirects to the second one that handles HTTPS....

Below are the iNotes HTTP virtual host changes:

a) The vitual host address needs to the be the IP address of this VM
b) The host name's should match whatever URL your users use to get to iNotes, in this case webmail.yourdomain.com

Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle

Below are the iNotes HTTPS changes:

a) The vitual host address needs to the be the IP address of this VM
b) The host name's should match whatever URL your users use to get to iNotes, in this case webmail.yourdomain.com
c) The SSL certificates need to match the ones you copied to the SSL folder, also update SSLCertificateChainFile to your providers bundle
d) The iwaredir.nsf needs to be changed to match your web mail redirector NSF file name
e) The ProxyPass and ProsyReversePadd host names need to be changed to your new iNotes server internal name (note this is also now a HTTP link, not HTTPS)

Image:Here is a freely available VM to reverse proxy Domino - shoot the poodle

11) Save the file
12) Restart Apache with the command /etc/init.d/apache2 restart
13) If you get errors, double check everything......and make sure to delete the vitrual hosts you don't need....like Quickr and Traveler for instance. After any changes restart apache
14) If it still doesn't work check the error log at /var/log/apache/ and look at the iNotes files.
15) If it still doesn't work then revert back to your original setup (I did tell you to take backups) and hire me.
16) At some point in the future, prevent Domino HTTP from being accessed anywhere but from the VM IP address.

This proxy has several advantages to IBM's approach of bolting IHS in front of Domino:
1.        You can have one and only one SSL certificate. I have a single wild-card certificate installed on the proxy and all proxied connections use this single certificate. That makes changing to SHA2/256 really, really simple.
2.        You don't have to patch server after server after server. One proxy, one set of patches.....heartbleed and shellshock anyone?
3.         I have significantly reduced my surface area on the web. Now all web servers traffic, be it Domino, Traveler, IIS or any other server are no longer directly connected to the evil internet.

In case you missed the link above, download the VM here (there is no warranty, implied or given by use of this VM).

AGAIN, you do this at your own risk. Unless your paying me to do this for you. you are on your own.
Darren Duke   |   October 15 2014 08:00:27 AM   |    domino  apache    |   Comments [4]

As some of you know, SHA2 support in the native Domino HTTP stack has been a bit of a fire starter of late. As IBM like to say "we've not heard that from our customers", here's your chance to change that.

How do you do that? Simple, if you are able to create a PMR against Domino (if you're on support for Notes and Domino you can) and mention that you want SPR # ABAI7SASE6 (APAR LO48388) addressed. Here's link to the IBM support portal, so head on over there and create an PMR via an Electronic Service Request (ESR) http://www-947.ibm.com/support/entry/portal/support

The actual technote about IBM Domino *not* supporting SHA2 is here http://www-01.ibm.com/support/docview.wss?uid=swg21418982

And because APAR and SPR are confusing, here's what they mean http://www-10.lotus.com/ldd/fixlist.nsf/0/e9c65ccbfc70cf4685256f8f00784bad?OpenDocument

What are you waiting for? Go be heard.

Darren Duke   |   August 20 2014 09:08:48 AM   |    domino    |   Comments [5]

August 19 2014 Tuesday

My customers don’t want Mail.Next

I have customers ranging from names you have heard of, to a few hundred seats, to  5 or less. I’m pretty sure that most customers I come into contact with are not on IBM’s radar. A few maybe, but most? Not so much.  Some of them occasionally ask about “mail.next” but none are excited. You see, these customers are not cutting edge. They are not chasing the next shiny ball of tinfoil. They cherish stability. Not constant change. Not constant “vaporware” demos of stuff that most think fluff. They  may not be cutting edge, but they expect anything they purchase (especially with the IBM name on it) to work. Just work. As advertised. Like they had always been promised when you buy from IBM.

Alas, this is not the IBM that exists any more. It is a shell of it’s former self. A veritable ghost town when you go looking for good technical folks. Product Managers and GM’s run awry creating and releasing software no one wants, no one uses and no one can install. IBM is a hell of a place to be an executive. No accountability that I can see. None. Nada. Zip. It appears to be the closest thing to an executive nirvana that has ever existed.

What my customers want is for IBM to fix things. Stop creating the fluffy, next big failure stuff. Just fix stuff you have already sold us on. The stuff my customers bought from you. The stuff I and IBM had promised works.  Except no, they won’t let things be fixed, to be made better. They just refuse to make an existing product better. That’s not what “executives” (now I need a shower after typing that word so many times) believe will keep them moving up. No, they have their 6-18 month plans.  And improving things is not a home run. Releasing something new is. Despite the odds that it will be an epic failure. But, no, up they go, off on their shareholder value beanstalks. Leaving a crater of crap for us clean up like SHA1, 40 and 56 bit SSL keys, inadequate web administration tools, languished development tools, the list goes on.

I’m pretty sure when an IBM executive (I need a second shower now) replies with “we’ve not heard that from *our* customers” they really mean “no one in my organization will tell me the truth because I’m a vindictive asshole, so don’t cross me or I will end your career”. Either that, or man, they live on another planet.

My customers don’t want mail.next. They want mail.now. But they want it to work in a sane and functional way. My customers are usually right. They didn’t want Symphony, or Workplace, or Mashups, or Alloy. They did want Notes, Domino, Quickr and Sametime.

 IBM, you should really start listening to my customers. They are correct far more often than IBM and it’s analysts are.

Darren Duke   |   August 19 2014 01:28:16 PM   |    domino    |   Comments [38]

With a little over 18 months since I've had to produce weekly tips, you've most likely missed my gems (OK, some weren't gems, but you get what you pay for)....

Anyway, this one is a gem, and I'm sure most of you know this but I surely did not.

On any Windows folder, hold down the Shift key and right click you get these additional options added to your context bar:

Image:How did I not know this feature of Windows existed? AKA - a useful tip

If you do anything with Websphere on Windows this will no doubt save you a ton of time.

Again, how did I not know about this?
Darren Duke   |   August 8 2014 09:55:58 AM   |    misc    |   Comments [0]

IBM are off creating the next "great" thing with Mail.Next. I can see the value of going "client-less" (i.e. web) but hopefully IBM are looking at IdeaJam and implementing many, many, many of the outstanding usability suggestions out there. So here you have it, my best, and last, suggestion to IdeaJam (and can you believe some people still post out there????).
Darren Duke   |   April 22 2014 10:50:26 AM   |    ideajam    |   Comments [5]

I'm also going to update the original post, but it appears there was an issue with the command in that original post:

-compactThreads 0 -updallThreads 2 -stoptime 5AM

You will see something like this in the notes.log on the server:

Image:Addendum to my Domino DBMT post (well, a correction)

use this one instead:

-compactThreads 0 -updallThreads 2 -stoptime 5:00AM

Yes, the ":00" makes all the difference.
Darren Duke   |   April 8 2014 07:39:04 AM   |    domino  dbmt    |   Comments [2]

So it has happened....yours truly is going to be a [non-IBM] Champion.....well for a day ;)

The Atlanta User Group (aka Atlanta Lotus User Group) is to host a pretty unique one-of-a-kind event on April 7th, 2014, that being the "Day of Champions". Each of the 11 "champions" gets a TED like 18 minutes or so to entertain, bedazzle and hopefully educate the attendees in all manner of things, technical or not-so-technical. Add to that a Connect-like Gurupalooza and speed-geeking and you are in for a real treat. Obviously this isn't your normal 2 hour quarterly meeting, this puppy is from 10AM until 4PM, hence the "DAY of Champions".

One of the things you'll notice about the 11 presenters is that most (10?) are from the Metro Atlanta area, an astounding abundance of resources for a single area of the country, especially as you will no doubt recognize every single presenters name. It's a veritable who's-who of the ICS community. Most, if not all, are current or former IBM Champions (hangs head in shame).

If that isn't enough two of the best IBM presenters walking this planet of ours are also presenting:
  • Scott Souder is giving the keynote and although he's from Texas we still love him. Mostly ;)
  • Louis Richardson. I could listen to this guy present on wall paper from the 1700's and still come away mesmerized.

You should come for these two alone, then add in the likes of Chris Whisonant, Tim Tripcony and Nathan Freeman to name but a few and you have a pretty stellar line up.

Anyway, there is no excuse not coming to see this event. Did I say it was pretty unique? Yeah, I did.....but again, it's unique. I think there will be attendees from far and wide so get on over to http://atl.ug/ and register. Right now. Now damn it!

For my part, I'll be presenting a cut down version of the "World according to Darren" presentation, that debuted at MWLUG last year. Yes, IBMers should be worried ;)

Champion for a day.....I can cross this off my bucket list now.
Darren Duke   |   March 24 2014 12:48:26 PM   |    ATLUG    |   Comments [0]

UPDATED 04/08/2014 - the command is NOT "5AM", but is "5:00AM"....it is now fixed below.....

So with the release of Domino 9, IBM added the Database Maintenance Tool (or DBMT). This new tool is a bit like a Swiss Army knife of server tasks. It can run compact, updall and full text index tasks all from one command.

But for some server environments (I'm thinking clusters using archived transaction logging here) all you really want is to have view indexed rebuilt so that is a user if failed over (or 100's of users fail over) then there is no delay while all these mail file view indexes are re-indexed as many simultaneous  users open their inbox on the new server. This is a pretty hefty kick in the balls for the cluster server.

Well, DBMT can fix that.

Bear in mind that I want to re-index views in the mail files of all users, WITHOUT also running a compact (with DBMT will do by default unless you tell it otherwise).

To get DBMT to run we use a Domino Program Document with the following command line:

-compactThreads 0 -updallThreads 2 -stoptime 5:00AM

like this:

Image:On Domino 9? Have a cluster? You’re using DBMT right?

The "-compactThreads 0" tells DBMT to not compact at the same time. I don't want a daily DBMT changing the DBIID, so that will stop this. Note the the program starts at 11:00PM and will run until 5:00AM (-stoptime). When it starts again the next day at 2:00AM, it begins where it left off.....how neat is that? Note also that I'm using a cluster name in the "Server to run on" field, this makes it easier to manage than a program document per server, although if you have need you can do that.

The last thing to do is to remove (or comment out) the "ServerTasksAt2=UpdAll" line from your server notes.ini.

Once your DBMT runs you see see something similar to this in the Domino Console and log:

Image:On Domino 9? Have a cluster? You’re using DBMT right?

Voila, now when users fail over from server A to server B, server B already has the default views built so that users don't have to wait as much. You can use this for huge performance gains on clusters that are active-passive servers or clusters where you have your user population split in an active-active scenario (even here, failed over users indexes may not be built, and this fixes that).

In the above example I'm using 2 updallThreads. You can increase as necessary depending on how good your I/O is on the Domino Data directory.

DBMT can do a few more tasks too (Swiss Army knife remember?) and you can find out more here:

Open Mic slides : http://www-01.ibm.com/support/docview.wss?uid=swg27039379

Domino 9 Admin Help page :http://www-12.lotus.com/ldd/doc/domino_notes/9.0/help9_admin.nsf/f4b82fbb75e942a6852566ac0037f284/5e0ecdb4cff85e9f85257b19005b5854?OpenDocument

Darren Duke   |   March 19 2014 08:03:27 AM   |    domino    |   Comments [4]