Nov 28
Easier certificates for MQ
icon1 Niklas | icon2 Tags: , . | icon4 11 28th, 2007| icon31 Comment »

Update: I’ve posted a follow up to this post with an important addition to the script to create your CA certificate.

The last couple of days I’ve been hard at work with setting up SSL for WebSphere MQ for a customer. A major part of that is creating certificates. WMQ uses the IBM security toolkit, GSKit. My previous experience with GSKit is not the best, I’ve found that it works inconsistently between versions and platforms and contains frequent bugs. Working with this time, the inconsistencies seems to still be there, but at least many of the bugs are now gone. One of the problems is that some GSKits doesn’t work with keys/certificates created with the Java keytool which has previously been my weapon of choice (with the Keystore Explorer GUI).

Finally giving up on keytool, I retreated to using the GSKit command line tool, gsk7cmd. Working with gsk7cmd is not the most pleasant experience (almost as bad as OpenSSL). Therefore, I’ve created a set of Linux shell scripts targeted for a streamlined flow for creating certs for WMQ. They are made available under the ASL 2 license. This is how they work.

Create a directory on some server where you like to create your keys. Copy the scripts to your directory. Edit the scripts to update some generic configuration, currently the DN used and the expiration time for the certificates.

Now start out by running create-ca.sh:

./create-ca.sh mySecretPassword

Make sure you choose a strong password. The command will create a CMS key store containing your CA root certificate. Save it somewhere safe.

Then, for each queue manager (or client), run:

./create-key.sh MyQueueManager mySecretPassword
or
./create-key.sh MyClientApplication mySecretPassword

This will create a directory containing a key created for your queue manager. Its certificate will be signed by the CA root key previously created.

If, for some reason you can not create the keys this way, try creating them on the target platform. For example, I failed to get any externally created keys working on iSeries, instead I had to use the DCM tool on the iSeries box. In this case, create the key and export a certificate request (I’ll write down the details for doing this on iSeries if anyone is interested). Save the certificate request in your keys directory created at the begining of this howto. Name it <queue manager name in lower case>.p10. Run:

./sign-key.sh MyQueueManager mySecretPassword

It will create a certificate response in the file called <queue manager name in lower case>.p7r. Import this together with ca_cert.crt on your target platform.

If your client happens to be a Java client, you need the key store in JKS format. To convert it, run the following command:

./convert-to-java.sh MyQueueManager mySecretPassword

Note that the signing steps does make a huge assumption. A CA must use a unique serial number for each certificate it signs. I didn’t want the overhead of keeping track of this. Instead I use a random number for the serial number. This means that if you sign lots of certs with your CA, you might very well get collisions. If you got that many certs, you probably need a better tool :-)

These scripts uses some ideas from the scripts published by Dale Lane. Thanks! If you want to port these scripts to some other platform, I will happily accept them into SVN. Same goes for bugs and improvements of course.

Technorati Tags: , , , ,

Nov 27
You learn something new every day
icon1 Niklas | icon2 Tags: . | icon4 11 27th, 2007| icon3No Comments »

Looks like IBM added automatic purging of expired messages on distributed platforms in WebSphere MQ 6.0. The documentation seems to fail mentioning this change so I had no clue. This is a great addition as for queues with frequent PUTs and rare GETs (e.g. with a slow or downed receiver), expired messages might build up fast.

The details of how and when the queue manager does the purging is not mentioned, only that you’re not guaranteed that it it will purge as fast as you want (which makes sense). Also ,the REFRESH QMGR TYPE(EXPIRY) command available on z/OS is not supported. It would explicitly purge messages on the named (including wild cards) queues. I hope they will get around to doing that as it will be a lot easier then doing GETs on all queues with possible expired messages when you really need to know that messages are purged..

Oct 19

Sometimes, when you’re out of luck, you need to use the old Base Java API for WebSphere MQ. That is, not JMS. And you might need to create, or even worse, read a RFH2 header. If this is you, you probably loath the mess that is the RFH2 header structure. It’s a mix of a leading binary structure, followed by zero or more almost-XML structures. I have yet to see a complete specification.

So, what’s a poor hacker to do. I’ve written up a utility class for creating and parsing RFH2 headers as part of the wmq-util library that I’m doing at work. The aim is simple: make it easy to use RFH2 headers without any risk of screwing up. Enjoy and report back bugs and suggestions.

Technorati Tags: , , ,

Oct 14
WMB init script
icon1 Niklas | icon2 Tags: , , . | icon4 10 14th, 2007| icon34 Comments »

For a client, I recently had to install a WebSphere Message Broker on Suse Linux. This also means getting it to start at boot time. Now, as far as I’ve been able to locate, WMB does not come with a init script. For WebSphere MQ, IBM recently added a service pac, MSL1, that does exactly this, but so far no luck for WMB. SLES uses LSB compliant init scripts, and has a command called insserv which automatically sets up the soft links to make all components start it the right order based on the dependencies you define as metadata in your script. So, with no further ado, here’s my current stab at the script:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          wmb
# Required-Start:    $syslog $network $remote_fs ibm.com-WebSphere_MQ
# Should-Start:      db2
# Required-Stop:     $syslog $network
# Should-Stop:
# Default-Start:     3 5
# Default-Stop:      0 1 2 6
# Short-Description: WebSphere Message Broker
# Description:       Starts WebSphere Message Broker
### END INIT INFO

# Source /etc/rc.status:
. /etc/rc.status

# Reset status of this service
rc_reset

case "$1" in
    start)
    	echo -n "Starting WMB "
    	su - mqm -c 'mqsistart BRK1D;mqsistart CFGMGRD'

    	# Remember status and be verbose
    	rc_status -v
	  ;;
    stop)
    	echo -n "Shutting down WMB "
    	su - mqm -c 'mqsistop BRK1D;mqsistop CFGMGRD'

    	# Remember status and be verbose
    	rc_status -v
	  ;;
    try-restart|condrestart)
	    ## Do a restart only if the service was active before.
	    $0 status
    	if test $? = 0; then
    		$0 restart
    	else
    		rc_reset	# Not running is not a failure.
    	fi
    	# Remember status and be quiet
    	rc_status
	  ;;
    restart)
    	## Stop the service and regardless of whether it was
    	## running or not, start it again.
    	$0 stop
    	$0 start

    	# Remember status and be quiet
    	rc_status
	  ;;
    force-reload)
    	$0 try-restart
    	rc_status
	  ;;
    reload)
	    rc_failed 3
	    rc_status -v
	  ;;
    status)
	   echo -n "Checking for service WMB "
	   /sbin/checkproc /opt/ibm/mqsi/6.0/bin/bipservice
	   rc_status -v
	  ;;
    *)
	    echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload}"
	    exit 1
	  ;;
esac
rc_exit

Note that the script will start WMB as the mqm user. If this is not what you want, you will have to make your changes at the su commands.

To use, create a file called /etc/init.d/wmb and paste the above content into. Make sure it’s runnable with:

chmod +x /etc/init.d/wmb

The run:

insserv wmb

And it should set everything nicely for you.

Note that this script is dependent on the WebSphere MQ init script, as provided in the support pac mentioned above. It will also start DB2 beforehand, if available. In my case, we are using DB2, I’m just not happy with the current script I got so I will hold off publishing it until I’m done.

I will be happy to get feedback in the comments on my attempt and will update the code above with any changes.

Technorati Tags: , , , , , , ,

Sep 18
MQ broken on video
icon1 Niklas | icon2 Tags: , , . | icon4 09 18th, 2007| icon31 Comment »

The session at Defcon where Martyn Ruks describes his findings about WebSphere MQ security is now online at Google Video. I’ve previously covered the presentation, but it’s way more interesting to hear the presentation in addition to the published slides. Thanks to the anon commenter who pointed me to the video.

Technorati Tags: , , ,

Aug 31
WebSphere MQ security
icon1 Niklas | icon2 Tags: , . | icon4 08 31st, 2007| icon32 Comments »

With all the recent fuzz on WebSphere MQ security I thought that I should try to collect the current thinking on the best way of securing your WMQ environment.

Lately, T-Rob and Martyn Ruks has held two separate presentations on WMQ security and the following posts will reflect a lot of what they presented. This combined with my experiences from securing WMQ environments make up most of the content. I hope to collect feedback from the community to improve over time.

Please note that these posts are my current understanding and should not be regarded as the final answer. I will continue to update these posts as I learn more on the topic. Any feedback is crucial so feel free to comment on any of these posts or send me an email directly. My experience is on the distributed platforms, so you’ll find very limited specifics on how to secure your z/OS environment, hopefully the community can help fill these gaps.

Most of the topics listed below are placeholders for future posts I intend to write, and this list will most likely be heavily modified before I’m done.

How to create a secure WMQ environment

  • Why secure?

  • Keep up to date

  • Policy

  • Secure by default

  • Securing channels

  • Securing queues

  • Publish/subscribe

  • Triggers and services

  • Securing applications

Breaking in to WMQ

Technorati Tags: , , , ,

Aug 31

This post is part of a series I’m doing on WebSphere MQ security.

As described in my previous post of in this series, this method of spoofing messages in WebSphere MQ should be fully attributed to T-Rob, famous from the WMQ mailing list for his deep knowledge of most odd corners in the WMQ world.

The exploit described below does not use any bug in WMQ. WMQ in this case works as documented, this is exploit relies on weakly secured WMQ environments.

As before, I’ll update this post as I learn new details around this attack or ways of protecting against it.

Description

WMQ has a feature called report messages. These are usually used by a sending application to get a receipt on whether the sent message has arrived, ended up a exception queue or was expirered. To use it is quite simple. The MQMD header (available in all WMQ messages) contain a Report attribute. Set this to the appropriate value and provide a reply queue and you will get receipts for your sent messages. You can also decide on what you want the report message to contain, either just an empty message, the first 100 bytes or the full data of the original message. In all of these, the report message will have a MsgType (in the MQMD) attribute set to the value Report (4) and a non-zero Feedback attribute (also in the MQMD).

One type of report message is the “confirm on arrival” (COA). This report message is sent by WMQ itself when a message successfully arrives on its target queue. As WMQ sends it, it is put using administrative authority.

This attack could also be done using other report types, like expiration, but COA is the attack the requires the least effort.

Now, if we get access to putting messages on any queue, for example using a hacked application or a badly secured queue manager, we can easily put a spoofed message with a Report attribute requesting a COA with the full message data. We can provide a reply-to queue and reply-to queue manager for any queue in the entire connected WMQ network. Now imagine that this message contains a spoofed message for some application on the WMQ network. An easy and effective attack unless the receiving application knows to defend itself.

Protect yourself

There is, as far as I know, no way of shutting of report messages in WMQ. It would be a good addition to WMQ security if we would be able to shut of reports on a queue manager and queue basis.

Sign your messages. Let the sending application digitally sign all messages it send and let the receiving application verify these signatures against known senders. Message with unknown or invalid signatures must be ignored, put on an exception queue and an alert raised.

Receiving applications should verify that they receive an expected message type. Normally, this would be a Datagram or a Request. All other types, and specifically Reports must be ignored, put on an exception queue and an alert raised. In fact, the built-in WMQ command server does this check and will ignore any messages which is not of the Request type. This shows that IBM is well aware of this exploit and are doing these checks themselves. Without this check in the command server, this exploit would have allowed administering WMQ.

If using JMS, here is some example code for how to check the message type:

if(!msg.propertyExists("JMS_IBM_MsgType") ||
    msg.getIntProperty("JMS_IBM_MsgType") == 1 ||
    msg.getIntProperty("JMS_IBM_MsgType") == 8) {
    // do your stuff
} else {
    // run screaming away from this message
}

Note that if you switch to another JMS provider, this code will still work as it first check for the existence of the specific IBM property.

If your using the Java Base API, this is what you could do:

if (message.messageType == MQC.MQMT_REQUEST
        || message.messageType == MQC.MQMT_DATAGRAM) {
    // do your stuff
} else {
    // run screaming away from this message
}

Code for most other WMQ API will work similarly to the Java Base example. If you got example code for any of the other APIs, feel free to send it to me and I’ll add it to this post.

Aug 17

This post is part of a series I’m doing on WebSphere MQ security.

Security in WebSphere MQ is heavily reliant on applications only being able to put messages where we want them to. Otherwise, an hacked application can be used to send messages where it is not supposed to, for example inserting spoofed messages destined for some other application. As messages commonly contain very important business data and have a high-priviliaged route into applications, spoofing messages can be a very efficient way of hacking.

Over at the WebSphere MQ mailing list, I described a simple but somewhat inefficient way of spoofing messages using the dead letter queue. T-Rob followed that by outlining an even simpler attack using confirm on arrival reports. In this first post, I’ll try to describe the first exploit and will then follow up with a post describing the exploit described by T-Rob. Please note that none of these two methods rely on any bug in WMQ, the product is working as designed.

This post is in the interest of full disclosure, what you don’t know is hard to protect yourself against.

Spoofing messages using the dead letter queue

Description

WMQ uses a shared queue called the dead letter queue as the ultimate destination for messages that can not be delivered. This might be due to many reasons, but the case of interest here is for applications that can not handle a message (due to for example resource problems or data corruption). In many cases, such an application will choose to store away this message on the dead letter queue to be able to continue processing further incoming messages.

A message stored on the dead letter queue will usually contain a header containing the orginal target queue and queue manager. The purpose of this is to automatically be able to retry the message when whatever the original problem was. The target queue can include the administration queues, or a queue on any other reachable queue manager.

On a typical system, this queue will be named SYSTEM.DEAD.LETTER.QUEUE. On a queue manager that is shared by multiple applications, a very common practice, this means that this queue might be used by several different applications.

WMQ also has what is called dead letter queue handlers, a process running that monitors the dead letter queue for messages and then, using a rules table, decide on what to do with the message. The options includes to retry for a number of times, store it somewhere else, delete it, leave it on the dead letter queue, and so on. A message that can not be successfully retried or deleted will finally be handled by an administrator, for example by fixing the troubling application and the retrying the message.

Now, if applications has put authority on the dead letter queue, it can simply fake the header containing the original target queue. The dead letter queue handler or a unobservant administrator might then try to retry the message by putting it on the faked target queue.

Protect yourself

There are a few, more or less, efficient ways of protecting against this exploit. The methods described below are not mutually exclusive, investigate what the most reasonable solution is in your case.

Don’t give applications put authority on the dead letter queue. For applications that support it, instead create a specific exception queue for only that specific application. For applications that still requires using the dead letter queue, investigate their behavior if they get an security exception when accessing it. If their failure behavior (probably rolling back the message to the original queue) is acceptable, go with that.

Let the source application digitally sign its messages and let the target application confirm the signature. This is a efficient method, albeit sometimes hard to implement due to, for example, the use off-the-shelf applications without such support.

If using a dead letter queue handler, make sure the rules table confirm that the message contain the expected USERID. It should contain the user with which the real source application accesses WMQ. It must not contain the value of any other, possibly hacked application.

Don’t retry messages targeted for the command server or any other administrative funtion.

Let the target application confirm the USERID, in the same fashion as the dead letter queue handler.

Jul 24
Defcon MQ sequrity session
icon1 Niklas | icon2 Tags: , . | icon4 07 24th, 2007| icon33 Comments »

Looks like there is going to be a very interesting session on WebSphere MQ security at Defcon. The presenter is Martyn Ruks who has a history of investigating IBM protocols. As WMQ and WMQ security in particular is of great interest to me, this session sounds like something really worth visiting. Too bad I won’t be anywhere near Vegas at the time, and I’m assuming Defcon won’t publish any video of the presentations. However, Martyn has published a presentation he held at Defcon 14 so I keep my hopes high.

Based on the published abstract, it doesn’t sound like any real new attack will be shown, but rather that Martyn will go through the usual, poor ways that WMQ are set up from a security standpoint. Fact is that at most places I’ve seen WMQ installed it has been wide open to any attacker. Most companies seems to think that it’s used internally and therefore is safe. Besides, it’s pretty invisible to most people, just humming along doing its work. Hackers on the other hand most surely know about it and how to attack it. And those of us consulting on WMQ really needs to learn the best ways of protecting an installation. And, I do think that IBM needs to do a better job of securing WMQ out of the box, currently it’s unsecure by default, something which should not be acceptable these days.

Update: this presentation is now available over at Google Video.

Technorati Tags: , , ,

Jul 12

As has already been reported in several blogs, IBM has released a new supportpac for exposing WebSphere MQ queues and topics as HTTP resources. The supportpac aims at being RESTful, at least that’s what you are lead to believe from the banner on the main WMQ site.

However, much like the ActiveMQ REST support, the RESTfulness, or even basic HTTP compliance does have it’s rough edges.

The supportpac exposes a queue on a URL looking like http://example.com/wmq/msg/queue/FOO. On this resource, three basic operations is allowed:

  • POST: will push a new message to the queue
  • GET: will peek at the first message from the queue
  • DELETE: will pop the first message from the queue

Now, there are several problems here. For example, the GET and DELETE is done at the queue URL, but actually works on a different resource, the message. The way the DELETE is done, the expectation would be to remove the queue itself. And the GET should return some representation of the queue, such as a listing of the available messages. On top of this, DELETE must be idempotent, however in this case, doing multiple DELETEs will result in very different results (different messages being read). In addition, re-doing a failed DELETE (for example due to being in-doubt because of a network failure) could lead to multiple messages being lost (note that IBM makes it clear that this is the case).

All that being said, as been shown on rest-discuss lately, designing a queue with the standard HTTP methods is not entirely straight-forward and might lead to a both complex and odd (from a queuing perspective) API. Adding simplicity in the mix, and not supporting assured delivery, the design chosen in the WMQ supportpac might be understandable.

Besides the issues above, I think a big improvement to the API would be if it supported resource discovery. The examples below describe this, in addition to suggesting a better design for pushing, peeking and popping messages:

  • http://example.com/wmq returning a list of the available queue managers
  • http://example.com/wmq/myqm returning a list of the available destinations on that specific queue manager.
  • As discussed above, the http://example.com/wmq/myqm/queue/FOO resource should return a list of available messages on that specific destination. This resource also supports the POST as described above.
  • http://example.com/wmq/myqm/queue/FOO/nextmsg redirecting to the first message of the queue
  • http://example.com/wmq/myqm/queue/FOO/123456789 identifying a specific message and supports the GET and DELETE requests as described above.

In a comment on an earlier post here, James Strachan of ActiveMQ notes that he would like to design a fully RESTful API for ActiveMQ. Maybe that could be a chance to dive deeper into this interesting area.

As noted on the Hursley WMQ blog, IBM is looking for feedback on the code, so hopefully some of these issues could be taken care of.

« Previous Entries