xorl %eax, %eax

OpenSolaris TCP Memory Leak DoS

leave a comment »

This vulnerability was released by Sun Microsystems and it affects OpenSolaris based upon builds snv_106 through snv_126 for both x86 and SPARC architectures. The buggy code can be found at usr/src/uts/common/inet/tcp/tcp.c. Here is the exact code…

tcp_sendmsg(sock_lower_handle_t proto_handle, mblk_t *mp, struct nmsghdr *msg,
    cred_t *cr)
	tcp_t		*tcp;
	uint32_t	msize;
	conn_t *connp = (conn_t *)proto_handle;
	int32_t		tcpstate;

	/* All Solaris components should pass a cred for this operation. */
	ASSERT(cr != NULL);

	ASSERT(connp->conn_ref >= 2);
	ASSERT(connp->conn_upper_handle != NULL);

	if (msg->msg_controllen != 0) {
		return (EOPNOTSUPP);

	switch (DB_TYPE(mp)) {
	case M_DATA:
		 * The application may pass in an address in the msghdr, but
		 * we ignore the address on connection-oriented sockets.
		 * Just like BSD this code does not generate an error for
		 * TCP (a CONNREQUIRED socket) when sending to an address
		 * passed in with sendto/sendmsg. Instead the data is
		 * delivered on the connection as if no address had been
		 * supplied.

		if (msg != NULL && msg->msg_flags & MSG_OOB) {
			SQUEUE_ENTER_ONE(connp->conn_sqp, mp,
			    tcp_output_urgent, connp, tcp_squeue_flag,
		} else {
			SQUEUE_ENTER_ONE(connp->conn_sqp, mp, tcp_output,
			    connp, tcp_squeue_flag, SQTAG_TCP_OUTPUT);

		return (0);


	return (0);

This is a well known function, as you can see after initializing the ‘connp’ pointer with the ‘proto_handle’ pointer, it performs some basic assertions. These are that: there are some initialized credentials for that socket, the connection’s reference counter is at least two and the ‘connp->conn_upper_handle’ which contains the upper handle for sockfs transport layer is not NULL.
Following, function tcp_sendmsg() will immediately return EOPNOTSUPP (aka Operation not supported on socket) in case of a ‘msg->msg_controllen’ not equal to zero. This variable is a ‘socklen_t’ counter that is used to contain the length of ancillary data buffer. However, as you can easily deduce, the ‘mp’ pointer which contains the message block descriptor as we can read from usr/src/stand/lib/sock/socket_impl.h will never be freed.

 * Message block descriptor copied from usr/src/uts/common/sys/stream.h.
 * We need to do that to simplify the porting of TCP code from core
 * kernel to inetboot.  Note that fields which are not used by TCP
 * code are removed.
typedef struct  msgb {
	struct  msgb	*b_next;
	struct  msgb	*b_prev;
  	struct  msgb	*b_cont;
 	unsigned char	*b_rptr;
 	unsigned char	*b_wptr;
 	unsigned char	*b_datap;
 	size_t		b_size;
} mblk_t;

This is a classic resource consuming DoS condition. The bug was fixed by adding the missing free routine like this:

	ASSERT(connp->conn_upper_handle != NULL);

	if (msg->msg_controllen != 0) {
+		freemsg(mp);
		return (EOPNOTSUPP);

	switch (DB_TYPE(mp)) {

Where freemsg() is a simple function located at usr/src/stand/lib/sock/socket.c which is a wrapper around freeb().

freeb(mblk_t *mp)
#ifdef DEBUG
 	printf("freeb datap %x\n", mp->b_datap);
	bkmem_free((caddr_t)(mp->b_datap), mp->b_size);
#ifdef DEBUG
 	printf("freeb mp %x\n", mp);
 	bkmem_free((caddr_t)mp, sizeof (mblk_t));

freemsg(mblk_t *mp)
 	while (mp) {
 		mblk_t *mp_cont = mp->b_cont;
 		mp = mp_cont;

Written by xorl

November 12, 2009 at 21:47

Posted in bugs, solaris

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s