xorl %eax, %eax

CVE-2009-1377: OpenSSL DTLS epoch record buffer memory DoS

leave a comment »

This bug was also found by Daniel Mentz. The issue was disclosed on 16 May 2009 and affects OpenSSL 0.9.8 up to, and including 0.9.8k. This is a design flaw in the records that arrive with a future epoch. Here is the vulnerable function from ssl/d1_pkt.c:

164 static int
165 dtls1_buffer_record(SSL *s, record_pqueue *queue, PQ_64BIT priority)
166 {
167     DTLS1_RECORD_DATA *rdata;
168         pitem *item;
170         rdata = OPENSSL_malloc(sizeof(DTLS1_RECORD_DATA));
171         item = pitem_new(priority, rdata);
209         return(1);
210     }

It is kind of obvious, this function immediately allocates space (line 170) for the record’s data but a malicous user could send numerous datagrams for records that have future epochs (meaning that they will be processed after finishing the corresponding handshake). This can lead to memory exhaustion. To fix this, the following function was added in crypto/pqueue/pqueue.c:

+pqueue_size(pqueue_s *pq)
+	pitem *item = pq->items;
+	int count = 0;
+	while(item != NULL)
+	{
+		count++;
+		item = item->next;
+	}
+	return count;

Which is a simple counter of the items in the pqueue_s pointer passed to it as an argument. This is used in dtls1_buffer_record() like this from now on:

 pitem *item;

+	/* Limit the size of the queue to prevent DOS attacks */
+	if (pqueue_size(queue->q) >= 100)
+		return 0;
 rdata = OPENSSL_malloc(sizeof(DTLS1_RECORD_DATA));

Using this, the maximum records to be buffered for future epochs are only 100.

Written by xorl

May 18, 2009 at 21:55

Posted in bugs

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s