CVE-2009-2463: Mozilla Firefox/Thunderbird Base64 Integer Overflows

This bug was reported by monarch2020 and disclosed on 21 July 2009 by Mozilla. The issue affects specifically the Base64 routines in Mozilla Firefox prior to 3.0.12 release. This code can be found at nsprpub/lib/libc/src/base64.c of 3.0.11 release of Firefox like this.

/*
 * PL_Base64Encode
 *
 * If the destination argument is NULL, a return buffer is 
 * allocated, and the data therein will be null-terminated.  
 * If the destination argument is not NULL, it is assumed to
 * be of sufficient size, and the contents will not be null-
 * terminated by this routine.
 *
 * Returns null if the allocation fails.
 */

PR_IMPLEMENT(char *)
PL_Base64Encode
(
    const char *src,
    PRUint32    srclen,
    char       *dest
)
{
    if( 0 == srclen )
    {
        srclen = PL_strlen(src);
    }

    if( (char *)0 == dest )
    {
        PRUint32 destlen = ((srclen + 2)/3) * 4;
        dest = (char *)PR_MALLOC(destlen + 1);
        if( (char *)0 == dest )
        {
            return (char *)0;
        }
        dest[ destlen ] = (char)0; /* null terminate */
    }

    encode((const unsigned char *)src, srclen, (unsigned char *)dest);
    return dest;
}

So, the above encoding routine uses PL_strlen() to retrieve the length of ‘src’ string value. However, strlen() returns size_t which in some architectures is defined as unsigned long. This might lead to truncation on some 64-bit systems since ‘srclen’ is always 32-bit long, unsigned integer as you can read from the above routine. In addition to this, another bug is present in the second if clause. Here, the calculation of ‘destlen’ can easily result into an integer overflow and consequently to heap memory corruption. Both bugs were fixed by applying this patch:

     if( 0 == srclen )
     {
-        srclen = PL_strlen(src);
+        size_t len = strlen(src);
+        srclen = len;
+        /* Detect truncation. */
+        if( srclen != len )
+        {
+            return (char *)0;
+        }
     }
 
     if( (char *)0 == dest )
     {
-        PRUint32 destlen = ((srclen + 2)/3) * 4;
+        PRUint32 destlen;
+        /* Ensure all PRUint32 values stay within range. */
+        if( srclen > (PR_UINT32_MAX/4) * 3 )
+        {
+            return (char *)0;
+        }
+        destlen = ((srclen + 2)/3) * 4;
         dest = (char *)PR_MALLOC(destlen + 1);

Now, a temporary variable ‘len’ of size_t type is used to detect possible truncations in 32-bit long integer ‘srclen’, and the calculation in the second if is checked for overflows before proceeding to the allocation using PR_MALLOC(). Similar vulnerabilities were also present in Base64 decoding routine from nsprpub/lib/libc/src/base64.c.