xorl %eax, %eax

Admin Mistake: F5 Load Balancer SNAT IP Address Apache Logging

with 10 comments

Background
Assume that you have a common three-tier architecture on a web farm with layers being web, application and database servers. The load balancing is performed by an F5 BIG-IP LTM 1600 load balancer and the logging takes place on the web farm that uses Apache web servers.

Problem
When you attempt to review the access logs of the Apache web servers the only IP address for all the requests is that of the F5 load balancer. Assuming that the load balancer address is 10.10.10.10, the log entries would always look like that:

10.10.10.10 - - [28/Sep/2012:15:06:18 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"
10.10.10.10 - - [28/Sep/2012:15:06:31 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"

Mistake
By default this F5 load balancer will perform SNAT (Source Network Address Translation) and this is why the requestor IP address is always the load balancer’s one.

Resolution
The solution is to utilize HTTP header field XFF. On the load balancer side you will first have to follow the below steps in the BIG-IP configuration utility:
– Go to “Local Traffic”
– Select “Profiles”
– On the “Services” menu choose “HTTP”
– Create a new profile by clicking on “Create”
– Activate “Insert X-Forwarded For” check box and select “Enabled” from the menu
– Finally click on “Update”
At last, you can use this new HTTP profile to the virtual servers you want to have the XFF HTTP header field.
Moving to the web server side you will have to create a new custom log format on the virtual hosts you want to have proper source IP address logging. So, here is an example custom log format that will include the XFF field.

LogFormat "%v %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" CUST_F5_XFF_LOG
CustomLog /somewhere/access_log CUST_F5_XFF_LOG

And assuming that the real IP address is 2.2.2.2 while the load balancer’s is 10.10.10.10, the log entries will be:

10.10.10.10 2.2.2.2 - - [28/Sep/2012:16:48:25 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"
10.10.10.10 2.2.2.2 - - [28/Sep/2012:16:41:28 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"
About these ads

Written by xorl

September 28, 2012 at 10:00

10 Responses

Subscribe to comments with RSS.

  1. Actually, by default a virtual server in LTM does *not* SNAT. SNAT is turned on when administrators deploy a configuration where the routing does not flow back through the BIG-IP such as a single arm deployment.

    James

    September 28, 2012 at 14:58

  2. Nice to know. Thank you James.

    xorl

    September 28, 2012 at 16:23

  3. You can use mod_rpaf on apache side.

    Eldar Zaitov (@kyprizel)

    September 28, 2012 at 16:26

  4. Thanks for the tip! In general I tend not to use additional software if the application is able to perform the required task.

    xorl

    September 28, 2012 at 18:55

  5. nice of you to actually read the F5 Support website!

    https://support.f5.com/kb/en-us/solutions/public/4000/800/sol4816.html

    Qunicy

    September 28, 2012 at 21:55

  6. Qunicy I actually read the documentation (Chapter 6: Managing Application Layer Traffic) which you can find here: http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm_configuration_guide_10_0_0/ltm_services_profiles.html#1190750

    But I think your link is much better.

    xorl

    September 29, 2012 at 04:28

  7. Since XFF is cumulative across proxy hops and is frequently spoofed, it’s advisable to use the Request Header Insert feature in the HTTP Profile. There, you can customize the header name to something like Real_Client_IP which your web servers can trust.

    Brian McHenry

    October 1, 2012 at 00:09

  8. Thank you Brian. I’ll look into that.

    xorl

    October 1, 2012 at 11:29

  9. will almighty xorl will be in ccc this year ?

    urmom

    November 19, 2012 at 11:38

  10. Unfortunately I don’t think so. Have fun!

    xorl

    November 20, 2012 at 13:59


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 64 other followers