Admin Mistake: F5 Load Balancer SNAT IP Address Apache Logging

Background
Assume that you have a common three-tier architecture on a web farm with layers being web, application and database servers. The load balancing is performed by an F5 BIG-IP LTM 1600 load balancer and the logging takes place on the web farm that uses Apache web servers.

Problem
When you attempt to review the access logs of the Apache web servers the only IP address for all the requests is that of the F5 load balancer. Assuming that the load balancer address is 10.10.10.10, the log entries would always look like that:

10.10.10.10 - - [28/Sep/2012:15:06:18 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"
10.10.10.10 - - [28/Sep/2012:15:06:31 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"

Mistake
By default this F5 load balancer will perform SNAT (Source Network Address Translation) and this is why the requestor IP address is always the load balancer’s one.

Resolution
The solution is to utilize HTTP header field XFF. On the load balancer side you will first have to follow the below steps in the BIG-IP configuration utility:
– Go to “Local Traffic”
– Select “Profiles”
– On the “Services” menu choose “HTTP”
– Create a new profile by clicking on “Create”
– Activate “Insert X-Forwarded For” check box and select “Enabled” from the menu
– Finally click on “Update”
At last, you can use this new HTTP profile to the virtual servers you want to have the XFF HTTP header field.
Moving to the web server side you will have to create a new custom log format on the virtual hosts you want to have proper source IP address logging. So, here is an example custom log format that will include the XFF field.

LogFormat "%v %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" CUST_F5_XFF_LOG
CustomLog /somewhere/access_log CUST_F5_XFF_LOG

And assuming that the real IP address is 2.2.2.2 while the load balancer’s is 10.10.10.10, the log entries will be:

10.10.10.10 2.2.2.2 - - [28/Sep/2012:16:48:25 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"
10.10.10.10 2.2.2.2 - - [28/Sep/2012:16:41:28 +0000] "GET / HTTP/1.0" 200 228 "-" "Wget/1.12 (linux-gnu)"