This is an overall page for an Apache reverse proxy, including installation, configuration, security and cache.

The reverse proxy in my case is a virtual. It will have three NICs, one for the outside, one for the inside and one in the management VLAN. During initial installation, we only use one NIC, the one on the outside. Later on we'll add the other NICs.

• VM NAME: SLES11-REVERSEPROXY
• CPU: 1
• MEM: 1024 MB
• DSK: 8 GB
• NIC: VMXNET3
• Extern IP: 10.10.10.20
  • VLAN: 10
• Intern IP: 10.10.12.120
  • VLAN: 12
• Mgmt IP: 10.10.253.120
  • VLAN: 253

Installation media: sles11 sp1 for vmware, 64 bits

• Language: English US
• Keyboard: English US
• Type installation: New Install
• Time and Timezone: Hardware Clock not set to local time - Europe - Netherlands

/dev/sda1 swap 1 GB
/dev/sda2 / ext3 7 GB - No Access Time

• Pattern:
  • Base System
  • 32 bits runtime environment
  • Minimal System (Appliance) - could not be unselected
• Graphical Environments
  • Gnome Desktop Environment for Server
  • X Window System
• Development
  • C/C++ Compiler and Tools

Note: I want to set runlevel 3 as the default runlevel but because I'm installing over VNC I won't do that untill after the installation.

• Root password: T-password
• Hostname: reverseproxy.getshifting.local
• IP-adres: 10.10.10.20
• Subnet Mask: 255.255.255.0
• Gateway: 10.10.10.1
• DNS server: 10.10.12.53

• IPv6 Disabled
• Firewall Enabled
• SSH Port Opened
• Remote Administration Enabled
• No Proxy Configured

• Not registered at Novell Customer Center

• CA configuration left at default
• No OpenLDAP

• Local Authentication
• No extra accounts

• Set the runlevel to 3
• Disable Remote Administration (VNC)

Reboot the box to commit all changes.

• /etc/ntp.conf:

server 10.10.12.123 prefer
tinker step 0

Set up LDAP Authentication as described in AD LDAP For Linux

Set up sudo as described in Sudo

For monitoring purposes and the automatic switching between “Production” and “Outage” we need some additional lines:

# Command Alias
Cmnd_Alias   MON = /bin/mv, /etc/init.d/apache2 reload

and

# User privilege specification
monuser ALL=(root) NOPASSWD: MON

Configure email
Change these lines in the /etc/postfix/main.cf file:

#myhostname = reverseproxy.getshifting.local
myhostname = getshifting.com
relayhost = mail.getshifting.local

Note: myhostname should be a real domain name if the relay server checks for domain names

* Restart postfix: /etc/init.d/postfix restart

• Add .forward files in homedirs of root and monuser to sjoerd_getshifting_com

Add the server to your

• syslog
• application monitoring

Add the server to your DNS server with the correct DNS and reverse DNS records

Install apache through yast:

• Start /sbin/yast2
• Go to Software → Software Management
• Search for apache
• Select Apache2
• Accept all dependencies

• Start yast
• Go to network services
• Select the “HTTP Server”
• Enter port 443 and only listen on interface 10.10.10.20
• Checkmark the box to “Open Port in Firewall”
• Do not enable any of the listed scripting languages
• Keep all of the other defaults

After going through the wizard, open “HTTP Server” again and enable these “Server Modules”:

• proxy
• Proxy_connect
• proxy_http
• rewrite

Only this:

sjoerd@reverseproxy:/etc/apache2> cat listen.conf | grep -v '^#' | grep -v '^$'
<IfDefine SSL>
    <IfDefine !NOSSL>
        <IfModule mod_ssl.c>
            Listen 443
        </IfModule>
    </IfDefine>
</IfDefine>

Open “HTTP Server” again in yast2 and enable these “Server Modules”:

• ssl

Copy the key files from the keystore and place them like this:

• /etc/apache2/ssl.crt/secure.getshifting.com.crt
• /etc/apache2/ssl.key/secure.getshifting.com.key
• /etc/apache2/ssl.crt/intermediate.crt

Copy the Outage, Error and Maintenance pages to “/srv/www/htdocs/local”, each in a separate directory:

sjoerd@reverseproxy:/srv/www/htdocs/local> ll
drwxr-xr-x 6 root root 4096 2011-01-19 11:29 maintenance
drwxr-xr-x 4 root root 4096 2011-01-19 11:30 outage
drwxr-xr-x 4 root root 4096 2011-01-19 11:31 error

Also, copy the favicon.ico to “/srv/www/htdocs/”.

NOTE: When using custom error pages, and when these error pages use for example images make sure you use direct links to these images, from the documentroot. If you don't, Apache will look for the images in the directory where the user is, which could be non-existent, wrong and almost never the directory where the images really are.

/etc/apache2/vhosts.d # cat getshifting-production.conf
<IfDefine SSL>
<IfDefine !NOSSL>

<VirtualHost _default_:443>

# Enable proxying
ProxyRequests off
ProxyPreserveHost on
ProxyVia off
SSLProxyEngine on

# Enable rewriting
RewriteEngine on
#RewriteLogLevel 2
RewriteLog /var/log/apache2/rewrite.log
RewriteRule ^/start$ /start/ [R]
RewriteRule ^/startdocuments$ /startdocuments/ [R]

#  General setup for the virtual host
DocumentRoot "/srv/www/htdocs"

# Error Logging
ErrorLog /var/log/apache2/error_log
CustomLog /var/log/apache2/access_log combined
# LogLevel info

# SSL Settings
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /etc/apache2/ssl.crt/secure.getshifting.com.crt
SSLCertificateKeyFile /etc/apache2/ssl.key/secure.getshifting.com.key
SSLCACertificateFile /etc/apache2/ssl.crt/intermediate.crt

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>

<Directory "/srv/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

# Error Documents
# Customized error pages are in documentroot/error/ and included in error.conf
ProxyErrorOverride On

# Proxying rules
# Do not proxy !
ProxyPass /local/outage !
ProxyPass /local/maintenance !
ProxyPass /local/error !
ProxyPass /favicon.ico !

# Follow these proxy rules in this order
ProxyPass /startdocuments https://10.10.12.20/startdocuments
ProxyPassReverse /startdocuments https://10.10.12.20/startdocument
ProxyPass /start https://10.10.12.20/start
ProxyPassReverse /start https://10.10.12.20/start
ProxyPass / https://10.10.12.20/start/
ProxyPassReverse / https://10.10.12.20/start/

</VirtualHost>

</IfDefine>
</IfDefine>

/etc/apache2/vhosts.d # cat getshifting-outage.conf_disabled
<IfDefine SSL>
<IfDefine !NOSSL>

<VirtualHost _default_:443>

# Enable proxying
ProxyRequests off
ProxyPreserveHost on
ProxyVia off
SSLProxyEngine On

# Enable rewriting
RewriteEngine on
#RewriteLogLevel 2
RewriteLog /var/log/apache2/rewrite.log
RewriteRule ^/$ /start/ [R]
RewriteRule ^/startdocuments/* /start/ [R]

#  General setup for the virtual host
DocumentRoot "/srv/www/htdocs/local/outage"

# Error Logging
ErrorLog /var/log/apache2/error_log
CustomLog /var/log/apache2/access_log combined
# LogLevel info

# SSL Settings
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /etc/apache2/ssl.crt/secure.getshifting.com.crt
SSLCertificateKeyFile /etc/apache2/ssl.key/secure.getshifting.com.key
SSLCACertificateFile /etc/apache2/ssl.crt/intermediate.crt

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>

<Directory "/srv/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

# Error Documents
# Customized error pages are in documentroot/error/ and included in error.conf
Alias /error/ "/srv/www/htdocs/local/error/"
<Directory "/srv/www/htdocs/local/error">
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>
ProxyErrorOverride On

# Proxying rules
# Do not proxy !
ProxyPass /local/outage !
ProxyPass /local/maintenance !
ProxyPass /local/error !
ProxyPass /favicon.ico !

<Directory "/srv/www/htdocs/local/outage">
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>

</VirtualHost>

</IfDefine>
</IfDefine>

<IfDefine SSL>
<IfDefine !NOSSL>

<VirtualHost _default_:443>

# Enable proxying
ProxyRequests off
ProxyPreserveHost on
ProxyVia off
SSLProxyEngine On

# Enable rewriting
RewriteEngine on
#RewriteLogLevel 2
RewriteLog /var/log/apache2/rewrite.log
RewriteRule ^/$ /start/ [R]
RewriteRule ^/startdocuments/* /start/ [R]

#  General setup for the virtual host
DocumentRoot "/srv/www/htdocs/local/maintenance"

# Error Logging
ErrorLog /var/log/apache2/error_log
CustomLog /var/log/apache2/access_log combined
# LogLevel info

# SSL Settings
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /etc/apache2/ssl.crt/secure.getshifting.com.crt
SSLCertificateKeyFile /etc/apache2/ssl.key/secure.getshifting.com.key
SSLCACertificateFile /etc/apache2/ssl.crt/intermediate.crt

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
</Files>

<Directory "/srv/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

# Error Documents
# Customized error pages are in documentroot/error/ and included in error.conf
Alias /error/ "/srv/www/htdocs/local/error/"
<Directory "/srv/www/htdocs/local/error">
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>
ProxyErrorOverride On

# Proxying rules
# Do not proxy !
ProxyPass /local/outage !
ProxyPass /local/maintenance !
ProxyPass /local/error !
ProxyPass /favicon.ico !

<Directory "/srv/www/htdocs/local/maintenance">
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>

</VirtualHost>

</IfDefine>
</IfDefine>

Note that it is possible to rewrite an url to something external as well:

^/start$ https://www.maintenancepage.com/systemdown/ [R] 

This is an example of a non-secure vhost:

<VirtualHost _default_:80>

# Enable proxying
ProxyRequests off
ProxyPreserveHost on
ProxyVia off

# Enable rewriting
RewriteEngine on
#RewriteLogLevel 2
RewriteLog /var/log/apache2/rewrite.log
RewriteRule ^/start$ /start/ [R]
RewriteRule ^/startdocuments$ /startdocuments/ [R]

#  General setup for the virtual host
DocumentRoot "/srv/www/htdocs"

# Error Logging
ErrorLog /var/log/apache2/error_log
CustomLog /var/log/apache2/access_log combined
# LogLevel info

# Error Documents
# Customized error pages are in documentroot/error/ and included in error.conf
ProxyErrorOverride On

# Proxying rules
# Do not proxy !
ProxyPass /local/outage !
ProxyPass /local/maintenance !
ProxyPass /local/error !
ProxyPass /favicon.ico !

# Follow these proxy rules in this order
ProxyPass /startdocuments http://10.10.12.20/startdocuments
ProxyPassReverse /startdocuments http://10.10.12.20/startdocument
ProxyPass /start http://10.10.12.20/start
ProxyPassReverse /start http://10.10.12.20/start
ProxyPass / http://10.10.12.20/start/
ProxyPassReverse / http://10.10.12.20/start/

</VirtualHost>

Please note that the ErrorDocument 500 with the message displayal will not work in Internet Explorer with the option “Show friendly HTTP error messages” enabled.

Goal: When the monitoring server sees specific error messages a script should run activating the 'outage' page and warn me.
Monitoring:

• ManageEngine Application Manager 9:

Create new Monitor:

• In AppManager click 'New Monitor' and select 'HTTP(s) URLs
• Fill in data:
  • Display Name: Secure Website
  • Url address: https://10.10.12.20/start/
  • Check for specific html content: Secure GetShifting
  • Error if Match: Internal Server Error
• (keep rest default)
• save

Go to the monitor and wait for data collection to start. Check that health is ok.
Configure Action:

• In AppManager click 'Actions' and select 'Execute Program'
• Fill in data:
  • Display Name: Start Outage Page in ACC
  • Script Location: Remote Server (10.10.10.20 (monuser))
  • Program To execute: /opt/Tools/Scripts/outagepage.sh start
  • Directory to execute Program: opt/Tools/Scripts
• (keep rest default)

Create the action and test it (start with a script that just creates a file in temp)
Als create one for switching back:

• Program To execute: /opt/Tools/Scripts/outagepage.sh stop

• outagepage.sh

#!/bin/bash
########################################################################################################################
# Author : Sjoerd Hooft
# Date Initial Version: 18 Jan 2011
# Comments: sjoerd_getshifting_com
#
# Description:
# This is a the script to automatically switch between two Apache vhost. Default is production, fallback is Outage.
#
# Recommendations:
# The script is designed for a 120 column terminal.
# The running user must meet these requirements:
# - monitoring user 
# - sudo commands for renaming config files and reloading apache
# - have write permissions in the base directory
#
# Changes:
# Please comment on your changes to the script (your name and email address, line number, description):
########################################################################################################################
 
# Script Variables
HOSTNAME_SHORT=`hostname -s`
BASEDIR=`dirname $0`
DATE=`date +%Y%m%d%H%M`
LOGFILE="$BASEDIR/outagepage$DATE.log"
TOMAIL=sjoerd_getshifting_com
 
# Apache directory
VHOSTS="/etc/apache2/vhosts.d"
 
# Redirecting all output to logfile
exec > $LOGFILE 2>&1
 
case "$1" in
 
start )
   echo "Switching from Production Page to Outages Page"
   sudo mv $VHOSTS/getshifting-production.conf $VHOSTS/getshifting-production.conf_disabled
   sudo mv $VHOSTS/getshifting-outage.conf_disabled $VHOSTS/getshifting-outage.conf
   sudo /etc/init.d/apache2 reload
   echo "Switching from Production Page to Outages Page... done"
   cat $LOGFILE | mail -s "ALERT - ${HOSTNAME_SHORT} switched to OUTAGEPAGE" $TOMAIL
;;
 
stop )
   echo "Switching from Outages Page to Production Page"
   sudo mv $VHOSTS/getshifting-production.conf_disabled $VHOSTS/getshifting-production.conf
   sudo mv $VHOSTS/getshifting-outage.conf $VHOSTS/getshifting-outage.conf_disabled
   sudo /etc/init.d/apache2 reload
   echo "Switching from Outage Page to Production Page... done"
   cat $LOGFILE | mail -s "ALERT CLEARED - ${HOSTNAME_SHORT} switched back to PRODUCTION" $TOMAIL
;;
 
* )
   echo
   echo "Usage: $0 COMMAND"
   echo "---------------------------"
   echo "start        - Switch from Production Page to Outage Page"
   echo "stop         - Switch from Outage Page to Production Page"
   echo
   exit 1
;;
esac
 
exit

Outages page gets activated! Success!
When the Application Server is back up, the production page gets activated again.

Outages page gets activated! Success!
When the webserver is up again, the production page gets activated again.

Since this is the reverse proxy, meaning first real point of access from outside, we want added security. This means a setup of three nics:

• Extern IP: 10.10.10.20
  • Purpose: external access to the application
  • Added to the demilitirized zone in the firewall
• Intern IP: 10.10.12.120
  • Purpose: internal access to the appplication and direct access to the application for customers
  • Added to the internal zone in the firewall
• Mgmt IP: 10.10.253.120
  • Purpose: Monitoring SSH access
  • Added to the external zone in the firewall

• Shutdown VM
• Add NICs in correct VLANs
• Startup VM
• Yast2 → Network Devices → Network Settings
• Configure NICs

Test if you can access the proxy over the new nic

• yast2 → Security and Users → firewall
  • Interfaces , set the new interface (eth1) in the Internal Zone
  • Allowed Services:
    • Internal Zone: HTTPS
    • External Zone: SSH
    • Demilitirized Zone: HTTPS
    • Logging Level:
      • Logging Accepted Packets: Log Only Critical
      • Logging Not Accepted PacketsL Log All

Please note that caching will only work for non-secure data. It is not possible to cache data from a HTTPS url.
To configure caching, we'll first have to enable it in apache:

• Start yast
• Go to network services
• Select the “HTTP Server”
• Go to “Server Modules”:
• Enable these modules
  • cache
  • diskcache

Save the changes.

Note: If you forget to enable the cache module you'll get this warning:

sjoerd@reverseproxy:/etc/apache2/vhosts.d> sudo /etc/init.d/apache2 restart
httpd2-prefork: Syntax error on line 116 of /etc/apache2/httpd.conf: Syntax error on line 26 of /etc/apache2/sysconfig.d/loadmodule.conf: Cannot load /usr/lib64/apache2-prefork/mod_disk_cache.so into server: /usr/lib64/apache2-prefork/mod_disk_cache.so: undefined symbol: ap_cache_cacheable_hdrs_out

Add a second disk to the VM and configure it to mount on /var/cache/apache, the default location for apache cache.

• Disk size: 8 GB
• file system ext3, no access time
• Mountpoint: /var/cache/apache

reverseproxy:~ # mount
...
/dev/sdb1 on /var/cache/apache type ext3 (rw,noatime,acl,user_xattr)

And set the owner:

sudo chown -R wwwrun:root /var/cache/apache

Set the configuration below inside the vhost config file.

# Caching
CacheRoot /var/cache/apache
CacheEnable disk /
CacheDirLevels 1
CacheDirLength 1
CacheDefaultExpire 7200
CacheMaxExpire 86400
CacheIgnoreNoLastMod On
CacheMaxFileSize 2048000
CacheStorePrivate On

http://httpd.apache.org/docs/2.2/caching.html
http://httpd.apache.org/docs/2.2/mod/mod_disk_cache.html
http://www.mnot.net/cache_docs/
http://en.wikipedia.org/wiki/List_of_HTTP_status_codes

This is not really possible, if you need “ProxyPreserveHost on” in Apache. Our application needs that to work through a reverse proxy, setting it to off breaks it. We wanted to configure the reverse proxy from https on the outside, to http on the inside but that seems impossible. It is either http to http, or https to https. I tested both, and they work, but unfortunely switching from https on the outside to http on the inside does not. I experimented with rewriterules, requestheader, and a couple of settings more, no luck.

We want to offload the application webserver as much as possible which means we'll also implement mod_security on the reverse proxy. This will offload and simplify the application webserver.
Mod Security 2.x has these requirements:

• Apache 2.2.x (highly recommended)
• Apache module mod_unique_id
• libapr & libapr-util
• libpcre
• libxml2

All modules are already installed by default. Note that libpcre is known as 'pcre' on SLES.
You just have to enable the module mod_unique_id as it is not enabled by default. Restart to make your changes effective and run httpd2 -M to see if all modules are loaded. If everything is loaded stop apache.

Make sure you have access to the SLES SDK Sources. Since we have an SLES Installation Update Server 11 I could download the SDK ISO DVD1 (which holds all required files) and add it to my software repository.
The ISO can be downloaded from here: SLES 11 SP1 SDK Download (A Novell account is required). The file you need to download is called: SLE-11-SP1-SDK-DVD-x86_64-GM-DVD1.iso
After installing, it's mostly just a module but not entirely. You need to to enable two modules now, the mod_unique_id which is a normal module and can be enabled the normal way:

• yast2 → network services → http server → server modules
• select the module and enable it

The second module to enable is mod_security. Since it is not recognized by apache as a module we have to manually add the module to the modulelist.
Find the APACHE_MODULES in the apache2 sysconfig file and add the module like below:

reverseproxy:/var/log/apache2 # vi /etc/sysconfig/apache2
APACHE_MODULES="authz_host actions alias auth_basic authz_groupfile authn_file authz_user autoindex cgi dir include log_config mime negotiation setenvif status userdir asis cache disk_cache imagemap proxy proxy_connect proxy_http rewrite ssl unique_id authz_default mod_security2"

Restart apache and check wether the modules are running by issuing the 'httpd2 -M' command:

reverseproxy:/var/log/apache2 # httpd2 -M
Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 authz_host_module (shared)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authz_groupfile_module (shared)
 authn_file_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 dir_module (shared)
 include_module (shared)
 log_config_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 setenvif_module (shared)
 status_module (shared)
 userdir_module (shared)
 asis_module (shared)
 cache_module (shared)
 disk_cache_module (shared)
 imagemap_module (shared)
 proxy_module (shared)
 proxy_connect_module (shared)
 proxy_http_module (shared)
 rewrite_module (shared)
 ssl_module (shared)
 unique_id_module (shared)
 authz_default_module (shared)
 security2_module (shared)
Syntax OK

Mod security has a default configuration file, and comes with a core rule set. The configuration works with include files which work for the modsecurity part like this:

httpd.conf
  |
  |-- default-server.conf . . . . . . . . .  set up the default server that replies to non-virtual-host requests
  |    `--conf.d/mod_security2.conf  . . . . enable mod-security default configuration
  |
  `--conf.d/modsecurity/*.conf  . . . . . .  add the core rule set

Since this include structure is not enabled by default (because the core rule set is not enabled by default) we have to include the core rule set manually.

• Create the correct directories and copy the core rule set config files to this directory:

reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # mkdir /etc/apache2/conf.d/modsecurity
reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # cp *.conf /etc/apache2/conf.d/modsecurity
reverseproxy:/usr/share/doc/packages/apache2-mod_security2/rules # cd /etc/apache2/conf.d/modsecurity
reverseproxy:/etc/apache2/conf.d/modsecurity # ll
-rw-r--r-- 1 root root 12325 Jan 31 14:03 modsecurity_crs_10_config.conf
-rw-r--r-- 1 root root  5164 Jan 31 14:03 modsecurity_crs_20_protocol_violations.conf
-rw-r--r-- 1 root root  3538 Jan 31 14:03 modsecurity_crs_21_protocol_anomalies.conf
-rw-r--r-- 1 root root  2496 Jan 31 14:03 modsecurity_crs_23_request_limits.conf
-rw-r--r-- 1 root root  6399 Jan 31 14:03 modsecurity_crs_30_http_policy.conf
-rw-r--r-- 1 root root  2720 Jan 31 14:03 modsecurity_crs_35_bad_robots.conf
-rw-r--r-- 1 root root 28726 Jan 31 14:03 modsecurity_crs_40_generic_attacks.conf
-rw-r--r-- 1 root root  2463 Jan 31 14:03 modsecurity_crs_45_trojans.conf
-rw-r--r-- 1 root root  8268 Jan 31 14:03 modsecurity_crs_50_outbound.conf

• Add the include line for the core rule set in the httpd.conf:

# Include Mod Security Core Rule Set
Include /etc/apache2/conf.d/modsecurity/*.conf

Now we will configure the config files themselves to run modsecurity first in DetectionOnly mode to prevent the risk for false positives. We also set the logfiles correctly:

vi /etc/apache2/conf.d/mod_security2.conf:
# Basic configuration options
#SecRuleEngine On
SecRuleEngine DetectionOnly

vi /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
SecRuleEngine DetectionOnly
SecAuditLog /var/log/apache2/modsec_audit.log
SecDebugLog             /var/log/apache2/modsec_debug.log
SecDebugLogLevel        3

Now restart apache:

reverseproxy:/var/log/apache2 # /etc/init.d/apache2 start
Starting httpd2 (prefork) [Mon Jan 31 14:30:35 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:30:35 2011] [warn] worker http://10.10.12.20/start already used by another worker

Core Rule Set Structure & Usage
====================================

To activate the rules for your web server installation:

  1) You may want to edit and customize modsecurity_crs_10_config.conf.
     Additionally you may want to edit modsecurity_crs_30_http_policy.conf
     which enforces an application specific HTTP protocol usage.

  2) Add the following line to your httpd.conf (assuming
     you've placed the rule files into conf/modsecurity/):

     Include conf/modsecurity/*.conf

  3) Restart web server.

  4) Make sure your web sites are still running fine.


Core Rule Set Content
=========================

In order to provide generic web applications protection, the Core Rule Set
uses the following techniques:

1. HTTP protection - detecting violations of the HTTP protocol and a locally
defined usage policy.

2. Common Web Attacks Protection - detecting common web application security
attack.

3. Automation detection - Detecting bots, crawlers, scanners and other surface
malicious activity.

4. Trojan Protection - Detecting access to Trojans horses.

5. Errors Hiding - Disguising error messages sent by the server

In addition the rule set also hints at the power of ModSecurity beyond
providing security by reporting access from the major search engines to your
site.

HTTP Protection - This first line of protection ensures that all abnormal HTTP
requests are detected. This line of defense eliminates a large number of
automated and non targeted attacks as well as protects the web server itself.
Common Web Attacks Protection Rules on the second level address the common web
application security attack methods. These are the issues that can appear in
any web application. Some of the issues addressed are:

- SQL Injection
- Cross-Site Scripting (XSS)
- OS Command execution
- Remote code inclusion
- LDAP Injection
- SSI Injection
- Information leak
- Buffer overflows
- File disclosure

Automation Detection - Automated clients are both a security risk and a
commercial risk. Automated crawlers collect information from your site, consume
bandwidth and might also search for vulnerabilities on the web site. Automation
detection is especially useful for generic detection of comments spam.


Trojan Protection - ModSecurity Core Rule Set detects access to back doors
installed on a web server. This feature is very important in a hosting
environment when some of this backdoors may be uploaded in a legitimate way and
used maliciously. In addition the Core Rule Set includes a hook for adding
an Anti-Virus program such as ClamAV for checking file uploads.

Errors Hiding - If all fails, the Core Rule Set will detect errors sent by
the web server. Detecting and blocking errors prevents attackers from
collecting reconnaissance information about the web application and also server
as a last line of defense in case an attack was not detected eariler.

Few Word of Caution
-------------------

As with every new technology, using the ModSecurity Core Rule Set requires some caution:

- Every Rule Set can have false positive in new environments and any new
installation should initially use the log only Rule Set version or if no such
version is available, set ModSecurity to Detection only using the SecRuleEngine
DetectionOnly command.

After running ModSecurity in a detection only mode for a while review the evens
generated and decide if any modification to the rule set should be made before
moving to protection mode.

From the mod security manual:

SecRuleEngine

Description: Configures the rules engine.
Syntax: SecRuleEngine On|Off|DetectionOnly
Example Usage: SecRuleEngine On
Processing Phase: Any
Scope: Any
Version: 2.0.0
Dependencies/Notes: This directive can also be controlled by the ctl action (ctl:ruleEngine=off) for per rule processing.
Possible values are:
* On - process rules.
* Off - do not process rules.
* DetectionOnly - process rules but never intercept transactions, even when rules are configured to do so.

Mod security is now configured as detection only. For now, we keep it like this, closely monitoring the mod security logfiles for false positives. When we are sure there are no more false positives (or at least nothing our customers will notice) we can set the SecRuleEngine to On.

This blog also explains how to deal with false positives: Handling False Positives

Starting httpd2 (prefork) [Mon Jan 31 14:20:51 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:20:51 2011] [warn] worker http://10.10.12.20/start already used by another worker
Syntax error on line 53 of /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
Invalid command 'SecRuleEngine', perhaps misspelled or defined by a module not included in the server configuration

The command line was:
/usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -DSSL

'Solution:' The module mod_security is not enabled. Check for the module with the command 'httpd2 -M'. If the module is really not there, add the module in /etc/sysconfig/apache2.

reverseproxy:/var/log/apache2 # /etc/init.d/apache2 restart
[Mon Jan 31 14:29:23 2011] [warn] worker http://10.10.12.20/start already used by another worker
[Mon Jan 31 14:29:23 2011] [warn] worker http://10.10.12.20/start already used by another worker
Syntax error on line 191 of /etc/apache2/conf.d/modsecurity/modsecurity_crs_10_config.conf:
ModSecurity: Failed to open the audit log file: /srv/www/logs/modsec_audit.log

'Solution:' The directory specified for the logs does not exist. Create the directory with this command:

• reverseproxy:/var/log/apache2 # mkdir -p /srv/www/logs/

or change the location to /var/log/apache2. Of course, the same message can be displayed for /srv/www/logs/modsec_debug.log.

You can test if mod security is running correctly by going to the index file of your website by ip-address and adding '?file=/etc/passwd' to the url:

https://10.10.10.20/start/index.html?file=/etc/passwd

This will be noticed, and displayed in the log (not stopped, remember, we're running in DetectionOnly mode):

• less modsec_debug.log

[31/Jan/2011:15:46:31 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98feb488][/start/0100_NavigationPublic.html][2] Warning. Pattern match "^[\d\.]+$" at REQUEST_HEADERS:Host. [
file "/etc/apache2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf"] [line "60"] [id "960017"] [msg "Host header is a numeric IP address"] [severity "CRITICAL"] [ta
g "PROTOCOL_VIOLATION/IP_HOST"]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match "^[\d\.]+$" at REQUEST_HEADERS:Host. [file "/etc/apach
e2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf"] [line "60"] [id "960017"] [msg "Host header is a numeric IP address"] [severity "CRITICAL"] [tag "PROTOCOL_VIOL
ATION/IP_HOST"]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match "(?:\b(?:\.(?:ht(?:access|passwd|group)|www_?acl)|glob
al\.asa|httpd\.conf|boot\.ini)\b|\/etc\/)" at ARGS:file. [file "/etc/apache2/conf.d/modsecurity/modsecurity_crs_40_generic_attacks.conf"] [line "114"] [id "950005"] [msg "Remote
File Access Attempt"] [data "/etc/"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"]
[31/Jan/2011:15:46:42 +0100] [10.10.10.20/sid#7f0c98cffdc8][rid#7f0c98fe2908][/start/index.html][2] Warning. Pattern match "(?:\b(?:(?:n(?:et(?:\b\W+?\blocalgroup|\.exe)|(?:map|
c)\.exe)|t(?:racer(?:oute|t)|elnet\.exe|clsh8?|ftp)|(?:w(?:guest|sh)|rcmd|ftp)\.exe|echo\b\W*?\by+)\b|c(?:md(?:(?:32)?\.exe\b|\b\W*?\/c)|d(?:\b\W*?[\\/]|\W*?\.\.)|hmod.{0,40}?\+.
{0,3}x))|[\;\|\`]\W*? ..." at ARGS:file. [file "/etc/apache2/conf.d/modsecurity/modsecurity_crs_40_generic_attacks.conf"] [line "133"] [id "950006"] [msg "System Command Injectio
n"] [data "/passwd"] [severity "CRITICAL"] [tag "WEB_ATTACK/COMMAND_INJECTION"]

• less modsec_audit.log:

Message: Warning. Pattern match "^[\d\.]+$" at REQUEST_HEADERS:Host. [file "/etc/apache2/conf.d/modsecurity/modsecurity_crs_21_protocol_anomalies.conf"] [line "60"] [id "960017"]
 [msg "Host header is a numeric IP address"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/IP_HOST"]
Apache-Handler: proxy-server
Stopwatch: 1296487473036980 19376 (997 2882 -)
Producer: ModSecurity for Apache/2.5.6 (http://www.modsecurity.org/); core ruleset/1.6.1.
Server: Apache/2.2.10 (Linux/SUSE)

http://www.modsecurity.org/
http://www.modsecurity.org/documentation/faq.html
http://www.modsecurity.org/documentation/modsecurity-apache/2.5.12/html-multipage/introduction.html
Install Modsecurity
Install core rule set

To prevent information leaks set the following options as described:

• Add ServerSignature Off
  • The ServerSignature directive allows the configuration of a trailing footer line under server-generated documents
• Add ServerTokens ProductOnly
  • This directive controls whether Server response header field which is sent back to clients includes a description of the generic OS-type of the server as well as information about compiled-in modules

These options are set in this file: /etc/sysconfig/apache2:

APACHE_SERVERSIGNATURE="off"
APACHE_SERVERTOKENS="ProductOnly"

After restarting Apache you can check these settings in

reverseproxy:/etc/apache2 # cat /etc/apache2/sysconfig.d/global.conf
#
# Files in this directory are created at apache start time by /usr/sbin/rcapache2
# Do not edit them!
#

# see /etc/sysconfig/apache2

ServerSignature off
UseCanonicalName off
ServerTokens ProductOnly
LogLevel warn
CustomLog /var/log/apache2/access_log combined

Apache can also be extra secured by setting restrictions on which files and directories may be served by the webserver. This is controlled by the 'Directory' directive, which can be used in several configuration files. By default, apache on SLES opens up several directories which are not necessary for our environment. This is the default directory include hierarchy:

httpd.conf  . . . . . . . . . . . . . . . .  denies access for the entire filesystem
  |
  |-- errors.conf . . . . . . . . . . . . .  includes the directory where Apache customized errors are located
  |
  `-- default-server.conf . . . . . . . . .  sets the document root, icons and cgi directory
        `--mod_userdir.conf . . . . . . . . opens user home directories

# forbid access to the entire filesystem by default
<Directory />
    Options None
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

The httpd.conf is all right like this. Access to the entire filesystem is denied by default and must be explicitly overridden by for example vhosts.

Alias /error/ "/usr/share/apache2/error/"

    <Directory "/usr/share/apache2/error">
        AllowOverride None
        Options IncludesNoExec
        AddOutputFilter Includes html
        AddHandler type-map var
        Order allow,deny
        Allow from all
        LanguagePriority en cs de es fr it ja ko nl pl pt-br ro sv tr
        ForceLanguagePriority Prefer Fallback
    </Directory>

Since we'll be using our own errors pages there is no use anymore for keeping this directory. It should be commented out, and new error pages should be defined.

Alias /local/error/ "/srv/www/htdocs/local/error/"
<IfModule mod_negotiation.c>
<IfModule mod_include.c>
    ErrorDocument 400 /local/error/index.html
    ErrorDocument 401 /local/error/index.html
    ErrorDocument 403 /local/error/index.html
    ErrorDocument 404 /local/error/index.html
    ErrorDocument 405 /local/error/index.html
    ErrorDocument 408 /local/error/index.html
    ErrorDocument 410 /local/error/index.html
    ErrorDocument 411 /local/error/index.html
    ErrorDocument 412 /local/error/index.html
    ErrorDocument 413 /local/error/index.html
    ErrorDocument 414 /local/error/index.html
    ErrorDocument 415 /local/error/index.html
    ErrorDocument 500 /local/error/index.html
    ErrorDocument 501 /local/error/index.html
    ErrorDocument 502 /local/error/index.html
    ErrorDocument 503 /local/error/index.html
    ErrorDocument 506 /local/error/index.html
</IfModule>
</IfModule>

<Directory "/srv/www/htdocs">
    Options None
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

This allows files to be served from the document root but nothing more. This is sufficient for our setup, since it allows the local error, outage and maintenance pages to be loaded.

Alias /icons/ "/usr/share/apache2/icons/"

<Directory "/usr/share/apache2/icons">
 Options Indexes MultiViews
 AllowOverride None
 Order allow,deny
 Allow from all
</Directory>

We will not use the Apache icons, they can be commented out.

ScriptAlias /cgi-bin/ "/srv/www/cgi-bin/"

<Directory "/srv/www/cgi-bin">
 AllowOverride None
 Options +ExecCGI -Includes
 Order allow,deny
 Allow from all
</Directory>

This allows cgi-scripts to be executed, but it doesn't allow includes. As far as cgi-scripts are concerned, this is a pretty safe setup, and can be left like this.

        <Directory /home/*/public_html>

                AllowOverride FileInfo AuthConfig Limit Indexes
                Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec

                <Limit GET POST OPTIONS PROPFIND>
                        Order allow,deny
                        Allow from all
                </Limit>

                <LimitExcept GET POST OPTIONS PROPFIND>
                        Order deny,allow
                        Deny from all
                </LimitExcept>

        </Directory>

According to the Apache documentation about mod_userdir the module is only used for serving pages from the home directory of users. This is a potential security risk and we will not only disable the module, we will also comment this out from the config file in case someone ever enabled the directory again.

Another way of making Apache more secure is by setting limits on the amount of processes Apache is allowed to process.
These settings consist of:

• Maximums for the module prefork
• Maximums for the module worker
• Timeout
• KeepAlive settings

On SLES these settings are already implemented in the server-tuning.conf:

/etc/apache2> cat server-tuning.conf | grep -v '^$' | grep -v '^#'
<IfModule prefork.c>
        # number of server processes to start
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
        StartServers         5
        # minimum number of server processes which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/prefork.html#minspareservers
        MinSpareServers      5
        # maximum number of server processes which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/prefork.html#maxspareservers
        MaxSpareServers     10
        # highest possible MaxClients setting for the lifetime of the Apache process.
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
        ServerLimit        150
        # maximum number of server processes allowed to start
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
        MaxClients         150
        # maximum number of requests a server process serves
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
        MaxRequestsPerChild  10000
</IfModule>
<IfModule worker.c>
        # initial number of server processes to start
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
        StartServers         3
        # minimum number of worker threads which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads
        MinSpareThreads     25
        # maximum number of worker threads which are kept spare
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads
        MaxSpareThreads     75
        # upper limit on the configurable number of threads per child process
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit
        ThreadLimit         64
        # maximum number of simultaneous client connections
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
        MaxClients         150
        # number of worker threads created by each child process
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild
        ThreadsPerChild     25
        # maximum number of requests a server process serves
        # http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
        MaxRequestsPerChild  10000
</IfModule>
Timeout 30
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

I also added awstats to the server, a statistics tool. I explained how on this page.