Posted on

Configuring Apache 2.4 Connections For WordPress Sites

Recently I upgraded my web server to PHP 5.6.14. Along the way the process managed to obliterate my Apache web server configuration files. Luckily it saves them during the upgrade process, but one thing I forgot to restore was the settings that help Apache manage memory. Friday night around midnight, because this stuff ALWAYS happens when you’re asleep… the server crashed. And since it was out of memory with a bazillion people trying to surf the site; every time I restarted the server I could not log in fast enough to get a connection and fix the problem.

Eventually I had to disconnect my AWS public IP address, connect to a private address with SSH, and build the proper Apache configuration file to ensure Apache didn’t go rogue and try to take over the Internet from my little AWS web server.

Here are my cheat-sheet notes about configuring Apache 2.4 so that it starts asking site visitors to “hold on a second” when memory starts getting low. That is much nicer than grabbing more memory than it should and just crashing EVERYTHING.

My Configuration File

I put this new configuration file in the /etc/httpd/conf.d directory and named it mpm_prefork.conf. That should help prevent it from going away on a future Apache upgrade. This configuration is for an m3.large server running with 7.4GB of RAM with a typical WordPress 4.4 install with WooCommerce and other plugins installed.

# prefork MPM for Apache 2.4
#
# use httpd -V to determine which MPM module is in use.
#
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxRequestWorkers for the lifetime of the server
#
# MaxRequestWorkers: maximum number of server processes allowed to start
#
#
# TOTAL SYTEM RAM: free -m (first column) = 7400 MB
# USED SYSTEM RAM: free -m (second column) = 2300 MB
#
# AVG APACHE RAM LOAD: htop (filter httpd, average RES column = loaded in physical RAM) = 87MB
# TOTAL APACHE RAM LOAD: (htop sum RES column) 1900 MB
#
# BASE SYSTEM RAM LOAD: USED SYSTEM RAM - TOTAL APACHE RAM LOAD = 2300 - 1900 = 400MB
#
# AVAILABLE FOR APACHE: TOTAL SYSTEM RAM - BASE SYSTEM RAM LOAD = 7400 - 400 = 7000MB
#
# ServerLimit = sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process
# MaxRequestWorkers = number of simultaneous child processes to serve requests , must increase ServerLimit
#
# If both ServerLimit and MaxRequestWorkers are set to values higher than the system can handle,
# Apache httpd may not start or the system may become unstable.
#
# MaxConnectionsPerChild = how many requests are served before the child process dies and is restarted
# find your average requests served per day and divide by average servers run per day
# a good starting default for most servers is 1000 requests
#
# ServerLimit = AVAILABLE FOR APACHE / AVG APACHE RAM LOAD = 7000MB / 87MB = 80
#
#

ServerLimit 64
MaxRequestWorkers 64
MaxConnectionsPerChild 2400

The Directives

With Apache 2.4 you only need to adjust 3 directives. ServerLimit, MaxRequestWorkers (renamed from earlier versions) , and MaxConnectionsPerChild (also renamed).

ServerLimit / MaxRequestWorkers

ServerLimit sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. MaxRequestWorkers is the number of simultaneous child processes to serve requests. This seems a bit redundant, but it is an effect of using the prefork MPM module which is a threadless design. That means it runs a bit faster but eats up a bit more memory. It is the default mode for Apache running on Amazon Linux. I prefer it as I like stability over performance and some older web technologies don’t play well with multi-threaded design. If I was going to go with a more stable multi-thread environment I’d switch to nginx. For this setup setting ServerLimit and MaxRequestWorkers to the same value is fine. This says “don’t ever run more than this many web servers at one time”.

In essence this is the total simultaneous web connections you can serve at one time. What does that mean? With the older HTTP and HTTPS protocol that means every element of your page that comes from your server is a connection. The page text, any images, scripts, and CSS files are all a separate request. Luckily most of this comes out of the server quickly so a page with 20 web objects on it will use up 20 of your 64 connections but will spit them out in less than 2 seconds leaving those connections ready for the next site visitor while the first guy (or gal) reads your content. With newer HTTP/2 (and SPDY) connections a single process (worker) may handle multiple content requests from the same user so you may well end up using 1 or 2 connections even with a page with multiple objects loading. While that is an over-simplification, the general premise shows why you should update your site to https and get on services that support HTTP/2.

Calculating A Value

# TOTAL SYTEM RAM: free -m (first column) = 7400 MB
# USED SYSTEM RAM: free -m (second column) = 2300 MB
# TOTAL APACHE RAM LOAD: (htop sum RES column) 1900 MB
# AVG APACHE RAM LOAD: htop (filter httpd, average RES column = loaded in physical RAM) = 87MB
# BASE SYSTEM RAM LOAD: USED SYSTEM RAM - TOTAL APACHE RAM LOAD = 2300 - 1900 = 400MB
# AVAILABLE FOR APACHE: TOTAL SYSTEM RAM - BASE SYSTEM RAM LOAD = 7400 - 400 = 7000MB
# ServerLimit = AVAILABLE FOR APACHE / AVG APACHE RAM LOAD = 7000MB / 87MB = 80

There you go, easy, right? Figuring our RAM resources can be complicated, but to simplify the process start with the built-in Linux free command and I suggest installing htop which provides a simpler interface to see what is running on your server. You will want to do this on your live server under normal load if possible.

Using free -m from the Linux command line will tell you the general high-level overview of your server’s memory status. You want to know how much is installed and how much is in use. In my case I have 7400MB of RAM and 2300MB was in use.

Next you want to figure out how much is in use by Apache and how much an average web connection is using per request. Use htop, filter to show only the httpd processes, and do math. My server was using 1900MB for the httpd processes. The average RAM per process was 87MB.

You can now figure out how much RAM is used by “non-web stuff” on your server. Of the 2300MB of used RAM, Apache was using up 1900MB. That means my server uses about 400MB for general system overhead and various background processes like my system-level backup service. That means on a “clean start” my server should show about 7000MB available for web work. I can verify that by stopping Apache and running free -m after the system “rests” for a few minutes to clear caches and other stuff.

Since I will have 7000MB available for web stuff I can determine that my current WordPress configuration, PHP setup, and other variables will come out to about 87MB being used for each web session. That means I can fit about 80 web processes into memory at one time before all hell breaks loose.

Since I don’t like to exhaust memory and I’m a big fan of the 80/20 rule, I set my maximum web processed to 64. 7000MB / 87MB = 80 * .8 = 64.

That is where you want to set your ServerLimit and MaxRequestWorkers.

MaxConnectionsPerChild

This determines how long those workers are going to “live” before they die off. Any worker will accept a request to send something out to your site visitor. When it is done it doesn’t go away. Instead is tells Apache “hey, I’m ready for more work”. However every-so-often one of the things that is requested breaks. A bad script in PHP may be leaking memory, for example. As a safety valve Apache provides the MaxConnectionsPerChild directive. This tells Apache that after this child has served this many objects to die. Apache will start a new process to replace it. This ensures and memory “cruft” that is built up is cleared out should something go wrong.

Set this number too low and you server spends valuable time killing and creating Apache processes. You don’t want that. Set it too high and you run the risk of “memory cruft” building up and eventually having Apache kill your server with out-of-memory issues. Most system admins try to set this to a value that has it reset once every 24 hours. This is hard to calculate unless you know your average objects requested every day, how many processes served those objects, and other factors like HTTP versus HTTP2 can come into play. Not too mention fluctuations like weekend versus weekday load. Most system admins target 1000 requests. For my server load I am guessing 2400 requests is a good value, especially since I’ve left some extra room for memory “cruft”.

Posted on

Configuring Apache Connections

Apache Banner

In preparation for WordCamp Charleston I updated my server to add more RAM.  The upgrade was the perfect opportunity to check my Apache connections configuration.  Here is the background on my calculations and how I configure my Apache 2.2 server on CentOS for both a 7GB and a 14GB dedicated server.

Apache 2.2 Configuration Directives

from: http://httpd.apache.org/docs/2.2/mod/prefork.html

The StartServers, MinSpareServers, MaxSpareServers, and MaxClients regulate how the parent process creates children to serve requests. In general, Apache is very self-regulating, so most sites do not need to adjust these directives from their default values. Sites which need to serve more than 256 simultaneous requests may need to increase MaxClients, while sites with limited memory may need to decrease MaxClients to keep the server from thrashing (swapping memory to disk and back). More information about tuning process creation is provided in the performance hints documentation.

While the parent process is usually started as root under Unix in order to bind to port 80, the child processes are launched by Apache as a less-privileged user. The User and Groupdirectives are used to set the privileges of the Apache child processes. The child processes must be able to read all the content that will be served, but should have as few privileges beyond that as possible.

MaxRequestsPerChild controls how frequently the server recycles processes by killing old ones and launching new ones.

Apache 2.2. Prefork Settings Summary

StartServers – how many connectors to start with (waiting for a HTTP request)

MinSpareServers – how many IDLE connectors to keep online ALWAYS

MaxSpareServers – how many IDLE connectors to keep online at one time

MaxClients – how many active connectors to spin up, total

ServerLimit – absolute maximum for MaxClients with runtime config tools

MaxRequestsPerChild – how many times to allow a connector to serve requests before dying to free resources

Determining Your Server RAM

 

  • TOTAL RAM
    Total RAM available to the OS.

 

    1. free -m
    2. TOTAL_RAM = “total” (first column) : 14,254 MB

 

  • USED RAM
    Total RAM in use for all applications.

 

      1. free -mf
      2. all running processes (second column): 1,717 MB
    1. APACHE RAM TOTAL / AVG
      The RAM used by Apache.
      Since Apache uses more RAM “under load” it is good to get this average both after startup and during/shortly-after an hour with peak user connections.

      1. ps aux | grep ‘httpd’ | awk ‘{print $6}’  // gets RSS Memory

 

  • APACHE TOTAL RAM

 

      1. sum 3a : 1,715MB

 

  • APACHE AVG RAM

 

      1. average 3a: 8GMB

Calculating MaxClients

The maximum number of simultaneous requests that can be served.

Max Clients = floor((TOTAL_RAM –  USED_RAM + APACHE TOTAL RAM) / AVG_RAM_LOAD) – 1

Let’s break that down:

  • floor( … blah … ) – 1
    the Be Conservative SectionThis rounds DOWN always and takes away 1 connection for a safety buffer.    This is a semi-conservative approach to avoid maxing out the memory resources and causing connection issues.
  • (TOTAL_RAM – USED_RAM + APACHE_TOTAL_RAM)
    the RAM For Apache SectionThe first part inside the floor function call determines how much RAM is available to Apache.  It takes the total RAM available to the system, takes away the amount of total RAM in use and adds back any RAM already in use by Apache.
  • (… blah …) / AVG_RAM_LOAD
    The Precise Possible Connections SectionThis calculates the exact number of possible connections that can be fit into the RAM available for Apache based on your average per-connection RAM load.

You may need to adjust the USED_RAM to accommodate more memory use of things like MySQL when the system is under load.  I find it best to run these calculations on a system that is running under load, adjust the Apache configuration and re-run numbers during peak load after each adjustment.

Operating system updates, web application updates including WordPress core, and other factors will change this number over time.  Re-run this calculation and update your configuration on a regular basis.

My Apache Server Calculations
My Apache Server Calculations

My 7GB Server Calculation

TOTAL RAM: 6853MB

USED RAM: 894MB

APACHE TOTAL RAM: 572MB

APACHE AVG RAM: 72MB

Max Clients = floor( (6853 – 894 + 572) /  72) = floor(6531 / 72) – 1 = 89

<IfModule prefork.c>
StartServers       20
MinSpareServers    15
MaxSpareServers   30
ServerLimit      61
MaxClients       60
MaxRequestsPerChild  300
</IfModule>

My 14GB Server Configuration

TOTAL RAM: 13,920MB

USED RAM: 3,495MB

APACHE TOTAL RAM: 3,044MB

APACHE AVG RAM: 105MB

Max Clients = floor ( (13920 – 3495 + 3044) / 105 ) -1 = 127

<IfModule prefork.c>
StartServers       20
MinSpareServers    15
MaxSpareServers   30
ServerLimit      128
MaxClients       127
MaxRequestsPerChild  300
</IfModule>

 

Posted on

WordPress Password Protected Content Not Working

Jetpack

One of my sites had problems displaying password protected content this morning.    The site worked fine, but after supplying the password on the protected content the content area was blank.   It turns out JetPack plus some missing PHP modules on my Linux server were the culprit.

How did I find the issue?

WordPress debugging.    This is usually my first stop when trying to isolate a WordPress problem.    Go to the root install of your WordPress site and edit the wp-config.php file.   Look for the line:

define('WP_DEBUG', false);

Change it:

define('WP_DEBUG',true);

 

While you are in there turn this on as well.   This will allow you to quickly turn this feature on-and-off and then investigate the debug.log file in the wp-content directory.  You don’t want to leave debug mode on with a site that visitors are accessing.

define('WP_DEBUG_LOG',true);

When I make these edits I prefer to copy both lines and set one of them to false while leaving the other true, then comment out the pair of lines I am not using.   That makes it easy for me to go in and turn debugging on/off by swapping the leading comment (#) marker.

Now go to the offending page on  your site.     You should have the errors logging to the ./wp-content/debug.log file which will give you some hints.

JetPack Needs

In order to properly render protected content JetPack uses the XML Dom Document processor.    Make sure you have installed php-xml on your server.

The other package JetPack uses heavily is multibyte string support.   Make sure you have php-mbstring installed.

On Red Hat flavored Linux variants you can get everything in order by using these commands:

yum install php-xml
yum install php-mbstring
service httpd restart

Make sure you execute the restart on the web server so the PHP modules are loaded properly.

 

This may not fix the issue on your server, but the debugging and module searches may help some of you.

Good luck.

 

 

 

 

Posted on

Apache Not Following Symlinks

Posted on

Upgrading Redmine From 8.6 to 9.3

After more than a year of using Redmine to help us manage our projects it was time to upgrade.  Redmine helps us manage our bug lists, wish lists, and to do lists.  It helps us communicate with our clients effectively and efficiently using a web based media in a consistent format that is easy to use for both our developers and our clients.  However, during the past year there have been several changes including the significant upgrades that came out in v9.x some months back.   Our busy schedule kept us from upgrading as each new release came out, and sadly we had fallen far behind.   This past weekend we decided it was time to upgrade.   The notes below record some of the problems we ran into and outlines how we resolved them.  If you are using Redmine for your own projects we hope this guide will help walk you through a major version update of your own.

These are Cyber Sprocket’s notes from our upgrade.  For more information you may want to visit the official documentation site.

Our Environment

The environment we were running before upgrading to Redmine 9.3:

  • Redmine 8.6
  • Apache 2.2.7

Preparation

The first thing we ALWAYS do before upgrading a system is to store a local copy of the database and the source code.  In order to make the archives as small as possible we post a note on the system that Redmine will be offline and at the posted time remove all the session “crud” that has built up.   The process includes a mysql data dump, a file removal, and a tarzip.

  • Go to the directory ABOVE the redmine root directory:
    cd /<redmine-root-dir>; cd ..;
  • Dump MySQL Redmine data:
    mysqldump –user=<your-redmine-db-username> -p <your-redmine-databasename> > redmine_backup.sql
  • Remove the session files:
    rm -rf <redmine-directory>/tmp/sessions/*
  • Tarzip:
    tar cvfz redmine-backup.tgz redmine_backup.sql ./<redmine-directory-name>

Issues

Updating Rails

We realized after some back & forth that our RoR installation needed to be upgraded.  Redmine 9.3 require Ruby 1.8.6 or 1.8.7 (we had 1.8.6 luckily) with Rails 2.3.5 (which we needed to upgrade) and Rack 1.0.1 (which we never touched).

gem install rails -v=2.3.5

Fetching 9.3

We could not perform a simple svn update since we are on an 8.X branch.  A new svn checkout was necessary.  We opted to move our old Redmine install to a different path and do the checkout in our original location:

svn checkout  /redmine

Generation session_store.rb

Later version of Redmine (even 8.X versions beyond 8.6) require a secret key in order for the session system to work.  If you don’t have this you can’t login.  After much trial & error we found that the following command WILL WORK if you have the latest Redmine source (Fetching 9.3) and the latest version of Rails (Updating Rails).   There is not file named config/initializers/session_store.rb in the code repository, it is created by the following rake command:

rake config/initializers/session_store.rb

Updating The Database

The database then needed to be migrated:

rake db:migrate RAILS_ENV=production

Database Upgrade Errors : Migrating Member_Roles and Groups

While performing the database update we immediately ran into a couple of errors about a table already existing. Turns out a simple renaming of the tables fixed the problem, no apparent harm done.

The error message was:

Mysql::Error: Table 'member_roles' already exists:

The fix was as simple as logging into MySQL from the command line and renaming the table:

mysql> rename table member_roles to member_roles_saved
mysql> rename table groups_users to groups_users_saved

Switching from CGI to FCGID

It turns out that RoR does not play well with plain ol’ CGI processing via Apache when running Rails v2.3.5.   We ended up having to upgrade our Apache server to enable mod_fcgid and tweaking our new Redmine install to use that.  We started by following this excellent guide go running Redmine on Apache.  Below are our notes about this process to help save you some time:

  • Do not install fcgi, instead use Apache’s mod_fcgid
  • chmod 755 /var/log/httpd so fgcid can run from Apache and access the socks directory it creates there
  • Modify <redmine-directory>/public/.htaccess to prevent looping with mod_rewrite

Installing FCGID

“Official Apache mod_fcgid”:http://httpd.apache.org/mod_fcgid/ this is the Apache version, seems newer and we had more luck with this than the Coremail hosted version below.

Fetch the code

cd /usr/local/src/
wget 
tar zxvf mod_fcgid.2.3.5.tgz
cd mod_fcgid.2.3.5

Configure and Install

./configure.apxs
make
make install

Permissions

chmod 755 /var/log/httpd
service httpd restart

Install Ruby Gem fcgi

You will need to tell Ruby to work with fcgi for this to work:

gem install fcgi

Errors Installing fcgi gem

If you see this error:

Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

You probably need the fcgi development kit. Get it from here, build it & install it… THEN do the gem install fcgi again.

http://www.fastcgi.com/drupal/node/5

Prevent Redirects

You may end up with looping with mod_rewrite if you had a CGI version installed first.   We commented out the non-fcgid lines and that kept things running smoothly.

Edit <redmine-directory>/public/.htaccess

Comment all the lines for the Rewrite rules for the dispatcher except the FCGI rule for fcgid

#<IfModule mod_fastcgi.c>
#       RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]
#</IfModule>
#<IfModule mod_fcgid.c>
       RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]
#</IfModule>
#<IfModule mod_cgi.c>
#       RewriteRule ^(.*)$ dispatch.cgi [QSA,L]
#</IfModule>

Getting Errors With FCGID?

This is a very common error.  For some reason Ruby + mod_fcgid do not always play well with each other.  We have two near-identical servers running CentOS 5, Apache 2.2.x, and the same exact versions of Ruby + Rails + gems installed.   Yet on one server Redmine works fine.  On the other we get this:

undefined method `env_table’ for nil:NilClass

The “magic pill” seems to be running Passenger.  While we didn’t believe this at first since we got it to work fine on our development server, it turns out that there are some gremlins buried deep within the bowels of Ruby & mod_fcgid.    These few steps fixed the problem on our production server:

gem install passenger
passenger-install-apache2-module

Edit the httpd.conf file and add these lines (check your paths that Passenger gives you during the install – they may be different on your server):

LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11
PassengerRuby /usr/local/bin/ruby

Restart httpd…

service httpd restart

Test your Redmine install.

Checking Logs

If you have problems check the log files in your Redmine installation directory, such as ./log/production.log. You may also want to check your Apache log files, assuming you’ve set those up. To log Apache messages you need to have an ErrorLog statement in your httpd.conf file that tells Apache where you want your log file written (normally /usr/local/apache/logs/redmine-error.log).

Posted on

Using Subdomains With Localhost

I do a lot of development work locally, running apache2, mysql, postgres, and any number of other things on my personal computer so that I can do my work. This offers me a lot of benefits: it’s faster, it doesn’t rely on an Internet connection, and it allows me to have complete control over my environment. There are some drawbacks to this though. Generally, you end up with many different projects and with each one comes a new directory, so after a while you have dozens of sites that look like http://localhost/somesitehere/.

This by itself can cause some issues. First of all, now none of your files are running directly off of the document root which often causes some issues with badly written software. Secondly, it confuses the hell out of firefox’s password manager because it’s host based. It also looks kind of ugly having to put in all those different directory names. So wouldn’t it be nice if you could just write ?

Well, you can, and it’s not even difficult.

First of all, you’re going to need to make some changes to your apache/httpd/whatever config. You need to explicitly set NameVirtualHost to your loopback address (which is almost guaranteed to be 127.0.0.1). You will also need to then set each VirtualHost listing to this address as well:

NameVirtualHost 127.0.0.1
<VirtualHost 127.0.0.1>
...
</VirtualHost>

Secondly, and unsurprisingly, you need to actually specify the subdomains you’re going to be using. If you’ve ever done this by hand before, you’ll know that this is also done with the VirtualHost tag:

<VirtualHost 127.0.0.1>
     ServerName subdomain.localhost
     DocumentRoot /place/where/files/be/at/
</VirtualHost>

At this point, Apache or whatever webserver you’re using is configured to handle the subdomains. However, your computer itself is not. Sure, it knows that “localhost” maps to 127.0.0.1 but it doesn’t know where ‘subdomain.localhost’ is. You can fix this by editing your hosts file. This can be done via various graphical interfaces on some systems, or it can be found at “/etc/hosts” on most systems. Once you’re in there all you have to do is add:

127.0.0.1          subdomain.localhost

If you’re paying attention, you’ll notice that a very similar line already exists in that file for “localhost”. In fact, you can map whatever you want in this file. Just remember that you’ll need to make an additional entry in both your webserver conf file and your hosts file for each subdomain that you want to use.

Posted on

Creating and Installing SSL Certs via SSH

Certificate Signing Request (CSR)

Apache + Open SSL

Login as root
cd /usr/bin/ (/your path to openssl/)
openssl genrsa -des3 -out <name_of_your_certificate>.key 1024

You will need to enter a passphrase for your key here, and then enter it again in the next step.

openssl req -new -key <name_of_your_certificate>.key -out <name_of_your_certificate>.csr

At this point you’ll have to enter information about the site/owner of the SSL cert. Keep in mind that the common name (CM) is actually the address of the site (without http://, etc), and that each cert is only for one host. IE: yoursite.com is not the same as www.yoursite.com. Some certs can be created with multiple/alternative common names.

You’ll need the contents of the csr file to create the cert on godaddy or whichever cert site you’re using. Either download it or VI, and copy and paste the contents.

Installing SSL Certificate and the Intermediate Certificate

Once you’ve gotten the actual certs from the cert site (CRT files), you’ll need to upload them onto the server. You can put them wherever you want, but try to keep things organized because you’ll need to reference them later.

Next you’ll need to edit your conf files. In Apache 1.x this will be httpd.conf, Apache2 will most likely be ssl.conf (or some variation thereof), if the server is using virtual hosts there will be either separate conf files for each host, or seperate entries for each host within the ssl.conf or httpd.conf.
Either add, or uncomment these lines:

SSLCertificateFile /path/to/your/certificate/file

this is the CRT file provided by the cert site

SSLCertificateKeyFile /path/to/your/key/file

this is the KEY file you created previously

SSLCertificateChainFile /path/to/intermediate/bundle/file

an intermediate CRT file also provided by the cert site, this file is only required by certain cert providers

Server Restart

Next you’ll need to restart the web server, this can be done in numerous ways, but if you need to restart via SSH use the command:

apachectl -k graceful

This will restart the server and also allow any connections currently in place to finish. You may need to perform the restart twice, and enter the passphrase that you created your original key file with.