Setting up supervisord to manage php-fcgi

This morning I started to check my email, then started looking into how to fix a problem I'm having with PHP FCGI becoming braindead (i.e. the process is still alive, but doesn't respond). First I read How to Stop Crashing / Hanging of php-cgi / spawn-fcgi with nginx / lighttpd, which says you should set the PHP_FCGI_MAX_REQUESTS environment variable to a low number (they suggest 1000), so that the PHP processes are recycled regularly.

Only problem is, I'm already doing this, and I'm using 500 for the value of PHP_FCGI_MAX_REQUESTS. So obviously, that isn't a solution to my problem.

So next I tried following the advice here, which was to use supervisord to start php instead of spawn-fcgi. So I looked into supervisord, but looking at the install instructions, it looked like it would only install in a location where you would need root privileges, which I don't have on the web server (since I'm on a shared hosting account).

Doing some googling, I found that actually WebFaction let you use the easy_install method for supervisor, and it will work okay.

So I installed supervisor on the web server, but on my local PC I had to install it as root, since I didn't know how to install it it a non-standard location.

After installing supervisor, I thought I might as well install the latest version of PHP, since it had a lot of bug fixes. It also meant that I could install it in a different location (at the moment it was installed inside the nginx folder, and I wanted to move it to the webapps folder). And installing it to a different location meant that I could keep my old PHP installation running until I had confirmed that the new installation was working okay.

Before installing PHP, I took the ./configure line for my current installation of PHP, and looked up each of the options I was using, to check that I did actually need them. There were quite a few modules being compiled in that I decided I didn't need, so removed those options from the configure command before running it.

But when I ran make, I got an error about undefined reference to `libiconv' when it was trying to compile in the xml-rpc module. So I googled for this, and found this bug report - Bug #31193 iconv.c:254 undefined reference to 'libiconv'. At the bottom was a comment, where someone says

Run configure like this:
[root@david php-5.0.3]# ./configure CFLAGS=-liconv

So I tried that, and it didn't make any difference, I still got the same error. So next I tried the advice given by the reporter of the bug, who says

SOLUTION:

Makefile ends up being line 70

EXTRA_LIBS = -lcrypt ...

if you add -liconv to the EXTRA_LIBS line in Makefile then Make works just fine.

After editing the Makefile, and adding -lcrypt to the EXTRA_LIBS line as directed, PHP would now make okay.

When I tried to make install PHP though, I got a message that I didn't have permission to create the php directory. It seems that for some cheesun, WebFaction won't let you write to your webapps directory. So what I had to do was to use the WebFaction Control Panel (web based, like cpanel) to create a custom app called 'php'. This then created the 'php' folder in webapps, so I could install php into it.

After installing PHP, I had to install the xcache module for the new PHP installation. When xcache was installed, I copied over the php.ini from my old PHP installation, and modified it to reflect the new path of the xcache module.

I made a shell script based on How do I write a bash script to restart a process if it dies? to start supervisord and keep it running, and edited my crontab to run the script @reboot.

I edited my shell script I use to start PHP currently, and commented out the relevant line in there. I modified the supervisord.conf file that I was telling supervisord to load with the -c option, pasting in the PHP config from
I edited the config for one of my sites, so that it would use the new php config from the nginx mailing list post (changing the paths to be correct for me), and then tried starting supervisord via my bash script. Unfortunately it kept asking me repeatedly for a password. So I had to quit my SSH session, and log in again.

After adding the path to where my supervisord installation had been installed (~/bin) to my PATH environment variable:

echo export PATH=$HOME/bin:$PATH >> ~/.bash_profile source ~/.bash_profile

I tried again, but still got the same problem of being endlessly asked for a root password (which obviously I don't have).

Then I realised that in the script on my local PC I had used sudo, since I needed to be the root user on my local PC to start supervisord. I had uploaded the script to the server, but forgotten to remove the sudo. So I exited my SSH session, logged in again, removed the sudo from the 'keep supervisor running' script, and then tried running it, and it worked okay. I checked it had started the PHP processes, and it had.

I restarted Nginx, then checked the site I had set to use the new PHP installation. The page I was trying to check was the xcache info page, but it said that xcache wasn't loaded, so I checked the php.ini file, and found I'd put the wrong path for the xcache module. So I changed that, then needed to restart PHP for the changes to come into effect.

I killed what I guessed was the parent PHP process (and I guess right), which kills its children as well. Then I killed supervisord. Then I started supervisord again using the shell script. This seemed a bit much, so I checked and actually all you need to do is to run superviorctl -c /path/to/supervisor.ini, which loads the supervisor control panel. This then gives you a list of all processes supervisor is managing. You can type help for a list of commands and what they do. The command to restart a process was, funnily enough, restart processname.

After restarting PHP, Xcache and PHP seemed to be working okay on my site using the new PHP installation, so I modified all my other sites to use the new PHP installation as well. After restarting nginx, I checked the sites, and they seemed to be okay.

I started writing this blog post, but when I went back to Ubuntu to copy the address of one of the webpages I used, I noticed that I had been logged out of SSH. I immediately thought that something must have happened and webfaction had killed all my processes. So I logged in again, and found I was correct. I checked my email, but hadn't had anything from webfaction about killing my processes (last time this happened they sent me an email with details of why the processes had been killed).

Since I thought it must have been something do with my new supervisord/php setup, I decided that rather than restart supervisord/php and get all my processes killed again, I'd be better off going back to my previous PHP setup and waiting for the info to come through from Webfaction as to why my processes were killed.

So I changed all the sites back to the older PHP installation, started nginx, mysql, and my old PHP via spawn-fcgi, and checked the sites were running okay.

I wrote most of this blog post, then checked to see if I'd had an email from WebFaction about why the processes were killed. I hadn't had any message, so after checking WebFaction hadn't opened any tickets on my account about the problem, I opened a ticket myself.

I received a reply from WebFaction pretty quickly with the details of why my processes had been killed - the problem was that I had started the PHP processes with supervisord before stopping the old PHP processes that had been started by spawn-fcgi, and so this had taken me over my memory limit. I was glad to find out that this was all the problem was, and that I hadn't done something stupid in setting up supervisord and the new PHP installation to make them eat memory.

So I changed all the site config files back to use the new PHP installation, but when I tried to upload them to the web server, I found that the internet wasn't working again. After trying for quite a while, I gave up and decided I might as well install one of the new hard drives Mauser got for me recently, since there's not a lot I can do with no internet.

After installing the new Hard drive, I had to format it. That meant I couldn't use my PC for the rest of the evening, since it was busy formatting the drive. After Mauser came home, he told me that actually you can just do a quick format on a new drive. Well, while the info was too late for this drive, at least it would save me the time of waiting for the 2 backup drives (for backing up the new drive) to format.

In the evening I watched a double episode of The Office (US) with Mauser, and also 'Night and the city'.

The weather was overcast all day.

Food
Breakfast: Strawberry Jam Toast Sandwich; Cup o' Tea.
Lunch: Mature Cheddar Cheese with Salad Cream Sandwich; Banana; Slice of Angel Layer Cake; Cup o' Tea.
Dinner: Breaded Fish Portion; Ground Black Pepper; Salt; Potatoes; Peas. Pudding was Lemon Meringue. Coffee; Piece of Sainsbury's Truffle Chocolate; Jelly Snake Sweet.