Speeding up phpMyAdmin
Here's a quick tip for speeding up phpMyAdmin when using it on a remote server. A big drain on rendering speed for the app seems to be the sheer number of theme related requests (images and stylesheets) the browser makes on every page load. An easy way around this is to use the Apache module mod_expires to send an expires header with these files, which tells the browser not to bother requesting them again for a set period. This cuts down the total requests by about 90%.
Firstly, make sure mod_expires is enabled:
sudo a2enmod expires sudo apache2ctl restart
then open the phpMyAdmin Apache configuration file (by default in located at /etc/apache2/conf.d/phpmyadmin.conf) in your text editor of choice. You'll see an IfModule block which sets up some php values:
<Directory /usr/share/phpmyadmin>
Options FollowSymLinks
DirectoryIndex index.php
<IfModule mod_php5.c>
AddType application/x-httpd-php .php
php_flag magic_quotes_gpc Off
php_flag track_vars On
php_flag register_globals Off
php_value include_path .
</IfModule>
</Directory>
insert the following mod_expires block below the existing </IfModule>:
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/gif "access plus 7 days"
ExpiresByType image/jpg "access plus 7 days"
ExpiresByType image/png "access plus 7 days"
ExpiresByType text/css "access plus 7 days"
ExpiresByType application/javascript "access plus 7 days"
</IfModule>
which should leave you with this:
<Directory /usr/share/phpmyadmin>
Options FollowSymLinks
DirectoryIndex index.php
<IfModule mod_php5.c>
AddType application/x-httpd-php .php
php_flag magic_quotes_gpc Off
php_flag track_vars On
php_flag register_globals Off
php_value include_path .
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/gif "access plus 7 days"
ExpiresByType image/jpg "access plus 7 days"
ExpiresByType image/png "access plus 7 days"
ExpiresByType text/css "access plus 7 days"
ExpiresByType application/javascript "access plus 7 days"
</IfModule>
</Directory>
Restart Apache (sudo apache2ctl restart) so the changes take effect.
What this does is tell Apache to send an expires header of 7 days into the future for all image, CSS and javascript files within phpMyAdmin. The initial request after the restart will be as before, but after that the browser knows that the files it got back are good for 7 days, so on subsequent requests it will only request the HTML page. You should see a noticeable speed improvement.
Normally when using mod_expires you would use a much longer expire time than 7 days, but this way if a future phpMyAdmin update does change these theme files you're less likely to get caught out.
Comments (0)
Digital Economy Bill
I wouldn't normally do a blog post on a political topic, but the Digital Economy Bill is big news at the moment, and I, like many, am worried about the direction in which it is going, particularly the parts aimed at addressing copyright infringement.
What problem is it trying to solve?
The copyright infringement parts of the bill seem to be written largely to answer the complaints of the music industry, whose arguments just don't add up.
The BPI claim that illegal file sharing cost the music industry £200m in 2009, however this figure seems to be based on the mistaken idea that a download is a lost sale.[1]
The BPI's own figures on UK music sales from last year paint a different picture. The full article is worth a read but here's the choice quote:
"Across the entire year, singles sales increased by 32.7% to a record 152 million, with 98% of those being digital downloads."
So last year, the height of a recession, was a record year for music single sales. Another analysis of the same figures, which merges single and album sales shows a steady increase in overall sales since 2003, which was the year that digital music seemed to take off.
So where is this £200m hole created by illegal file sharing?
It seems obvious to me that if 10 years ago the majority of single sales were CD singles costing ~£3.99, but now the majority of single sales are digital downloads costing ~99p; that this is going to result in a revenue loss. But this isn't something I've ever seen pointed out in news stories on this topic. This also has nothing to do with file sharing.
At the same time the industry seems to be ignoring some potentially positive aspects of illegal file sharing.
In the BBC's Panorama on this topic (15th March 2010), UK singer Kate Nash attributed file sharing to launching her career. Her view is that after starting on MySpace, people downloaded her songs via. file sharing networks and came to her gigs as a result. She has since had three successful albums and three top 40 hits in the UK.
The overall increase in music 'consumption' is also likely to be why 2009 was the first year in which income from live music (festivals, concerts etc.) overtook income from recorded music.
As musician Billy Bragg put it in the Panorama programme mentioned above - "the Music Industry is thriving, it's the Record Industry that is dying on its feet".
Protecting our creative industries
Apparently the measures put forward to combat file sharing are required to protect our creative industries. However none of the aspects of the bill provide any extra money to the content creators, or pave the way for innovation that might do this. In fact the opposite is true - the music industry will likely bear the majority of the cost of identifying illegal downloaders, and the administrative costs of dealing with complaints will be born by the ISPs. The latter will inevitably be passed on to the consumer. How does this benefit anyone?
Even some Government estimates apparently concede that these measures might cost £500m to implement. And this is to try and address the music industry's dubious claim of £200m in lost revenue?
An exercise in futility
If these measures had any hope of working, perhaps opinion might be different. But the measures in the bill are so trivially easy to circumvent that it seems unlikely they will result in any reduction file sharing.
As long as there is a demand for the content, technology will exist to meet that demand. And the technology will always be 10 steps ahead of any methods to try and combat it.
In the Panorama programme I mentioned above, one of the musicians interviewed likened the plans to cut off persistent downloaders to "going round and confiscating people's record players, and then complaining that no-one's buying records".
Democracy at work
I've watched a few of the House of Lords' sessions debating this bill via. the iPlayer and the BBC's Democracy Live website. Not being one to generally show much interest in politics, it was interesting for me to see how the process works and the sort of things being debated. At the same time it was scary to see how few people in the debates seemed to have a good understanding of the issues being discussed, and how these stalwart few are the sole reason the bill doesn't just get approved without question or modification.
Even more worrying is the way in which this bill is being pushed through parliament, a concern that was raised in the House of Lords debates.
Here's an example. The item in the bill that caused the most controversy was "clause 17", which would have given the Secretary of State sweeping powers to amend copyright law at will, without parliamentry oversight. Apparently this was to allow laws to quickly be introduced to combat future forms of piracy. However no justification was provided to suggest why existing laws might not be sufficient (after all, if it's illegal it's illegal, future technology is not going to change that).
The Lib Dem's put forward a "clause 18" to replace clause 17. This was the 'web blocking' clause which caused an equal amount of controversy. It would have allowed court injunctions to be issued forcing ISPs to block access to websites that enable access to copyright material. Ignoring the futility of such an action, filtering the Internet is an extremely dangerous road for the Government to go down. This is exactly the sort of thing that recently earnt Australia a place on the international "Internet Enemies" list due to their own filtering plans; a list currently occupied by countries such as Iran and North Korea.
This clause was also eventually withdrawn. Hooray for common sense.
However what seems to happen now is that the Government will replace these two clauses with 'something else'. This 'something else' will not be discussed in the House of Lords, cannot be amended, and will be first seen in something called the "wash up", which is where Parliament tries to rush through lots of last minute bills before it is disbanded in preparation for the upcoming General Election.
Even members of the House of Lords seem to have concerns over this. Here's a quote from the Earl of Erroll (one of the few people talking sense), from the final debate on the DEB where clause 18 was being discussed:
"I'm delighted that the noble minister realises the shortcomings of this clause and also that this clause has replaced the previous clause 17 which I didn't like either, and therefore something better will appear. On the other hand, I think that the method by which it's appearing is by complete and absolute abuse of parliamentry process, and I'm not quite sure why we bothered to sit and debate any laws at all if basically the front benchers in another place can get together and put whatever they like into legislation."
Digital Economy Bill, House of Lords, 15/3/10, approx 1:24 in.
My fundamental problem with the Digital Economy Bill is that I believe it will do nothing to stimulate the digital economy, and will in fact have the opposite effect, giving record companies reason to continue their doomed attempts to try and regain control over the distribution of music.
The Internet is the greatest medium for content distribution ever invented. Remember those music sales figures I quoted from 2009? 98% of single sales last year were digital downloads - this medium didn't even exist 10 years ago.
Spotify has over 2 million users in the UK, however last month Warner Music announced it was going to withdraw its music from streaming services, saying such services were "clearly not positive for the industry" (which I read as meaning they will never get as much revenue from it as they would equivalent music sales).
Surely the Digital Economy Bill should be implementing measures that encourage innovation and new business models, paving the way for the iTunes and Spotifys of the future. Until these services exist, any attempts to combat illegal file sharing will fail, and generations of people will grow up knowing file sharing software as the only way to get music.
1. Whilst there will certainly be some people who will illegally download songs that they would have previously gone out and bought, there will also be a (in my opinion much larger) proportion of people who will download large amounts of music simply because it's free, and readily available. How many people have built up collections of many thousands of mp3s through file sharing? Would these people have instead spend tens of thousands of pounds on buying music? I think not.
Tags: Deploying Zend Framework apps with Capistrano
One of the good things the Ruby community has brought us is Capistrano, a command line tool for automated deployment. Although it was written for Rails apps, it can be used with other languages, as proven by a blog post from a few years ago detailing how to use it to deploy PHP code. I thought I'd update this to show how it can be used to deploy Zend Framework applications.
If you're not familiar with Capistrano, once it's all setup you'll be able to run:
cap production deploy
from your ZF project folder. This will automatically login to your remote server, checkout a copy of your code, run any custom tasks (e.g. setting permissions), and then switch the live site to point at the new release. If anything goes wrong with any of these steps, the deployment will stop, the new checkout will be removed, and your site will remain exactly as it is. This allows you to very easily deploy code updates and new versions with basically no downtime.
I'm going to assume you already have Capistrano installed (if not, there are plenty of guides around for this). You'll also need the capistrano-ext gem for the multistage stuff I'm going to mention later. For reference, at the time of writing I have versions capistrano 2.5.5 and capistrano-ext 1.2.1 of these gems.
Capistrano has a command capify which creates the files it requires, however this doesn't work with ZF applications because the file structure is slightly different. So we're going to create these files manually. If you're using the recommended ZF application structure (i.e. you have an application/configs folder), I've packaged up my examples below into an archive you can just extract into the root folder of your ZF project, but read on anyway so you know what the files do.
The first of these is Capfile, which needs to live in the root directory of your ZF project. This should look something like this:
load 'deploy' if respond_to?(:namespace) # cap2 differentiator load 'application/configs/deploy'
this file simply loads in your main deployment configuration which in the above example will be application/configs/deploy.rb. If you aren't using the standard ZF app structure, adjust this path to point at wherever you want your deployment configuration to live.
The next file is deploy.rb, which is where most of the deployment settings live and is the file referenced above. Create this file in your application/configs folder. Here's an example:
# general settings
default_run_options[:pty] = true
set :use_sudo, false
# source control settings
set :scm, :git
set :deploy_via, :remote_cache
set :repository, "ssh://git@example.com/yourapp.git"
# stages
set :stages, %w(staging production)
set :stage_dir, "application/configs/deploy"
require 'capistrano/ext/multistage'
namespace :deploy do
task :migrate do
# overrides the standard Rails database migrations task
end
task :start, :roles => :app do
end
task :stop, :roles => :app do
end
task :restart, :roles => :app do
# no restart required for Apache/mod_php
end
end
Adjust the source control section to contain details for the repository your application lives in. The example above shows typical settings for a git repo (replace git@example.com/yourapp.git with your clone URL). If you're using Subversion, try something like this:
set :scm, :subversion set :repository, "svn+ssh://example.com/yourapp/trunk" set :scm_username, "your_svn_username" set :scm_password, "your_svn_password" set :deploy_via, :export
The stages section allows you to define a number of different environments for your application. In my example I'm defining two stages - production and staging. You then need a file for each which contains the appropriate login details. :stages_dir defines where these files live, in my example it's application/configs/deploy/. These files need to be named the same as your stages (with a .rb extension), e.g. production.rb. Here's an example:
role :app, "example.com" role :web, "example.com" role :db, "example.com", :primary => true set :deploy_to, "/path/to/your/app/" set :user, "your_ssh_user" set :password, "your_ssh_password"
Set :app to be the hostname of your remote server. :web and :db will usually be the same, unless your have separate dedicated servers for each of these and need to run any custom tasks on them later. :deploy_to should be the full path to the location on your remote server where you want your application to live. Change :user and :password to be the SSH login details for your remote server.
To add a new stage, simply add it to the :stages variable in deploy.rb (%w(staging production) is the Ruby equivalent of array('staging', 'production')), and then create a file in application/configs/deploy/ with the appropriate login details.
The deploy.rb example above also contains a section that overrides the default Capistrano tasks :migrate, :start, :stop and :restart. These are pretty Rails-specific, so we override them because we don't need them. However if you have any custom restart requirements you'd define them here. E.g. if you're using APC with stat set to 0, you might want to add some code to the :restart task to restart Apache (this will run after each deployment).
Assuming you've got all that setup correctly, it's time to give it a try. (Remember these commands will be performing actions on your remote server!) First run:
cap production deploy:setup
this will create default Capistrano folders on your remote server. After running it, login to your remote server and in your :deploy_to location you should see two folders: releases, and shared. Then (from your local machine again), run:
cap production deploy:cold
this will run the first full deployment of your application. If you check your remote server again, you'll find a new timestamped folder in the releases folder that contains a full copy of your code. You'll also see a 'current' symlink at the :deploy_to location that symlinks to the latest release. So just set your vhost to point at (your path)current/public and your site should be up and running!
Then in future, to deploy a new update you just have to run:
cap production deploy
this will checkout a new copy of the code, and assuming no problems, switch the 'current' symlink to point at the new release. And that's it! To perform the same thing on your staging setup (or any other stages you add), simple replace 'production' with the name of the stage in the commands above. E.g. cap staging deploy.
Hopefully that should be enough to get you started. Run cap -T from your ZF project folder to see what other commands you can use, and the Capistrano FAQ is a good place to start if you need to customise anything.
Optimizing server configuration for PHP applications
We recently migrated all of our websites to some new servers, and for the first time we have some dedicated servers for our PHP CMS platform. This gave me the opportunity to fine tune configuration specifically for PHP, and I was surprised at how much difference could be made without even touching the application code.
The setup includes two web servers and a dedicated MySQL server, all with 1G of RAM, running CentOS. I was using Apache bench for the tests, with the parameters ab -c 10 -n 500 http://www.example.com/ (10 concurrent requests, 500 in total). The starting point was a standard yum-installed LAMP setup except for the addition of the Remi repositories for more recent PHP packages (PHP 5.2.9, MySQL 5.0.45, Apache 2.2.3).
If you're not familiar with Apache bench, the 'requests per second' stat is the thing to look out for in the results, higher is better. Each test was run after 'warming up' the caches, and there weren't any failed requests.
Firstly the baseline test (fresh after installation):
Requests per second: 13.31 [#/sec] (mean)
Time per request: 751.477 [ms] (mean)
Time per request: 75.148 [ms] (mean, across all concurrent requests)
Transfer rate: 43.27 [Kbytes/sec] received
Then after installing APC:
Requests per second: 18.52 [#/sec] (mean)
Time per request: 539.833 [ms] (mean)
Time per request: 53.983 [ms] (mean, across all concurrent requests)
Transfer rate: 60.24 [Kbytes/sec] received
Then started the tweaking. The Remi php-apc package has apc.shm_size set to 32M by default (slightly higher than the PHP default of 30M). As I understand it, this variable controls how much RAM is reserved by APC. If it is set too low, APC won't be able to cache all of your commonly included PHP files; but as you increase it, once you pass the point at which your whole app can be cached you'll start to see a reduction in performance, since you're reducing the amount of memory that can be used by Apache. After some experimentation I kept this at 32M.
Next I turned off apc.stat. With this option enabled APC checks the last modified time of each included PHP script on each request to see if it needs to be recompiled. With it off you save quite a lot of file system calls, especially if you have a lot of file includes, but it means you need to restart Apache after any code changes. Results:
Requests per second: 18.89 [#/sec] (mean)
Time per request: 529.346 [ms] (mean)
Time per request: 52.935 [ms] (mean, across all concurrent requests)
Transfer rate: 61.43 [Kbytes/sec] received
Not much difference! The application code in this case is stored on a shared NFS drive, so perhaps the local NFS client is caching some of the file information.
Next I enabled apc.include_once_override, which gets around the problem APC used to have caching files included using include_once or require_once:
Requests per second: 23.00 [#/sec] (mean)
Time per request: 434.698 [ms] (mean)
Time per request: 43.470 [ms] (mean, across all concurrent requests)
Transfer rate: 74.81 [Kbytes/sec] received
If your application uses Zend Framework you'll probably see even more benefit here, since ZF uses include_once extensively.
Next came the MySQL tweaking. Although MySQL comes with some suggested configuration files for different setups, the defaults are quite conservative. If you're not familiar with the different MySQL config options, I'd suggest installing MySQLTuner, a free script you can run which will analyse your installation and suggest some changes.
By far the biggest improvements you can make are enabling the query cache and increasing key_buffer_size. If you're running a dedicated server for MySQL you can assign a decent chunk of RAM to these two. The settings I used:
key_buffer_size=128M
query_cache_size=256M
query_cache_limit=16M
thread_cache_size=4
table_cache=128
Results:
Requests per second: 61.60 [#/sec] (mean)
Time per request: 162.339 [ms] (mean)
Time per request: 16.234 [ms] (mean, across all concurrent requests)
Transfer rate: 199.42 [Kbytes/sec] received
Big improvement!
Obviously there are a number of application-level things that can be done to improve performance further (page/fragment caching, caching objects in APC/memcache), but this shows how much improvement you can gain just by tweaking some config files.
Experiences with Zend_Captcha_Image
Yesterday I had to add a CAPTCHA to a form on a site already using some ZF components, which gave me a chance to try out the CAPTCHA component in ZF.
As well as the usual obfuscated image, Zend_Captcha supports figlet (ASCII words) and reCAPTCHA (a 3rd party CAPTCHA service to help transcribing old books). Although I love the idea of reCAPTCHA, I've always thought its standard input box looks a bit like a banner advert (and thus is easy to miss), so I went with the familiar squiggly image (Zend_Captcha_Image).
Most of the standalone CAPTCHA scripts I've used in the past work as follows: You include an image tag in your HTML which points at a PHP script. This script generates an image on the fly (and shows it), as well as storing the generated word in a session. You then include a chunk of code in your form processing script that checks the submitted data against the session word.
The ZF component works slightly differently, in that it generates an actual image file (stored on the file system), and then outputs an image tag that points at this. The generated word is stored in the session (using Zend_Session), and the object has a method for comparing the two when the form is submitted.
Although not a requirement, the component was obviously designed to be used in conjunction with Zend_Form, and it took some digging around in the code to work out how to use it without. Certainly the code example in the docs is incomplete, so hopefully this article will help anyone trying to do the same.
Firstly the absolute minimum amount of code required to generate (and show) a CAPTCHA image:
<?php
require_once('Zend/Captcha/Image.php');
$captcha = new Zend_Captcha_Image(array(
'font' => '/path/to/fonts/Example.ttf'
));
$id = $captcha->generate();
require_once('Zend/View.php');
echo $captcha->render(new Zend_View());
this will create a PNG image in the folder images/captcha (which needs to be writeable). There are garbage collection methods that will automatically remove old images after a configurable amount of time. You will need to replace the font path with the path to an actual TTF font file on your system. You can either download a free font (of which there are many on the Web), or point it at a font already installed on your server (check /usr/share/fonts/truetype/ in Ubuntu).
View this script in a browser and you should see an image like this:
so, now you need to wrap this in a form, provide a box for the user to enter their input, and then check this data after a post. And this is where the documentation starts to let it down, since no example of this is provided.
Checking the POST data is done using the function isValid(), which has two parameters - $value and $context. $context is optional, and even after looking at the code I'm not clear what this is for, so I've ignored it. You would assume $value to be the string the user enters, but it actually needs to be an array with two keys: input and id. $value['input'] should be what the user typed into the form, and $value['id'] should be the MD5 hash returned when the CAPTCHA was generated ($id in my code example above).
If you are using this with Zend_Form, the supplied decorator will output form fields that generate this array for you. But if not, here's a working example:
<?php
require_once('Zend/Captcha/Image.php');
$captcha = new Zend_Captcha_Image(array(
'font' => '/path/to/fonts/Example.ttf'
));
if (!empty($_POST)) {
if ($captcha->isValid($_POST['captcha'])) {
echo 'Passed CAPTCHA check';
}
}
$id = $captcha->generate();
?>
<form method="post" action="">
<?php
require_once('Zend/View.php');
echo $captcha->render(new Zend_View());
?>
<input type="text" name="captcha[input]" />
<input type="hidden" name="captcha[id]" value="<?=$id?>" />
<input type="submit" name="submit" value="Submit" />
</form>
It's important that the isValid() check happens before the generate() (as in the example above), as otherwise you'll be checking against a new CAPTCHA and the check will always fail. Normally you would only run the generate() on non-POST requests, or if the check fails (to give the user a fresh image to try).
A number of other options can be supplied to customise the image, e.g.:
<?php
$captcha = new Zend_Captcha_Image(array(
'wordLen' => 5,
'font' => '/path/to/fonts/Example.ttf',
'imgDir' => './captcha/',
'imgUrl' => '/captcha/',
'width' => 150,
'height' => 55,
'dotNoiseLevel' => 40,
'lineNoiseLevel' => 3
));wordLen controls the number of characters in the image (default is 8). imgDir and imgUrl control where the image files are stored. width and height affect the dimensions of the generated image. dotNoiseLevel and lineNoiseLevel control the number of dots and lines added.