nabeel shahzad

Reducing HTTP Requests

In doing some optimizing for phpVMS, I noticed I have a lot of HTTP requests:

As you can see, there are 27 HTTP requests, 15 of them JavaScript files. This means, the browser has to reach out 15 times to grab individual JS files. This seems like a straight forward problem, if we can just condense those 15 files into one, then we can almost half the number of total requests.

So we can do this manually – copy and paste each file, into one master file. The problem is, when you update one file, you have to go and search around in your other files, and figure out exactly where to update, or just redo the entire condensing process. It’s a pain. but these seems like a good place to write some code to do it.

As part of Codon now, I’ve included a class called CodonCondenser. It’s just a basic class that will condense requested files into one file. I set out some requirements:

1. It’s a class, so we can reuse it
2. Use it for any file types
3. Have a cached file of our condensed batch, so we don’t need to dynamically condense it every time, which could be expense.
4. Be able to add/remove files at will, and it will handle that

Since we have some basic requirements, let’s plan how it’ll work:

1. Set our options, the path to the files, the URL to the files since our function will return the URL to the generated file, set the file type, and a cache
2. Pass an array, with the list of our files
3. Generate a MD5 hash of the array we’ve passed, this will be our file name. If the array changes, as in we add files, or remove files, the hash will change, so we satisfy requirement #4… OR… use a filename that we generate, or pass in.
4. Check this file name, see if it exists; if it does, see if the file age is older than we want (satisfying #3). If it’s not too old, then use this file
5. If it’s too old, or doesn’t exist, then generate a new condensed file and return the URL to it.

So let’s start:

[php]
class CodonCondenser
{
public $path;
public $url;
public $timeout = 24;
public $file_ext = ”;
public $filename;
}
[/php]

Our basic class, with the settings. Next is our function to set these options:

[php]
public function SetOptions($path, $url, $file_ext, $timeout=24)
{
$this->path = $path;
$this->url = $url;
$this->timeout = $timeout;
$this->file_ext = $file_ext;
}
[/php]

We pass the $path, which is the absolute path to the files, and $url, which is the path to where the file is will be publicly accessible.

$file_ext is the extension of the condensed file (js, css, htm, etc), and $timeout is the time that the condensed file is considered ‘fresh’. Passing this as blank ($timeout=’’), will disable the time check.

Next is our function to check the cached version. We’ll return true or false if the file passed is valid:

[php]
protected function getCachedFile()
{
if(!file_exists($this->path.’/’.$this->filename))
{
return false;
}

# Check if the version that exists
# is older than the timeout we have alloted
# Value of "" skips the time check
if($this->timeout == ”)
{
if ((time() – @filemtime($file)) > ($this->timeout*3600))
{
# It is older, so delete it
@unlink($this->path.’/’.$file);
return false;
}
}

# The cache file is ok
return true;
}
[/php]

So what we are checking, if the file doesn’t exist, then return false. Next, if the timeout value is blank, just skip the check, and only base it on whether the condensed file exists or not (as we saw the option up above). Otherwise, we check the modification time of the file, and if the current time – the time file created (in seconds), is greater than our timeout value, then return false. If we have 0 (zero) as a timeout value, it will generate a new file every time.

Note that the function is protected, so we can’t access it directly, and instead, go through the main function:

[php]
public function GetCondensedFile($files, $filename=”)
{
if($filename != ”)
{
$this->filename = $filename;
}
else
{
$this->filename = md5(implode(”,$files));
$this->filename .= ‘.’.$this->file_ext;
}
}
[/php]

First, here we are checking if we passed an optional filename. I use the filename option, depending on whether it’s being used in the admin area, or in the front-end client area. I’ll explain this later.

Otherwise, we will build a filename, we check for the cached file, using the function above:

[php]
# Check if we’ve already made this condensed cache file
# If we have, then just give the URL of that file
if($this->getCachedFile() == true)
{
return $this->url.’/’.$this->filename;
}
[/php]

If the getCachedFile() returns true, that mean’s that the cached file is okay, and we exit and just return the full URL path to the condensed file. If it’s not condensed, then we build our condensed file:

[php]
$fp = fopen($this->path.’/’.$this->filename, ‘w’);

foreach($files as $file)
{
fwrite($fp, file_get_contents($this->path.’/’.$file));
}

fclose($fp);
return $this->url.’/’.$this->filename;
[/php]

This is pretty straightforward, we just open the condensed file, and write every file into it, then return the full URL path to the file. So this is how we’ll use it:

[php]
$condenser->SetOptions(‘/var/www/lib/js’, ‘http://mysite.com/lib/js’, ‘js’);

$files = array( ‘jquery.min.js’, ‘jquery-ui.js’,
‘jquery.dimensions.pack.js’,
‘jquery.form.js’, ‘jqModal.js’,
‘jquery.bigiframe.js’,
‘jquery.sparklines.js’,
‘jquery.autocomplete.js’,
‘jquery.tablesorter.pack.js’,
‘jquery.tablesorter.pager.js’,
‘jquery.metadata.js’, ‘jquery.impromptu.js’,
‘jquery.listen-min.js’, ‘nicEdit.js’);

$url = $condenser->GetCondensedFile($files);
[/php]

And then to link the Javascript into our page:

[javascript]
<script type="text/javascript" src="<?php echo $url?>"></script>
[/javascript]

Our result in the end:

14 less requests! Almost half the number of requests. Woohoo! While it’s a basic class, and we can certainly expand it by adding features such as compression (such as implementing JSMin-PHP into), to minify and pack the JS. Be careful if your JS files are already packed if you re-pack them, it could cause errors. I use files that have already been packed, and merge them together this way.

You can also use this for HTML files, CSS, any other text files which are all brought in together in separate requests.

Click to download the full script.

Airplane Movie Quotes

I just did a quick thing for the phpVMS admin panel, displaying random quotes from the movie Airplane! in the footer. Just call the function randquote(), and it’ll return the string, so you can do whatever you want with it.

And don’t call me Shirly…

Click to download it (zip file)

MySQL “Improved”

Found an old post I wrote, about the new “MySQLi” functions in PHP 5+. PHP6 is almost on the horizon, and I personally haven’t seen too much wide adoption of this new MySQL functionality.

The i in “MySQLi” stands for i”mproved”. While most of the functions between MySQL and MySQLi will work exactly the same (just by replacing the mysql_ with mysqli such as mysqli_query()), there are several cool things that come with this new "improved" MySQL library. One thing that’s available is to use the database connector itself as a "class", making it easy to create your own database library that is customized to your application. This makes it real easy to separate the logic and extend your database code, for those of us who are OO nuts

Another advantage is that you can also use an SSL connection easily with the database, without jumping through hoops by just providing the path to the certificate. But the best improvement is the ability to use prepared statements. This makes queries (more) secure by telling MySQL exactly which types of values to expect, and also stores a template of those values in memory. The result is security, and since it’s cached, it ends up being much faster on repeat queries and inserts.

I will use the procedural version of the code in these examples, to keep it a bit easier, and I’m also leaving out error checking, for the sake of example.

[php]
//First we create our connection
$db_link = mysqli_connect(‘localhost’, ‘username’, ‘password’, ‘db name’);

//Now we want to prepare our statement
// So we first have to initialize it.
//Now we prepare the actual statement. The ? replace the variables
// We’re going to "insert" those later on
mysqli_stmt_prepare($db_link, ‘SELECT * FROM data WHERE stringvalue=? AND numbervalue=?’);
[/php]

So we’re prepared the statement, putting in question marks where there are going to be values placed in. We are “binding” a value into there, so we have to use a function mysqli_stmt_bind_param().  If we look at the parameters for this function, it is:

[php]
mysqli_stmt_bind_param(statement, types, variables);
[/php]

The types, are defined as:

• i – integer (any whole number)
• d – double (number, with decimals)
• s – string
• b – blob

Since we have to variables in the query above (string value and number value), our "types" are going to be "si". The first is a string (s) and the second is a number (i).

[php]
mysql_stmt_bind_param($statement, ‘si’, $string_value, $number_value);
[/php]

You might be wondering “What’s the point?”. It helps a great deal with security, when it comes to inserting data:

[php]
$statement = mysqli_prepare($link, ‘INSERT INTO table (name, idnumber, address) VALUES (?, ?, ?)’);

mysqli_stmt_bind_param($statement, ‘sis’, $name, $idnumber, $address);

$name = ‘First Last’;
$idnumber = 123456789;
$address = ‘I live here’;

mysqli_stmt_execute($statement);
mysqli_stmt_close($statement);
[/php]

As you can see, MySQL is now expecting certain types for the data which is being inserted, and treats it as such – strings as strings, integers as integers, and so on, and will automatically take the best precautions to ensure the data is valid.

For more info, check out the Zend information on MySQLi.

Profiling PHP Code

After or during writing a project, it’s important to visit your code with a profiler – see how long your code takes to run, and identify potential bottlenecks, and places where you can improve your code.

The profiler I use is Xdebug, a slick PHP debugger, and profiling tool. I use XAMPP for my testing (no running services, turn on Apache and MySQL when I need it), on my machine, and it comes with the xdebug extension by default. I leave xdebug enabled, as it gives great debugging information, including a caller/stack trace. You also need a program called WinCacheGrind, which will allow you to view the profile traces.

To enable Xdebug in your XAMPP config, go to: C:\xampp\apache\bin (I installed it in C:\xampp), and open up your php.ini file, and go down to the “extensions”, and uncomment this line:

[code]
;extension=php_xdebug.dll
[/code]

So it reads:

[code]
extension=php_xdebug.dll
[/code]

And then add this at the bottom of your .ini file (Note: your php.ini file may already have this, just needs to be uncommented)

[code]
[XDebug]
;; Only Zend OR (!) XDebug
zend_extension_ts="C:\xampp\php\ext\php_xdebug.dll"
xdebug.remote_enable=true
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000
xdebug.remote_handler=dbgp
xdebug.profiler_enable=1
xdebug.profiler_output_dir="C:\xampp\tmp"
[/code]

This will enable the profiler, as well as the debugging. Restart Apache in the XAMPP control panel, and run a single PHP page (ie, I goto http://localhost/phpvms). Just run it once, and goto your C:\xampp\tmp directory. There will be a file like “cachegrind.out.<number>”. Open this file using WinCacheGrind, and you’ll see something similar to:

I’ve expanded it out, but you can see the time it takes to run each function, load any include()’s, etc. This is a great way to peek into your scripts and see what’s going on in there, and where you can concentrate your optimization efforts. I’ll share some of the things I did to optimize in the future.

Using Random Images using CSS and PHP

I want my page’s background, navigation bar, and and main content to be randomized together on my main phpVMS site. Now, I don’t want to touch any HTML, just do it in CSS. My background is split into three files, spanning 3 divs, even though it’s one image.

My CSS looked like this:

[css]
#body {
background-image: url(images/topbanner.png);
}
[/css]

So the image has to be random, but it has to be matched up. Sounds like a job for PHP. First thing we do, is create a folder in our /images directory:

[code]
/images/bg/1.png, 2.png, etc
[/code]

Next we create our PHP file to pick out a background image for us. I called it “bgimages.php”, and placed it in the root of my site (one directory up from /images). Next we change out background-image in CSS to the PHP file (remembering that paths in CSS are relative to the CSS file; I placed it with my CSS file):

[css]
#body {
background-image: url(bgimages.php);
}
[/css]

First we want to setup the total number of images we have, and then the path to our images:

[php]
$total = 4;
$basepath = ‘images/bg/’;
[/php]

We next randomize:

[php]
$num = rand(1, $total);
[/php]

And then finally show the file:

[php]
readfile($basepath.$num.’.png’);
[/php]

readfile() will just throw the entire file out, without processing, which is what we want to do.

Another thing we can do, instead of using the $total and $basepath, is to use the glob() function:

[php]
$files = glob(‘images/bg/*.png’);
$num = rand(0, count($files));
readfile($files[$num-1]);
[/php]

glob() will do a wildcard match in a directory, and return an array of files which match. The above will match any file ending in the .png. glob() is slower though, and since I keep my images in a separate directory, and don’t change much, I decided to go with giving the total number. The downside is whenever I add an image, I have to change the total number of images, but since that’s not that often, it’s okay. It also saves some load on the server from using glob().

The whole script:

[php]
<?php
header(‘Content-type: image/png’);
$total = 4;
$basepath = ‘images/bg/’;
$num = rand(1, $total);
readfile($basepath.$num.’.png’);
[/php]

Not too shabby for 4 lines.

CSS is great (this time)

I always read about CSS being great because you could just change your site’s design without touching a single HTML file. I didn’t think it could be done, not without touching at least some HTML, but lo-and-behold… I did it (only after 5+ years of doing this…)! I changed my design across 3 separate apps – phpBB, dokuwiki, and my main site by just changing the images, sizes, and positioning in my main styles.css. Now that was cool. And it worked perfectly. I’m glad I properly went through and did the main styles properly. Now only if it would work flawlessly in IE all the time (Achilles Heel)

Obviously the index page changed, since I’m not going to use Moveable Type anymore, even though I really liked the ability to create multiple blogs with one installation. That’s the one thing I miss with WordPress. I still have a few sites running under Moveable Type, so I can’t phase it out just yet… maybe soon. It would be a pain to swap things out. It was nice to create one set of users and give them permissions to one blog or another. Made things easy, since it was all in one place, didn’t have too much to keep track of. It also used one database, and stored the content in flat-files. I’ll have to find a WordPress plugin which does that (which I’m sure there is). I am also going to cross-post phpVMS news here, and have it shown on the phpVMS site based on the category. Hmm…ideas for a new blog post and something to play with there. For now, I’m going to post up some GD code I wrote to create forum signatures, so messing with sizes and manipulating text. I may write that up for next week.

I’m going to be playing with SEO sometime soon, check out some good tricks and ideas to move phpVMS up to the top of search results. It’s really maturing to the point where I want to take over the world. Muahahaha…

WordPress Update Script

I like all the practice I’m getting with command-line scripts. I had to update WordPress today, which meant…

1. Downloading the latest version
2. Extracting the latest version
3. Backing up older version
4. Uploading newer version

/ol>

So there’s a few ways this could be done through a bash script – wget the latest URL, and then un-zipping or un-tar’ing that into the WP directory. This is a little more involving, since the download URL will change based on the version.

This is similar to the upgrade script I have for phpVMS, and also the upgrade script I’m offering to users to work with for upgrading. So it was good timing that it coincided with the WordPress update. Two birds with one stone.

So looking through the WordPress site for a viable upgrade path, I saw they use SVN for their version control – both dev and stable versions. Woohoo! Subversion to the rescue. Since this URL won’t change anytime soon, it’s a reliable source to use for upgrades. They house all of the “final” versions in the /tags directory, one directory per version, which made it even easier. They have a guide on Updating with Subversion, which was helpful on the procedure for the files which need to stay, so I came up with a small script:

[bash]
#!/bin/bash
#
# Nabeel Shahzad
# http://www.nsslive.net
#
# This follows the general flow on the following page:
# http://codex.wordpress.org/Installing/Updating_WordPress_with_Subversion

#
# Variables
#

# Where wordpress lives on our server
WP_DIR="/path/to/wordpress"
# WordPress stores latest versions under tags directory
SVN_ROOT="http://svn.automattic.com/wordpress/tags/"

# Go!
echo "Retreiving latest version"

LASTVERSION=$(svn list $SVN_ROOT | tail -1)
echo "Found $LASTVERSION"

# Clean out our temp directory
rm -drf wp-temp

# Check it out
svn checkout $SVN_ROOT/$LASTVERSION wp-temp
echo "Copying files"

# Copy our files over
cp -p $WP_DIR/wp-config.php $WP_DIR/.htaccess wp-temp
echo "Copying everything into $WP_DIR"

# Now copy everything backwards into where our live WordPress lives
cp -rpf wp-temp/* $WP_DIR
echo "Removing temporary directory"

# Clean up the temp directory and delete it
rm -drf wp-temp
echo "Done!"
[/bash]

What I do is list the versions (svn list) in the /tags directory. The last line is the latest version available (ie, 2.5.0/), so I extract this (using tail –1), and then checkout the latest version into a temporary directory. Then according to the WordPress guide, I copy over some of the essential files (config file, htaccess) into that temporary directory, and then copy all those files into where WordPress lives.

The only problem is… any files which you have modified that are part of the WordPress package will be overwritten and not copied over. You can add those files into the “Copy our files over” section, but it’s meant to be “non-destructive”, so it will only touch the WordPress files. I have it in my public_html root, where there are tons of other files, I don’t have to worry about those getting messed up or overwritten as part of the upgrade.

All that’s left after this is running the updater, which just involves logging into the admin panel and running their upgrade script. 

Cheers!

phpVMS 1.1.400 released, lessons learned

I put out the update for 1.1.400 (version bump, went from 1.0 to 1.1 because of “major” features), then the 400 being the revision number from SVN. So far no problems, my main concern were the tons of database changes, including the removal of foreign keys, which made me a bit nervous, since I guess different versions react differently.

Instead of doing scratch installs, and then doing an update (time consuming), I wrote a quick SQL file, and another bash script which would automatically insert the older 1.0 database (basically running the install.sql), and then run the update.sql, and report any errors which are thrown:

[bash]
#!/bin/bash

# MySQL Installer and Update Check
# Nabeel Shahzad <http://www.nsslive.net>

DBUSER=""
DBPASS=""
DBNAME=""
DBSERVER=""
INSTALLFILE=’install.sql’
UPDATEFILE=’update.sql’

mysql -u uname dbname -e "show tables" | grep -v Tables_in | grep -v "+" | \
gawk ‘{print "drop table " $1 ";"}’ | mysql -u uname dbname

mysql -u $DBUSER -p $DBPASS -h $DBSERVER $DBNAME < $INSTALLFILE
clear

mysql -u $DBUSER -p $DBPASS -h $DBSERVER $DBNAME < $UPDATEFILE
[/bash]

The line to drop all the tables was from this page here (thanks!). My install.sql can also include the SQL inserts for default values (I will post the code for my installer soon, since it’ll just read any .sql file), but what I will do is create a backup of an existing DB with all my data, and run the update against that using this script.

Pretty basic, but really handy to just viewing errors.

Subversion as build tool

I use Subversion to manage all of my projects, since it’s an easy tool to setup, and manage repositories with. I use the SmartSVN client to manage all my check-ins and checkouts. But something I have with phpVMS to to automatically create builds with the proper version numbers. I have an update script, which will make database changes, as well as update the version numbers. I base the version number off of the Subversion revision number, which is incremented every time code is checked in. The steps I take for a build are:

1. Check the current revision number
2. Update the update script to the next revision number
3. Commit the check in
4. Export the SVN code
5. Create a tar file with the full install, with the build number
6. Create another update install, with the local config removed, with the build number
7. Move these files to the download area
8. Move the exported code into my test area to keep it updated.

It’s annoying to do with each check in, and whenever there’s an update, I have to do it again. This is the perfect flow to create a script to handle. So I created a bash script, which can be called through a cron-job, on the command-line, or as a post-commit hook in the repository.  So here’s the script (read through the comments!)

[bash]
#!/bin/bash

# Variables
REPOS="http://svn.devjavu.com/phpvms/trunk"
EXPORT="/path/to/temp/export"
MOVETO="/path/to/test/directory"
DOWNLOADSPATH="/path/to/downloads/directory"

# Revision Number will be tacked on the end
VERSION="1.0."

# Cleanup our export directory
rm -drf $EXPORT

# Export and get the revision number
echo "Exporting from SVN"
REVISION=`svn export –force $REPOS $EXPORT | grep -i -o -e "revision.*[0-9]"`
REVISION=${REVISION:9}

echo "Retrived revision $REVISION, adding to update file"
# Add the version number into the update script
# Replace the <<UPDATE>> with the version + revision number
sed -i "s/<<UPDATE>>/$VERSION$REVISION/g" $EXPORT/install/update.php

# Create the full install
echo "Creating full install file"
rm -f $EXPORT.$REVISION.full.tgz
tar czf "$EXPORT.$REVISION.full.tgz" $EXPORT

# Create the update
# Remove the local.config.php file
echo "Creating the update"
rm -f $EXPORT.$REVISION.update.tgz
rm -f $EXPORT/core/local.config.php
tar czf "$EXPORT.$REVISION.update.tgz" $EXPORT

# Clean up the destination directory
# But I don’t want to since my config file is there
#if [ -d $MOVETO ]; then
# rm -drf $MOVETO
#fi
#mkdir $MOVETO

echo "Moving to test area"

# Now move it to our test area
cp -Rf $EXPORT/* $MOVETO

# Move our tars to our download area
echo "Moving downloads"
mv -f $EXPORT.$REVISION* $DOWNLOADSPATH

echo "Done!"
[/bash]

So I do an export of the SVN cache, and then run “grep” to get the revision number. After that, it’s just basic directory and file manipulation. It’s also helpful if you need to compile some code.

In my update.php, I placed the following:

[php]
define(‘UPDATE_VERSION’, ‘<<UPDATE>>’);
[/php]

Which then gets replaced by the version number with this line:

[bash]
sed -i "s/<<UPDATE>>/$VERSION$REVISION/g" $EXPORT/install/update.php
[/bash]

If you have multiple files, it’s just a matter of copy/pasting that line to include your files with the update. Enjoy!

Keys to an excellent web-app: API

During the development on phpVMS, I found I was splitting my time as 95% programming new features, and 5% debugging. I attribute this to something, which is often over-looked when creating a web-application, an API – "Application Programming Interface". When you hear the word(s), you probably are thinking more towards actual applications, something running on your desktop, or maybe in Windows or Linux. But an API serves well in web applications also. While frameworks, like my own Codon, or Symfony, Zend, etc all have APIs, it serves well to plan and write out your own API for your application.

To start, what’s an API? It’s a common interface to do things. For example, database operations, an API serves well to managing all your queries from a central place. But what’s the point? Look at my new code to bugs ratio. An API speeds up development time, allowing you to concentrate on logic, rather than fumbling with getting the basics working. Frameworks serve the same purpose, but a framework is only as good as the API backing it.

Designing your data model

I first designed my database for phpVMS. I knew I needed a table to contain all of the pilot reports, “pireps” for short. So there are a few steps to designing a good API:

• Adding a PIREP
• Removing a PIREP
• Editing a PIREP
• Changing the status of a PIREP
• Retrieving PIREPs
• Retrieve PIREPs on a certain date
• Retrieve PIREPs for a certain pilot
• etc

Instead of writing them in-line, while I’m writing the front-end code, I like to work backwards, starting with the database functions I would need., working my way to the controller, than then finally the HTML template. This works well combined with a feature of PHP 5 – static classes.

[php]
class PIREPData
{
public static function GetAllPIREPS()
{
$sql = "…";
return db_results($sql);
}
}
[/php]

Now with the above code, in any piece of code, I can do:

[php]
$allpireps = PIREPData::GetAllPIREPS();
foreach($allpireps as $pirep)
{
echo …
}
[/php]

And have all of my PIREPS, and I can cleanly display it. If I need to do it anywhere else, I don’t need to copy/paste SQL code, which I may have to debug or change later on.

Being Consistent

This is the next step to designing a good API. Consistency means:

1. Keep return data uniform
2. Keep variable names consistent
3. Make sure you use it!

Returning uniform datasets:

This seems a little odd, but I’ve seen code where one API function will return HTML, another will return just a dataset. I keep as a rule to only return datasets, and then decide what I want to do with that dataset. For instance, I keep track of “flights” through an interface. I use the same API call to build an HTML table, and also the same API call to then build an XML result-set from that data to use elsewhere.

Keep variable names consistent

This one bugs me when I see it. For instance, I use the variable name $pilotid. I only use $pilotid. Not $p_id, $pid, $pilot_id, or the worst I’ve seen, $p (wth is $p??).

Make sure you use it!

I’ve often seen a good API, but no one bothers using it, instead re-writing the code to either it’s in the format that they (and only they!) want, even though the API function can easily be processed to provide the functionality, or just plain-out re-writing code. Always drill it into your head and programming to look before you code! Spending 15 minutes to analyze an existing data model can save a lot of time and head-ache in the long run.