Tom's Blog » Programming

Unit testing your Javascript: Just Do It

Tom — Tue, 15 Dec 2009 20:26:12 +0000

Nike’s slogan is the only one fit for the issue I’m about to raise in this article: Unit testing. Deep down, everyone knows the benefits from unit testing your code. Unit testing can give you that warm feeling when you go to bed, knowing that the changes you made, didn’t break previously working code. It makes you happy and it gives you confidence.

Yet, a lot of us (including me) don’t actually start unit testing our code. There are a number of reasons for that, but I’ll have none of that now. Because now, I’m going to show you just how easy it is to unit test your Javascript. I’ll be using QUnit, made by John Resig, creator of jQuery. Even though I’ve just begun using QUnit, the code I presented in my previous three blog posts, has really benefitted greatly from it.

Setting up the environment

To set up your basic unit testing environment, you won’t need to go through a lot of trouble. It’s as easy as creating an HTML page which loads the latest jQuery instance, QUnit instance and a stylesheet.



    
        
        Tag library Unit Tests
        
        
        
    
    
        Tag library Unit Tests

At this point, nothing much will be shown when you run the HTML file in your browser, as we still have to add our unit tests.

Environment without tests

Adding Unit Tests

Now that you’ve seen how easy it is to set up your testing environment, it’s time to actually write a test. I will provide you with some samples from what I’ve used to test the Tag Javascript library. If you need an API, or you want to know all the possibilities, just head over to the QUnit website.

To keep my stuff organized, I try to reflect the regular structure as much as possible in my tests. This is what I mean:

File structure

All tests are in separate files, reflecting the original structure. The tests for /tag/tag.pubsub.js are in the file /tests/tag/tag.pubsub.js. I think it’s a good idea to keep your testing in modules. One big file, or even in the environment HTML file would also work. It’s just not that easy to work with if you have a lot of tests.

Since I’m all for the modular way, first thing we add is an indication that a new module is tested

$(function(){
    /**
     * Define the module:
     */
    module("Tag base class");
});

This goes in the file /tests/tag.js (see image above), and will be referenced in the HTML environment we set up as our first step:



    
        
        Tag library Unit Tests
        
        
        
 
        
        
        
    
    
        Tag library Unit Tests

In the environment, we first include the file containing the code to be tested. Next we include the file containing the tests for that code. You can add other files for each plugin you want to test.

Ok, our module is defined. The first test I’m adding is to verify if the file was loaded, and if the Tag namespace is defined in the window scope.

    /**
     * Define the module:
     */
    module("Tag base class");
 
    /**
     * Verify existence of the Tag object:
     */
    test("Tag", function() {
        expect(2); // optional
        equals(typeof window.Tag, "object", "Verifying existence of Tag object");
        equals(typeof window.Tag.constructor, "function", "Verifying constructor type");
    });

The tests to verify the existence of the Tag object are grouped inside a single test() case. Inside the test case, we can have as many assertions as we want. Here I have added 2 assertions: I verify if the type of window.Tag is an object, and if the constructor is a function. If both these assertions succeed, then that’s enough proof for me that the Tag base class is loaded and available on the page.

The basic premise of unit testing is that you test the smallest unit that makes up your code. In the case of the Tag library, the smallest unit will be a method from a plugin or the base class. So the next thing we’re obviously going to do, is test each of the methods inside the Tag object.

As an example, here are the assertions I’ve written to test the extend() functionality of Tag:

    /**
     * Verify working of extend method:
     */
    test("Tag.extend()",function() {
        // an assertion using ok()
        ok(typeof window.Tag.extend === "function","Method exists");
 
        // a dummy temporary plugin (also called a mock)
        var tmp = {
            _initCheck:false,
            init:function() {
                this._initCheck = true;
            }
        };
 
        // Verify reserved namespaces:
        var reserved = ["extFN", "extLocalFN"];
        $.each(reserved, function() {
            ok(!window.Tag.extend(this,tmp), "Namespace '" + this + "' cannot be overwritten");
        });
 
        // add test plugin
        ok(window.Tag.extend("test",tmp), "Namespace 'test' taken");
        ok(!window.Tag.extend("test",tmp), "Namespace 'test' cannot be taken twice");
        equals(typeof window.Tag.test, "object", "Tag.test plugin exists");
        equals(typeof window.Tag.fn.test, "object", "Test plugin added to FN");
 
        // verify if plugin is extended with extra functionality:
        $.each(reserved, function(index, ns) {
            var obj = window.Tag[ns];
 
            $.each(obj,function(key, value){
                same(window.Tag.test[key], obj[key], "Verified existence of '" + key + "'");
            });
        });
 
        // verify if init() was called:
        equals(true, window.Tag.test._initCheck, "Init was run");
    });

The extend() method does a lot of things, naturally, all these things need to be tested. It’s the only way to be sure that the method does what it promises to do. For complex methods, with lots of execution paths, it will become difficult to test everything. That’s a reason to keep your methods small, and work with subroutines: they’re just easier to test (and to understand half a year after you last touched that part of the code).

Unit testing AJAX calls

The Tag.core plugin has functionality to asynchronously load javascript files. The problem with AJAX is that it is asynchronous, while unit testing is synchronous. The tests will all be executed in order, no test will run in parallel with another test. Yet we still need to be able to test asynchronous calls. QUnit provides us with the asyncTest() method for this purpose. I haven’t used that method though, it’s possible to do it “manually”.

We just need to pause the chain of tests before our asynchronous test. Once that test is completed, we can resume the rest of the tests. This is basically just waiting for an asynchronous test to complete, so all things considered, we test an asynchronous test synchronously. Nothing will happen in parallel.

    /**
     * Verify working of loadJS method:
     */
    test("Tag.core.loadJS()",function() {
        // verify loading:
        window.Tag.core.loadJS("dummy.js");
        // when testing asynchronous functionality, we must temporarily stop the
        // running of the tests:
        stop();
        window.setTimeout(function(){
            equals(typeof window.dummy, "object", "Verifying existence of loaded dummy");
            // start the rest of the tests:
            start();
        },250);
    });

First, the chain of tests is paused via the stop() method. As soon as the test is done, the start() method will resume the normal chain of events. Simple but effective.

The asyncTest() method from QUnit does exactly the same as we have done here. It just executes the stop() method for you. You only need to indicate when to resume the tests.

Beware, an asynchronous test may only contain 1 asynchronous call. E.g. not executing 2 AJAX calls in 1 test but rather just 1. The test itself can contain multiple assertions.

    test("Tag.core.loadJS() multiple",function() {
        // verify loading multiple:
        window.Tag.core.loadJS(["dummy.js","dummy2.js"]);
        stop();
        window.setTimeout(function(){
            equals(typeof window.dummy, "object", "Verifying existence of loaded dummy");
            equals(typeof window.dummy2, "object", "Verifying existence of loaded second dummy");
            // start the rest of the tests:
            start();
        },250);
    });

The test results

While writing tests, you will want to see some results. Just save your files, and open the environment HTML file in your browser. The tests will be run, and you’ll be presented with the results:

Complete results, no errors

All failed assertions are marked in red. Any test can be clicked to see in a more detailed manner which assertions had what response:

Suite with errors and detail

If you provided meaningful descriptions in your tests, it should be easy to see where an assertion failed.

Conclusion

Unit testing your Javascript functionality couldn’t be made easier. Getting started shouldn’t be the threshold, it’s as easy as creating one HTML file. Writing tests can be a bit of an adventure. At first you’ll probably make some mistakes, but that’s okay. The more you write tests, the more efficient you’ll make them. But you have to start somewhere. Just practice.

Remember, I can only show you the door. You have to walk through it.

All code from previous three blog posts now has a complete coverage from unit tests. You can still see the code at my Codaset repository. Please, feel free to download, fork, comment, …

Building your own Javascript library (part 3)

Tom — Wed, 02 Dec 2009 16:59:34 +0000

Goal

In this final part of the series, I’m going to create a Publish/Subscribe provider for my new framework. With this so-called pubsub provider, it will be possible for any function to create or fire an event. Any other function may then listen for that event, and will get triggered when the event occurs. My main goal is to make sure that this provider is easy to use, not only for plugins, but also for regular functionality. Besides that, I also want to have my code as loosely coupled as possible. For that, I’m going to use a little bit of Dependency Injection + lazy loading. All this to make sure that plugins are as agnostic as possible about the outside world. If you want to know more, please read on!

Updates to the base library

I’m going to start with some amends to the functionality I have already created in the first part of the series. Don’t know if you remember, but there I created functionality that is copied inside each plugin that registers itself into the base class. One of these methods is called “toString”. During my testing, I have discovered that using that name is not such a good idea. When using Firebug, the “toString” method is constantly called when logging a method. That’s why I decided to rename the method to “_toString”.

Besides the name change, I have also added a local cache in the method. This is to make sure that the expensive for loop is executed only once. The result is then put in a local cache. On each subsequent call of “_toString”, the local cache is used.

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
 
            /*
             * @var Object - Contains all plugins
             */
            this.fn = {};
 
            /**
             * Basic extend functionality. Each new plugin should extend
             * the TAG library, and become part of its namespace
             *
             * @param namespace String - Namespace of the new plugin
             * @param obj Object - The new plugin
             *
             * @return void
             */
            this.extend = function(namespace,obj){
                if (typeof this[namespace] == "undefined"){
                    if (typeof this.fn[namespace] === "undefined" && typeof this[namespace] === "undefined"){
                        // extend each namespace with core functionality
                        $.extend(obj,this.fn.extFN);
 
                        // load the new plugin in the namespaces:
                        this.fn[namespace] = obj;
                        this[namespace] = this.fn[namespace];
 
                        // initialize the new library if necessary
                        if (typeof this[namespace].init === "function"){
                            this[namespace].init();
                        }
                    } else {
                        alert("The namespace '" + namespace + "' is already taken...");
                    }
                }
            };
 
            /**
             * Recursively set values in an object
             *
             * Method allows to set individual settings, without overwriting
             * existing other values in the settings.
             *
             * @param object Object - The object to modify
             * @param data Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            this.setObjectData = function(object, data) {
               var path = (typeof arguments[2] !== "undefined")?arguments[2]:[];
               for(var k in data){
                  path.push(k);
                  if (typeof data[k] === "object"){
                      this.setObjectData(object, data[k], path);
                  } else {
                      var tmpObject = object;
                      for(var i=0,l=path.length;i<l-1;i++){
                          tmpObject = tmpObject[path[i]];
                      }
                      tmpObject[k] = data[k];
                  }
               }
            };
        };
        window.Tag = new Tag();
 
        window.Tag.extFN = window.Tag.fn.extFN = {
            /*
             * @var Object - Cache object
             */
            _cache:{},
 
            /**
             * Set the settings of a plugin
             *
             * @param settings Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            setConfig:function(settings) {
               window.Tag.setObjectData(this.settings, settings);
               return this;
            },
 
            /**
             * Automagically determines correct name of a plugin
             *
             * @return String
             */
            _toString:function() {
                // Because a loop is costly, we'll only do it once, and cache
                // the result of the loop
                if (typeof this._cache['_toString'] === 'undefined') {
                    // the name is not cached, so we need to cache it:
                    this._cache['_toString'] = '';
                    for(var k in window.Tag.fn) {
                        if (window.Tag.fn[k] === this) {
                            this._cache['_toString'] = 'Tag.' + k;
                            break;
                        }
                    }
                }
 
                // notify the user that this is an unknown plugin
                if (this._cache['_toString'] === '') {
                    alert('Could not determine the name of the plugin');
                }
 
                // return the cached name:
                return this._cache['_toString'];
            }
        };
    }
})(jQuery);

While I was busy refactoring the “toString” method, I added a “getParent” method too. This method should be used when you want to refer to the Tag base class from within a plugin, without having to know the name. The advantage is that when Tag gets renamed, the plugins still need very little change in their code.

            //... inside the extFN part of the base object:
 
            /**
             * Returns the base object
             *
             * @return Object
             */
            getParent:function() {
                return window.Tag;
            },

For such a small method, I’m not going to repeat the entire base class again. Saves me some space, because this will be a long article

Creating the Pubsub plugin

Now for the new code. We start of with our basic template again:

(function(){
    var pubsub = {
        /**
         * @var object settings - The configuration of this library
         */
        settings:{}
    };
 
    // Let's put this in the 'pubsub' namespace of TAG:
    Tag.extend("pubsub",pubsub);
})();

Here I do need the settings. It will be filled up a bit later. A Pubsub provider will have 3 basic methods. In other implementations, they may have different names than what I’ll use, but that’s not very important. The methods I need are these:

A method to listen for an event: “subscribe”
A method to stop listening for an event: “unsubscribe”
A method to fire off an event: “notify”

I’ll start with subscribing to an event. The premise here is that a method registers itself as a listener for a particular event. So I need to know: what event? what should happen when that event occurs?

I have to keep track of that information. For this purpose, I’m going to create 2 properties in the plugin: One for keeping track of what method should be called on what event. And another one to make sure a subscription can easily be undone.

(function(){
    var pubsub = {
        /*
         * @var Object - Contains all subscriptions
         */
        subscriptions:{},
 
        /*
         * @var Object - Contains all subscription ID's
         */
        subscriptionIDs:[],
 
        /**
         * @var object settings - The configuration of this library
         */
        settings:{
            separator:'.'
        },
 
        /**
         * Create a subscription to a given notification
         *
         * @param plugin String - Name of the plugin that will send out the notification
         * @param evt String - Name of the notifiation event
         * @param fn Function - Callback function to be executed on notification
         *
         * @return integer - The id of the subscription
         */
        subscribe:function(plugin, evt, fn){
            var path = this._constructPath([plugin, evt]);
            if (typeof this.subscriptions[path] === 'undefined') {
                this.subscriptions[path] = [];
            }
 
            this.subscriptions[path].push(fn);
            this.subscriptionIDs.push({
                path:path,
                index:(this.subscriptions[path].length - 1)
            });
 
            return (this.subscriptionIDs.length - 1);
        },
 
        /**
         * Construct a path from the arguments
         *
         * @return String
         */
        _constructPath:function(list) {
            return list.join(this.settings.separator);
        }
    };
 
    // Let's put this in the 'pubsub' namespace of TAG:
    Tag.extend("pubsub",pubsub);
})();

Let me walk you through the code. The properties for keeping track are added and I have introduced a setting. This setting will be used in the method “_constructPath”. The “subscribe” method is added, and takes 3 parameters: the plugin that will fire the event, the event name, and the function that should be executed when that event takes place.

First, I create a ‘path’ from the plugin name and the event name. If the plugin is ‘Tag.autosuggest’, and the event is ‘gotResponse’, then the path will be ‘Tag.autosuggest.gotResponse’. I have created a method just to create the path. If I ever find out that the path should change, then I only need to change it in that one method. That’s called keeping your code DRY.

Second, the properties are filled up with the necessary data. Each path will be stored in the “subscriptions” property, along with a list of functions that will be executed when that event occurs. If two listeners subscribe to an event, then the index of that event will have a list of 2 functions.

Third, to make it’s easy to unsubscribe from an event, a second list is kept. Each subscription gets an entry in that list. The index of that subscription is then returned at the end of the “subscribe” method. With this ID, it’s possible to unsubscribe without having to specify the plugin and event.

(function(){
    var pubsub = {
        /*
         * @var Object - Contains all subscriptions
         */
        subscriptions:{},
 
        /*
         * @var Object - Contains all subscription ID's
         */
        subscriptionIDs:[],
 
        /**
         * @var object settings - The configuration of this library
         */
        settings:{
            separator:'.'
        },
 
        /**
         * Create a subscription to a given notification
         *
         * @param plugin String - Name of the plugin that will send out the notification
         * @param evt String - Name of the notifiation event
         * @param fn Function - Callback function to be executed on notification
         *
         * @return integer - The id of the subscription
         */
        subscribe:function(plugin, evt, fn){
            var path = this._constructPath([plugin, evt]);
            if (typeof this.subscriptions[path] === 'undefined') {
                this.subscriptions[path] = [];
            }
 
            this.subscriptions[path].push(fn);
            this.subscriptionIDs.push({
                path:path,
                index:(this.subscriptions[path].length - 1)
            });
 
            return (this.subscriptionIDs.length - 1);
        },
 
        /**
         * Cancels a subscription to a given notification
         *
         * @param id Integer - ID of the subscription
         *
         * @return Object - Pubsubhub plugin for chainability
         */
        unsubscribe:function(id) {
            if (typeof this.subscriptionIDs[id] !== "undefined" && this.subscriptionIDs[id] !== null) {
                // don't remove it with splice, or other id's will be FUBAR
                this.subscriptions[this.subscriptionIDs[id].path][this.subscriptionIDs[id].index] = null;
                this.subscriptionIDs[id] = null;
            } else {
                alert('Unable to remove this subscription');
            }
            return this;
        },
 
        /**
         * Construct a path from the arguments
         *
         * @return String
         */
        _constructPath:function(list) {
            return list.join(this.settings.separator);
        }
    };
 
    // Let's put this in the 'pubsub' namespace of TAG:
    Tag.extend("pubsub",pubsub);
})();

The “unsubscribe” method removes the entries from our lists. Via the given ID, it looks up where the callback function is listed in the “subscriptions” property. It is very important that we don’t physically remove the entry from the array. Instead, I just overwrite what was there with “null” values.

If we remove the entries via “Array.splice”, then the array will be re-indexed. This will invalidate all other ID’s, so we can’t use “Array.splice”. Using the regular “delete” method…. well, let’s just say that in this case it would work, but I’m not too fond of using “delete” with arrays. It doesn’t actually delete the entry, it just writes null values to that index. Yeah, that’s exactly what I’m doing manually. But that’s not called “deleting an entry”, so let’s make sure we don’t get confused. We don’t want to end up refactoring “delete” with “Array.splice”, because we forgot that we don’t want an actual delete.

(function(){
    var pubsub = {
        /*
         * @var Object - Contains all subscriptions
         */
        subscriptions:{},
 
        /*
         * @var Object - Contains all subscription ID's
         */
        subscriptionIDs:[],
 
        /**
         * @var object settings - The configuration of this library
         */
        settings:{
            separator:'.'
        },
 
        /**
         * Create a subscription to a given notification
         *
         * @param plugin String - Name of the plugin that will send out the notification
         * @param evt String - Name of the notifiation event
         * @param fn Function - Callback function to be executed on notification
         *
         * @return integer - The id of the subscription
         */
        subscribe:function(plugin, evt, fn){
            var path = this._constructPath([plugin, evt]);
            if (typeof this.subscriptions[path] === 'undefined') {
                this.subscriptions[path] = [];
            }
 
            this.subscriptions[path].push(fn);
            this.subscriptionIDs.push({
                path:path,
                index:(this.subscriptions[path].length - 1)
            });
 
            return (this.subscriptionIDs.length - 1);
        },
 
        /**
         * Cancels a subscription to a given notification
         *
         * @param id Integer - ID of the subscription
         *
         * @return Object - Pubsubhub plugin for chainability
         */
        unsubscribe:function(id) {
            if (typeof this.subscriptionIDs[id] !== "undefined" && this.subscriptionIDs[id] !== null) {
                // don't remove it with splice, or other id's will be FUBAR
                this.subscriptions[this.subscriptionIDs[id].path][this.subscriptionIDs[id].index] = null;
                this.subscriptionIDs[id] = null;
            } else {
                alert('Unable to remove this subscription');
            }
            return this;
        },
 
        /**
         * Send out a notification to all subscribers
         *
         * @param plugin String - Name fo the plugin
         * @param evt String - Name of the event
         *
         * @return Object - Pubsubhub plugin for chainability
         */
        notify:function(plugin, evt) {
            var path = this._constructPath([plugin, evt]);
            if (typeof this.subscriptions[path] !== 'undefined') {
                for(var i=0,l=this.subscriptions[path].length;i<l;i++){
                    // make sure we only execute functions, not strings, integers, null's, ...
                    if (typeof this.subscriptions[path][i] === 'function') {
                        this.subscriptions[path][i]();
                    }
                }
            }
        },
 
        /**
         * Construct a path from the arguments
         *
         * @return String
         */
        _constructPath:function(list) {
            return list.join(this.settings.separator);
        }
    };
 
    // Let's put this in the 'pubsub' namespace of TAG:
    Tag.extend("pubsub",pubsub);
})();

With the “notify” method, you fire off an event. There are two arguments: name of the plugin and name of the event. From these two arguments, the path is constructed again. Then I look up in the “subscriptions” list if that entry exists. If it does, then all functions listed there will be executed.

The “notify” method works as a fire-and-forget system. If there are no listeners for an event, then nothing will happen. There is no communication back to the method that fired the event. That’s not necessary. It’s not its task to know if the event will be correctly dispatched.

At the moment, “notify” doesn’t transmit a payload to the subscribers. I’m sure I will need that functionality eventually. I’ll add it later, when the need is actually there.

The problem of scope

When a method subscribes to an event, and then gets executed, there will be a problem with the scope of the method. The Pubsub plugin doesn’t know anything about the callback function, other than it is a function. It can either be an anonymous function, or it can be part of an object. Pubsub doesn’t know. Hence, when the function is called, there won’t be a scope. The scope is the function itself. For an anonymous function, that’s good. For a method that’s part of an object, that’s not good

Possible solution 1: Just execute the callback in the scope of the Pubsub provider, like this:

// ...
this.subscriptions[path][i].call(this);
// ...

This is not a good solution, because anonymous functions suddenly have a scope and methods from an object suddenly have a scope that isn’t theirs. If the method is ‘Tag.autosuggest.updateSuggestions’, then the scope should be ‘Tag.autosuggest’, and not ‘Tag.pubsub’.

Possible solution 2: Execute the callback in the scope of the notifier. Via some code, it should be easy to determine the scope of the notifier. But again, my same objections apply as with the first solution: Anonymous functions shouldn’t have a scope, and I don’t want to change the scope of methods from an existing object

Possible (and accepted) solution 3: Make sure the callback already has the correct scope before executing it. (see below)

Hitching a ride

(I told you this was going to be a long article) The idea comes from “dojo.hitch”: Via another method making sure that a given function is executed in a given scope. So I created “Tag.core.hitch” for this purpose. The name comes from Dojo, and the implementation is derived from dojo’s implementation.

        // inside the Tag.core plugin:
        /**
         * Runs fn in the given scope
         *
         * @param scope Object - The scope fn should run in
         * @param fn Function|String - The function to run
         *
         * Extra parameters are optional, and will be passed on to fn
         *
         * @return Function
         */
        hitch:function(scope, fn) {
            // arguments looks like an Array, but is an Object
            // here we convert it to an array
            var args = Array.prototype.slice.call(arguments);
            if (typeof scope === 'undefined' || typeof fn === 'undefined') {
                // both scope & fn should be defined
                alert('Tag.hitch: both scope and function should be defined');
                return null;
            }
            if(typeof fn === 'string'){
                if (typeof scope[fn] === 'undefined') {
                    alert('The function is not defined inside the given scope');
                    return null;
                }
                return function(){ return scope[fn].apply(scope, args.slice(2)); };
            }
            return function(){ return fn.apply(scope, args.slice(2)); };
        }

This method returns an anonymous function. Inside that anonymous function, the scope of the given function is set to the scope that’s defined in the arguments. Quite simple actually. But very effective.

With the hitch method, a subscriber is now in charge of making sure that the callback function is executed in the correct scope.

Dependency Injection

Before I come to the usage examples, there is one more thing I’d like to implement. You know how I like keeping my plugins agnostic. Well, instead of letting a plugin use the pubsub provider like this: “Tag.pubsub.notify(…)”, I want to make sure that it’s not important that the pubsub provider is called “Tag.pubsub”. In fact, if someone comes along and writes a better provider, it should be possible to use that provider without having to change the code of the plugins.

This can be accomplished with a little Dependency Injection. (DI via getters and setters to be precise). I’ll extend core plugin with 2 methods: “getPubsubProvider” and “setPubsubProvider”. With the getter, the Pubsub provider is returned. Via he setter it’s possible to specify another Pubsub provider. As the icing on the cake, I’ll add some lazy loading: if no pubsub provider is specified, then “getPubsubProvider” defaults to “Tag.pubsub”.

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
 
            /*
             * @var Object - Contains all plugins
             */
            this.fn = {};
 
            /**
             * Basic extend functionality. Each new plugin should extend
             * the TAG library, and become part of its namespace
             *
             * @param namespace String - Namespace of the new plugin
             * @param obj Object - The new plugin
             *
             * @return void
             */
            this.extend = function(namespace,obj){
                if (typeof this[namespace] == "undefined"){
                    if (typeof this.fn[namespace] === "undefined" && typeof this[namespace] === "undefined"){
                        // extend each namespace with core functionality
                        $.extend(obj,this.fn.extFN);
 
                        // load the new plugin in the namespaces:
                        this.fn[namespace] = obj;
                        this[namespace] = this.fn[namespace];
 
                        // initialize the new library if necessary
                        if (typeof this[namespace].init === "function"){
                            this[namespace].init();
                        }
                    } else {
                        alert("The namespace '" + namespace + "' is already taken...");
                    }
                }
            };
 
            /**
             * Recursively set values in an object
             *
             * Method allows to set individual settings, without overwriting
             * existing other values in the settings.
             *
             * @param object Object - The object to modify
             * @param data Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            this.setObjectData = function(object, data) {
               var path = (typeof arguments[2] !== "undefined")?arguments[2]:[];
               for(var k in data){
                  path.push(k);
                  if (typeof data[k] === "object"){
                      this.setObjectData(object, data[k], path);
                  } else {
                      var tmpObject = object;
                      for(var i=0,l=path.length;i<l-1;i++){
                          tmpObject = tmpObject[path[i]];
                      }
                      tmpObject[k] = data[k];
                  }
               }
            };
        };
        window.Tag = new Tag();
 
        window.Tag.extFN = window.Tag.fn.extFN = {
            /*
             * @var Object - Cache object
             */
            _cache:{},
 
            /*
             * @var Object - Pubsub provider
             */
            pubsubProvider:null,
 
            /**
             * Set the settings of a plugin
             *
             * @param settings Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            setConfig:function(settings) {
               window.Tag.setObjectData(this.settings, settings);
               return this;
            },
 
            /**
             * Inject another Pubsub Provider
             *
             * @param obj Object - The new Pubsub Provider
             *
             * @return Object - Current plugin for chainability
             */
            setPubsubProvider:function(obj) {
                // verify if necessary functionality is present:
                var fns = ["subscribe","unsubscribe","notify"];
 
                var validObject = true;
                $.each(fns,function(){
                    if (typeof obj[this] !== "function"){
                        validObject = false;
                        return;
                    }
                });
 
                if (validObject) {
                    this.pubsubProvider = obj;
                }
 
                return this;
            },
 
            /**
             * Get the Pubsub Provider
             *
             * @return Object - Pubsub Provider
             */
            getPubsubProvider:function() {
                // lazy loading:
                if (null === this.pubsubProvider) {
                    this.pubsubProvider = window.Tag.pubsub;
                }
 
                return this.pubsubProvider;
            },
 
            /**
             * Returns the base object
             *
             * @return Object
             */
            getParent:function() {
                return window.Tag;
            },
 
            /**
             * Automagically determines correct name of a plugin
             *
             * @return String
             */
            _toString:function() {
                // Because a loop is costly, we'll only do it once, and cache
                // the result of the loop
                if (typeof this._cache['_toString'] === 'undefined') {
                    // the name is not cached, so we need to cache it:
                    this._cache['_toString'] = '';
                    for(var k in window.Tag.fn) {
                        if (window.Tag.fn[k] === this) {
                            this._cache['_toString'] = 'Tag.' + k;
                            break;
                        }
                    }
                }
 
                // notify the user that this is an unknown plugin
                if (this._cache['_toString'] === '') {
                    alert('Could not determine the name of the plugin');
                }
 
                // return the cached name:
                return this._cache['_toString'];
            }
        };
    }
})(jQuery);

That’s the entire base class for now. As you can see, “setPubsubProvider” verifies if the new provider has all three required methods. I added this so that the plugins don’t have to be rewritten. If a new Pubsub Provider is introduced, with different names for the methods, then all you have to do is write an adapter for it, and set it via “setPubsubProvider”.

Usage

A lot of code has been written and added. Here are some examples how all this can be used. For most examples, I’ll use a fictive “Tag.autosuggest” plugin:

// inside the Tag.autosuggest plugin, we throw an event:
this.getPubsubProvider().notify(this._toString(),'thresholdReached');

(you see, no ‘Tag’ mentioned here. I like!)

Inside another plugin, we listen for the above event:

var id = this.getPubsubProvider().subscribe(this.getParent().autosuggest._toString(),'thresholdReached',function(){
    alert('Treshold has been reached');
});

(Still no mention of ‘Tag’ here. True, we have to write a lot of code, but it’s future proof)

Instead of an anonymous function, I want to execute a method of an object. We have to keep scope here:

var id = this.getPubsubProvider().subscribe(this.getParent().autosuggest._toString(),'thresholdReached',this.getParent().core.hitch(this,this.callbackFunction));

(“callbackFunction” will be executed in the correct scope)

An alternative to the above is:

var id = this.getPubsubProvider().subscribe(this.getParent().autosuggest._toString(),'thresholdReached',this.getParent().core.hitch(this,"callbackFunction"));

(Here, “callbackFunction” is specified as a string. Other than that, the exact same will happen as the previous example)

Unsubscribe from an event:

var id = this.getPubsubProvider().subscribe(this.getParent().autosuggest._toString(),'thresholdReached',this.getParent().core.hitch(this,this.callbackFunction));
 
this.getPubsubProvider().unsubscribe(id);

All above examples implied another plugin. But the code can also be used in regular Javascript, provided that Tag and Tag.pubsub are available on the page.

function doCalculation (a, b) {
    Tag.pubspub.notify('doCalculation','beforeCalculation');
    var result = a + b;
    Tag.pubspub.notify('doCalculation','afterCalculation');
    return result;
}
 
var id = Tag.pubsub.subscribe('doCalculation','afterCalculation',function(){
    alert('Calculation complete');
});
 
Tag.pubsub.unsubscribe(id);

Instead of the plugin name, I provided the function name to the “notify” and “subscribe” methods.

Conclusion

Finally, the end of a lengthy article. With the pubsub plugin, it should be easy to make your plugins event-driven, and loosely coupled. Already, a shortcoming is visible: It should be possible to transmit data from the notifier to the subscriber. In the coming days or maybe next week, I’ll refactor the code to allow that. Once that feature is added, it will be possible to write your application as a blackboard system, with all functionality independent from each other.

In these articles, I have introduced some principles from the OO world, like Dependency Injection, modular design, lazy loading, DRY, … If you want to read them again, the first article is available here, and the second article is available here. The code of this library is under constant development. I’ve made it available under the MIT license on my Codaset account on http://www.codaset.com/theanalogguy/the-analog-guy-javascript-library. Feel free to fork it, add change requests or bug reports. If you want to use the code, and you create plugins for it, please let me know. I’m curious if and how this will be used by other people.

Thank you for reading my article. Hope you enjoyed it.

Building your own Javascript library (part 2)

Tom — Tue, 01 Dec 2009 16:15:53 +0000

Goal

In this second installment of my 3-part library-building journey, I’m going to make the core plugin for the library. This plugin will contain utility methods. Stuff that can be used, either by other plugins, or just by regular Javascript code. So far, I haven’t needed many utility methods, so this chapter will be rather short. If you’re ready, read on!

Creating the Core plugin

So far, the only functionality I have needed, was dynamically loading other Javascript files on a page. jQuery is perfectly capable of doing just that. I’m just going to wrap it up in a method or two.

First, let’s start off with the base of my plugin. A self-contained, anonymous function that will be executed immediately.

(function(){
    var core = {
 
    };
 
    // Let's put this in the 'core' namespace of TAG:
    Tag.extend("core",core);
})();

In my previous blog post, there were some extra’s there: settings and an init method. I won’t be needing those at this moment, so I left them out.

There are two important things I need when dynamically loading other Javascript files. I want to be able to do something when the file is loaded. And in some cases I want to make sure that a file is loaded only once. The reason for that last feature is that when I want to create self-contained partials, I don’t want to load the same file over and over again when that partial is put on my page more than once.

That being said, the first method will accept a file URL and optionally a callback. The callback allows me to execute code after the file was loaded.

(function(){
    var core = {
        /**
         * Loads a JS library via Ajax and execute an optional callback function
         *
         * @param url String - The location of the script
         * @param callback Function - Optional callback to execute after script was loaded
         * @return Object - Core plugin for chainability
         */
        loadJS:function(url,callback){
            if (typeof callback === "undefined") {
                callback = function(){};
            }
 
            // load it via Ajax
            $.getScript(url,callback);
            return this;
        }
    };
 
    // Let's put this in the 'core' namespace of TAG:
    Tag.extend("core",core);
})();

As you can see, I just used jQuery’s getScript() method, so all restrictions apply. The return value of the method is the current object or scope. This allows you to chain multiple methods, like this:

	Tag.core.loadJS('/path/to/first.js').loadJS('/path/to/second.js').someOtherFunctionFromCore();

This keeps it all nice and tidy. You can read up on chainability in Javascript here on Javascript Ant. Chaining methods is commonly used in PHP as well. I like that principle, because it makes your code shorter and more readable.

Now that I can load 1 file, I want to be able to make sure it’s loaded only once. For that the plugin needs an extra method, and a list to put the loaded libraries in.

(function(){
    var core = {
        /**
         * @var array - List of loaded JS scripts
         */
        libraries:[],
 
        /**
         * Loads a JS library via Ajax and execute an optional callback function
         *
         * @param url String - The location of the script
         * @param callback Function - Optional callback to execute after script was loaded
         * @return Object - Core plugin for chainability
         */
        loadJS:function(url,callback){
            if (typeof callback === "undefined") {
                callback = function(){};
            }
 
            // load it via Ajax
            $.getScript(url,callback);
            return this;
        },
 
        /**
         * Loads a JS library once via Ajax, with optional callback
         *
         * @param url String - The location of the script
         * @param callback Function - Optional callback to execute after script was loaded
         * @return Object - Core plugin for chainability
         */
        loadOnce:function(url,callback){
            if (typeof callback === "undefined"){
                callback = function(){};
            }
 
            if ($.inArray(url,this.libraries) == -1) {
                // load JS:
                this.loadJS(url, callback);
 
                // append to list:
                this.libraries.push(url);
            } else {
                // just execute the callback when the library is already loaded
                callback();
            }
            return this;
        }
    };
 
    // Let's put this in the 'core' namespace of TAG:
    Tag.extend("core",core);
})();

The “loadOnce” method takes the same input as the “loadJS” method. But as a bonus, it keeps track of the files it has loaded. When some code is trying to load a file for the second time, then there is no penalty. The callback will just be executed, just as if it were the first time the file was loaded.

Usage

I’ll just give you some examples. Simplest case, you just want to load a file:

	Tag.core.loadJS('/path/to/file.js');

Load a couple of files:

	Tag.core.loadJS('/path/to/file.js');
	Tag.core.loadJS('/path/to/anotherfile.js');
 
	// OR
	Tag.core.loadJS('/path/to/file.js').loadJS('/path/to/anotherfile.js');

Load a file, and then execute some code:

	Tag.core.loadJS('/path/to/file.js',function(){
		alert('File is loaded');
	});

Load a file after another file was loaded:

	Tag.core.loadJS('/path/to/file.js',function(){
		Tag.core.loadJS('/path/to/anotherfile.js');
	});

Load a plugin, and then set the correct settings for it:

	Tag.core.loadJS('/path/to/myPlugin.js',function(){
		Tag.myPlugin.setConfig({
			url:'http://www.example.com/',
			selectors:{
				exampleLink:'.example'
			}
		});
	});

The list could go on forever. All these examples are also valid for the “loadOnce” method.

In the last part of this series, I’ll create my Pubsub plugin: an event dispatcher. For this, I will have to modify the base class a bit, as I want to make use of Dependency Injection. But that’s for tomorrow.

All code is still available on my Codaset account here: http://www.codaset.com/theanalogguy/the-analog-guy-javascript-library

Enjoy!

Building your own Javascript library (part 1)

Tom — Mon, 30 Nov 2009 18:39:56 +0000

Goal

There are two programming languages I like: PHP and Javascript. In this mini series, I will explore Javascript and jQuery, and build my own library/framework. I won’t re-invent the wheel here. I’ll just make a collection of methods and functionality I need, making use of some functionality in jQuery.

The goal of the first part of my mission is to create a small, lightweight, self-contained, easily extensible base for my framework. Recently I created my own little company, called “The Analog Guy”. I’ll just call my library “TAG”.

Creating the base

One thing I have learned from using other libraries, is that I don’t like initializing an object in Javascript. I just want to include the script on my page, and it has to work out of the box. I’ll use a self-executing anonymous function to encapsulate my class, initialize it, and add some functionality. We start with the blueprint of the self-executing anonymous function:

(function($){
    // rest goes here
})(jQuery)

This is an anonymous function that will be executed immediately. As a parameter to this anonymous function, I give the jQuery object. This makes sure that inside my anonymous function, $ refers to the jQuery object. Even if on the rest of the page, another library is used that defines $. That’s called the scope of the object.

So well, next thing I want to do, is reserve my own TAG namespace. So I’ll add a little code for that:

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // rest goes here
        }
    }
})(jQuery)

If another script uses the “Tag” namespace in the global scope, then that might indicate that the user has accidentally included the tag.js file twice on the same page. I’ve made sure that it the namespace only gets defined just once.

In my goal, I said I was going to build this library on top of jQuery. I want to make sure that $ is defined, and that it is jQuery. I don’t want to support other libraries at this moment.

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
        }
    }
})(jQuery)

Now that I know that jQuery is available, the next thing I want to add, is extensibility. I want to keep the base lightweight: only add the functionality that is needed. All the other functionality, that is non-essential, should be added via plugins or extensions. This makes it possible:

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
 
            // Object FN will contain all plugins
            this.fn = {};
 
            /**
             * Basic extend functionality. Each new plugin should extend
             * the TAG library, and become part of its namespace
             *
             * @param namespace String - Namespace of the new plugin
             * @param obj Object - The new plugin
             *
             * @return void
             */
            this.extend = function(namespace,obj){
                if (typeof this[namespace] == "undefined"){
                    if (typeof this.fn[namespace] === "undefined" && typeof this[namespace] === "undefined"){
                        // extend each namespace with core functionality
                        $.extend(obj,this.fn.extFN);
 
                        // load the new plugin in the namespaces:
                        this.fn[namespace] = obj;
                        this[namespace] = this.fn[namespace];
 
                        // initialize the new library if necessary
                        if (typeof this[namespace].init === "function"){
                            this[namespace].init();
                        }
                    } else {
                        alert("The namespace '" + namespace + "' is already taken...");
                    }
                }
            };
        }
    }
})(jQuery)

The extend method has two arguments: the name of the namespace the plugin wants to use, and then the plugin itself. First there is a check if that namespace is still available. If it is, the first thing I do, is add some basic functionality to the plugin, that will make life easier for the developer of that plugin. Don’t worry, we’ll see immediately where I add that core functionality. A next step is to inject the plugin into the base class. For this, I use the “this.fn” object, because that makes it easier to detect which plugins are loaded. As a “bonus”, I also inject that same plugin in the base namespace. That makes it more convenient to use the plugin. “Tag.fn.myPlugin” can be referred to as “Tag.myPlugin”. Lastly, if that plugin needs some automagical setup, it can define the “init” method, which will get triggered when loading the plugin.

This provides basic extensibility for my framework. The next method I need, is a utility method. I cannot place it on a plugin, because it is something basic: a method that recursively sets a value inside an object

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
 
            // Object FN will contain all plugins
            this.fn = {};
 
            /**
             * Basic extend functionality. Each new plugin should extend
             * the TAG library, and become part of its namespace
             *
             * @param namespace String - Namespace of the new plugin
             * @param obj Object - The new plugin
             *
             * @return void
             */
            this.extend = function(namespace,obj){
                if (typeof this[namespace] == "undefined"){
                    if (typeof this.fn[namespace] === "undefined" && typeof this[namespace] === "undefined"){
                        // extend each namespace with core functionality
                        $.extend(obj,this.fn.extFN);
 
                        // load the new plugin in the namespaces:
                        this.fn[namespace] = obj;
                        this[namespace] = this.fn[namespace];
 
                        // initialize the new library if necessary
                        if (typeof this[namespace].init === "function"){
                            this[namespace].init();
                        }
                    } else {
                        alert("The namespace '" + namespace + "' is already taken...");
                    }
                }
            };
 
            /**
             * Recursively set values in an object
             *
             * Method allows to set individual settings, without overwriting
             * existing other values in the settings.
             *
             * @param object Object - The object to modify
             * @param data Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            this.setObjectData = function(object, data) {
               var path = (typeof arguments[2] !== "undefined")?arguments[2]:[];
               for(var k in data){
                  path.push(k);
                  if (typeof data[k] === "object"){
                      this.setObjectData(object, data[k], path);
                  } else {
                      var tmpObject = object;
                      for(var i=0,l=path.length;i<l-1;i++){
                          tmpObject = tmpObject[path[i]];
                      }
                      tmpObject[k] = data[k];
                  }
               }
            };
        }
    }
})(jQuery)

I know jQuery.extend() exists. I just don’t think it can do what I want: set some properties of an object, without touching other existing properties. The method “setObjectData” I just created will do just that. It recursively loops through a set of values, and sets these values on the given object. This method will be used later, for setting values in a plugin configuration.

For now, I’m happy with that functionality. I don’t have an immediate use for other functionality. So we can initialize our class now. Since this takes only 1 line of code, I’ll immediately add some more functionality.

There are certain methods, I want to be available in every plugin, without the plugin having to know the name of my base class. I think it’s a basic need of the plugins, to be as agnostic as possible about the rest of the world. The fewer the dependencies, the better. If I ever decide to rename my company, and I rename my base library. Then the plugin should still work with little or no effort. These methods are “setConfig”, to set values in a plugin config, and “toString”, to automagically determine the name of the plugin.

(function($){
    if (typeof window.Tag == "undefined"){
        var Tag = window.Tag = function(){
            // We have a dependency on jQuery:
            if (typeof $ === "undefined" || $ !== window.jQuery){
                alert("Please load jQuery library first");
            }
 
            // Object FN will contain all plugins
            this.fn = {};
 
            /**
             * Basic extend functionality. Each new plugin should extend
             * the TAG library, and become part of its namespace
             *
             * @param namespace String - Namespace of the new plugin
             * @param obj Object - The new plugin
             *
             * @return void
             */
            this.extend = function(namespace,obj){
                if (typeof this[namespace] == "undefined"){
                    if (typeof this.fn[namespace] === "undefined" && typeof this[namespace] === "undefined"){
                        // extend each namespace with core functionality
                        $.extend(obj,this.fn.extFN);
 
                        // load the new plugin in the namespaces:
                        this.fn[namespace] = obj;
                        this[namespace] = this.fn[namespace];
 
                        // initialize the new library if necessary
                        if (typeof this[namespace].init === "function"){
                            this[namespace].init();
                        }
                    } else {
                        alert("The namespace '" + namespace + "' is already taken...");
                    }
                }
            };
 
            /**
             * Recursively set values in an object
             *
             * Method allows to set individual settings, without overwriting
             * existing other values in the settings.
             *
             * @param object Object - The object to modify
             * @param data Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            this.setObjectData = function(object, data) {
               var path = (typeof arguments[2] !== "undefined")?arguments[2]:[];
               for(var k in data){
                  path.push(k);
                  if (typeof data[k] === "object"){
                      this.setObjectData(object, data[k], path);
                  } else {
                      var tmpObject = object;
                      for(var i=0,l=path.length;i<l-1;i++){
                          tmpObject = tmpObject[path[i]];
                      }
                      tmpObject[k] = data[k];
                  }
               }
            };
        };
        window.Tag = new Tag();
 
        window.Tag.extFN = window.Tag.fn.extFN = {
            /**
             * Set the settings of a plugin
             *
             * @param settings Object - New values for settings
             *
             * @return Object - Current plugin for chainability
             */
            setConfig:function(settings) {
               window.Tag.setObjectData(this.settings, settings);
               return this;
            },
 
            /**
             * Automagically determines correct name of a plugin
             *
             * @return String
             */
            toString:function() {
                for(var k in window.Tag.fn) {
                    if (window.Tag.fn[k] === this) {
                        return 'Tag.' + k;
                        break;
                    }
                }
                alert('Could not determine the name of the plugin');
                return '';
            }
        };
    }
})(jQuery);

That’s it for the moment. I’ll certainly need more functionality inside the base class, but I will build it was I need it.

Usage

Well, it’s a basic class really, there is no real usage for now. The strength will come from the plugins. To include the library in your files, simply include script tags which point to the correct JS file.

Writing your own plugins

Since I’m emphasizing plugins, I’ll give you the basic structure of how a plugin should be built. In future parts, I’ll build at least two plugins that will further extend the functionality: a “core” plugin, and a “pubsub” plugin. Here’s the template:

(function(){
    var myPlugin = {
        /**
         * @var object settings - The configuration of this library
         */
        settings:{},

        /**
         * Will be called as soon as this library is loaded
         * into the main Tag library
         *
         * @return void
         */
        init:function(){},

        // other methods go below
    };
    Tag.extend("myPlugin",myPlugin);
})();

That’s all there is to it. Just an Object Literal notation of an object, again inside a self-executing anonymous function. Include your JS file in your HTML, after you included the Tag library, and your plugin will be available as “Tag.myPlugin”.

In the next part of this mini series, I’ll create the “core” plugin. This plugin will create functionality that is quite basic, but not essential to the library. Usually the methods will be utility methods which can be used by other plugins or regular Javascript code.

In the last part of the series, I’ll introduce my “pubsub” plugin. It’s a plugin that serves as a dispatcher for events. With it, it is possible to send out event notifications. Other parts of your code will be able to (un-)subscribe to events, and will get triggered when the event is fired.

If you have any remarks, please let me know. I’m open to suggestions and improvements. My code is available at http://www.codaset.com/theanalogguy/the-analog-guy-javascript-library.

Keeping your code DRY is also in the details

Tom — Fri, 27 Nov 2009 20:01:05 +0000

What you should know already…

If you have been a developer for a while, chances are you’ll have come across the term DRY. It stands for Don’t Repeat Yourself, and it is actually a form of art. Keeping your code DRY ensures you have to think well before you start coding. Instead of randomly throwing code in your IDE, you should create the reflex of thinking how you can achieve what you want, without having to type too much code.

Not out of laziness, but because you’re smart. If you ever have to change your code, you’ll be glad you decided to write that particular bit only in that particular file. Sometimes it’s much easier to just copy/paste code from one class into another. But if you have to modify that piece of code, you’ll have to remember all the places that same code appears. Unless you kept it DRY of course.

Usually when an article talks about DRY, it will be in the context of Object Oriented Programming and model abstracts/inheritance/whatnot and making sure you don’t repeat the same code in different places. But the same principle can also be applied to other aspects of our code.

Other aspects that can be kept DRY

Use class constants

Not only does the DRY principle apply to OOP and methods in classes specificly. Other aspects of your classes should be kept DRY too. Recently, I’ve become a big fan of Class Constants in PHP. I use them to replace almost anything that is prone to change. I’ll just give you some examples:

Status checking

class MyClass {
    public function getStatus() {
        return "initialized";
    }
}
 
$object = new MyClass();
if ($object-&gt;getStatus() === "initialized") {
    // do some stuff
}

If it is decided that the status “initialized” should be renamed to “init”, then I have to change it everywhere I used that string literally.
Here’s how you could do it better:

class MyClass {
    const STATUS_INITIALIZED = "initialized";
 
    public function getStatus() {
        return self::STATUS_INITIALIZED;
    }
}
 
$object = new MyClass();
if ($object-&gt;getStatus() === MyClass::STATUS_INITIALIZED) {
    // do some stuff
}

If we now have to change “initialized” to “init”, then we only have to change the constant. All other code is left unchanged, and will still work. Now that is DRY.

Database metadata

In a database, the ENUM datatype is often used to restrict the content of a field to a specific set of data it can contain. Excellent examples are the status of the previous example, flags, … Basically, anything you want to have full control of what could be a possible entry in that field. Keep these values also in Class Constants. If you change the database, then the only other manual thing you have to do, is update the associated constant.

Configuration

Since I use the Zend Framework on a daily basis, here’s an example of how Zend Framework uses constants. First let’s see how it should not be done:

$translate = new Zend_Translate(
    'gettext',
    'my_it.mo',
    'auto',
    array('scan' =&gt; 'filename'));

Instead, they use constants:

$translate = new Zend_Translate(
    'gettext',
    'my_it.mo',
    'auto',
    array('scan' =&gt; Zend_Translate::LOCALE_FILENAME));

Keep your templates DRY

Not only is it important to keep your business logic DRY, the templates you write should benefit from the same principle. For your website basic, you won’t rewrite the entire HTML structure for each page on your site. Even if you don’t use PHP in an OOP way, you will probably still use includes to include parts on your site that will return on every page. Like the menu, or the header and footer. That’s DRY. But there’s more.

Use partials (in partials)

Partials is a term from Zend Framework. They are small templates that can be reused anywhere in your larger templates. Say you have a community site, with a messaging feature. Users can send each other messages, users can receive site notices, friend invitations, … In the inbox of a user, according to the type of message, a different partial can be shown:

foreach ($messagesList as $message) {
	$this-&gt;partial('/path/to/inbox_' . $message-&gt;__toString() . '.phtml',array('item' =&gt; $message));
}

For each messagetype that can be shown in the inbox, we create a new template. Inside that template, we do the markup for that specific message. Now in the inbox, we want to show the avatar and username of the person who sent you the message, or who invited you to become his friend. It is very tempting to write that part of the template once in inbox_message.phtml, and then copy/paste it in inbox_invitation.phtml and inbox_notice.phtml. It usually happens in a flurry of “yaaay it works !!”, and you’re so happy that you don’t care if your code becomes sloppy.

Of course, the avatar and username should be put in a new template. That new template is then used in each inbox_* template. Hence the partials inside partials. You also want to add the user ranking? No problem, only 1 file to update.

Conclusion

The DRY principle isn’t only reserved for the big chunks of your business logic. It is also an important part of the details. The small things that can easily lead to a search-and-replace rampage throughout your files. The templates are also often forgotten to be kept DRY.

As with many things: think before you act

How to let DomPDF and Zend Framework play along

Tom — Thu, 27 Aug 2009 18:06:28 +0000

For a recent project, I had to generate a PDF for a catalog. Since I’m using Zend Framework for my development, first thing that sprang in mind was Zend_Pdf. After some investigation, I found it too “difficult” to use. Flexible though that component is, I didn’t feel much for using coordinates to draw each and every line and text of that PDF. So I went to find another solution: DomPDF. DomPDF can generates PDF output, from HTML input. Exactly what I wanted and needed. It was however quite tricky to get it working. If you also have problems with this, read on!

DomPDF isn’t part of the Zend Framework library, and as such doesn’t really follow the ZF conventions. That’s what caused me trouble: I couldn’t load the needed classes for it to work.

So how do you set everything up properly?

First step: set include path

When you’re setting your include paths, just add one for DomPDF.

1	set_include_path(APPLICATION_PATH . "/../../library/dompdf-0.5.1" . PATH_SEPARATOR . get_include_path());

Second step: make ZF & DomPDF play along

DomPDF has an autoloader which should take care of autoloading the classes for you. All you need to do, is make Zend Framework aware of that autoloader. The solution I found the most was this:

1 2	require_once 'dompdf_config.inc.php'; spl_autoload_register('DOMPDF_autoload');

However this may not work if you use Zend Framework 1.8 or newer. The real solution I found on a forum, where Matthew Weier O’Phinney enlightened another soul with the same problem:

1
2
3

require_once 'dompdf_config.inc.php';
$autoloader = Zend_Loader_Autoloader::getInstance(); // assuming we're in a controller
$autoloader->pushAutoloader('DOMPDF_autoload');

Final step: generate the PDF

That’s the simplest one, as I found no problems with that:

$dompdf = new DOMPDF();
$dompdf->set_paper("a4","portrait");
$dompdf->load_html($html);
$dompdf->set_base_path($_SERVER['DOCUMENT_ROOT']);
$dompdf->render();
$dompdf->stream("document.pdf");

There you go, this should set you up to get DomPDF up and running.

Leveraging Zend_Auth for building your authentication

Tom — Thu, 07 May 2009 21:06:09 +0000

In a series of posts, I will address the issue of authentication and authorization of users into your application. When you build a website with any form of back office, you will need to grant users access to the back office (authentication), and determine what actions they are allowed to take (authorization). The Zend Framework has two tools just for that job: Zend_Auth and Zend_ACL. In this first part, I will build a custom User class, that will allow the programmer to perform a simple authentication of a user. This User class will be integrated into a so called “code base” or “framework” that you can use for your own applications. As of this moment, that framework doesn’t exist yet. I will gradually build it as my blog posts are added. Please look for the tag “codebase” if you want all of these posts.

First things first: if you have read my previous posts about securing your Flex – PHP services, you will have seen that I have wrapped the bootstrapping process of that application in a class named “CB_Application”. That class is still under very heavy development. Since the release of Zend Framework 1.8, they have introduced Zend_Application, just for that purpose. So it is very likely that in the future, I will replace my CB_Application class with something that makes use of Zend_Application. For now, I will focus on the other functionality, and leave CB_Application as it is, without bothering you with the details. But full disclosure will come, somewhere in the future.

Structure

I haven’t decided on some kind of structure for this framework yet. I think it will evolve as I add more classes and functionality to it. Below is a screenshot of how it currently looks like.

Codebase Structure

As you can see, quite empty. What we will be constructing during this blog post, is the CB_User class, the CB_Table_Users class and the CB_Auth_Adapter_Chap class. Once that is complete, you should include the Code Base folder into your include paths in your bootstrap. If that is done, you can just simple use your framework anywhere in your models or controllers.

Database tables for the User class

I’m going to concentrate on the authentication of the users, so I will keep the amount of data small. It will be the base for the User class, and will be extended with functionality and other data as the framework evolves. Right now, the only properties for a user we need, are: a username, a password, an e-mail address, and a flag to indicate if the user is active or not.

CREATE TABLE IF NOT EXISTS `cb_users` (
  `id` int(5) NOT NULL AUTO_INCREMENT,
  `username` varchar(30) NOT NULL,
  `password` varchar(255) NOT NULL,
  `email` varchar(255) NOT NULL,
  `active` tinyint(1) NOT NULL DEFAULT '1',
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8;

Small remark:All passwords are encrypted via the AES encryption method. I won’t use a hashing algorithm, because during the authentication process, I would like to know the exact password. In case I want to build PHP services for Flex, I will need that password, and I can’t come from the client directly. But that’s nothing to worry about now; just remember that the password will be encrypted via AES.

For testing purposes, we will add a dummy user via SQL now. Normally this will be done during some sort of registration process, but that’s for the future. Let’s add the user “test” with password “password” into the database. As key for the AES encryption, I have used “c0d3ba53″. You know, l33t speak as they say You should remember that key. I will hard code it into the code, as this is just a demonstration. In my real code, I have this key in a configuration file, and it is fetched by CB_Application. (I know, it’s irritating, but it will come up here an there during this blog post. Sorry!)

1	INSERT INTO cb_users (username,password,email,active) VALUES ('test','S í‹ôÚ×ýfÕ‘ˆ? ','test@example.com',1);

Building the authentication

Authenticating a user will be very straightforward. There is only 1 condition: I want to be able to use CHAP and regular authentication. CHAP requires 3 parameters (username, challenge, signature) and regular authentication requires only 2 (username, password). I will incorporate that into 1 authentication function that will dynamically decide which authentication we will be doing, via the amount of parameters.

As I said before, the authentication process will make use of the Zend_Auth component of the Zend Framework. There are many different ways you could authenticate a user. Most commonly is against a database. Zend_Auth can accommodate to all these different ways, by making use of adapters. An auth adapter basically does the authentication, and Zend_Auth uses it. There are already some predefined adapters, but for the CHAP authentication, we will have to create our own.

To access all the data from the cb_users table in the database, We extend the regular Zend_Db_Table class with our own information:

class CB_Table_Users extends Zend_Db_Table_Abstract {
	protected $_name = 'cb_users';
	protected $_primary = "id";
}

Nothing fancy, we just specify our table name, and the primary key, and then we’re good to go.

Before we can do the authentication, we first need to create our custom CHAP authentication adapter. The Zend Framework is designed to be extensible, and there is in interface for creating your own authentication adapters. The only method you need to implement is the authenticate() method. But you can add other methods if you deem that necessary. I will implement some setters for the credentials, and then a method that mimics the behaviour of the getResultRowObject() method from Zend_Auth_Adapter_DbTable. I added this extra functionality, because that will make it easier to process everything in the CB_User class. I have overdone myself documenting all the code, so I won’t explain it in depth anymore. If you are considering using this code, please replace all references to CB_application with your own code.

/**
 * CB_Auth_Adapter_Chap class
 *
 * @author Tom Van Herreweghe
 */
 
class CB_Auth_Adapter_Chap implements Zend_Auth_Adapter_Interface {
    private $_username = '';
    private $_signature = '';
    private $_challenge = '';
 
    private $_data = null;
 
    /**
     * Set the username
     *
     * @param string $username
     */
    public function setUsername($username){
        $this->_username = $username;
        return $this;
    }
 
    /**
     * Set the challenge, used to calculate the signature
     *
     * @param string $challenge
     */
    public function setChallenge($challenge){
        $this->_challenge = $challenge;
        return $this;
    }
 
    /**
     * Set the client-side calculated signature
     *
     * @param string $signature
     */
    public function setSignature($signature){
        $this->_signature = $signature;
        return $this;
    }
 
    /**
     * Authenticate the user
     *
     * @return Zend_Auth_Result
     */
    public function authenticate(){
        // First, we need to retrieve the data, associated with the given username.
        // The password will be decrypted for later use:
        $table = new CB_Table_Users();
        $select = $table->select()->from(CB_application::getInstance()->getTablePrefix("cb") . "users",array("id","username","email",CB_application::getInstance()->getDbAdapter()->quoteInto("AES_DECRYPT(password,?) AS password",CB_application::getInstance()->getKey())));
        $select->where("username = ?",$this->_username)->where("active = 1");
        $result = $table->fetchAll($select);
 
        $authenticated = false;
        // If we found a match, verify the other credentials
        if (count($result) == 1){
            $password = $result[0]->password;
            $this->_data = $result[0]->toArray(); // store the database data locally
 
            // Compare calculated signature with given signature:
            $calcSignature = md5(md5($password) . $this->_challenge);
 
            // if the given signature and the calculated signature match, the user is authenticated:
            $authenticated = ($calcSignature == $this->_signature);
        }
        // We prefill the authentication result with a FAILURE
        $authResult = Zend_Auth_Result::FAILURE_UNCATEGORIZED;
        $authMessages = array();
        if ($authenticated){
            // user is authenticated, overwrite the auth result:
            $authResult = Zend_Auth_Result::SUCCESS;
        } else {
            // Couldn't authenticate the user, set a message:
            $authMessages[] = 'Login failed';
        }
        // return the result:
        return new Zend_Auth_Result( $authResult, $this->_username, $authMessages );
    }
 
    /**
     * Mimics (actually almost copies) the behaviour of the getResultRowObjec() from Zend_Auth_Adapter_DbTable
     *
     * @param array $returnColumns
     * @param array $omitColumns
     * @return stdClass
     */
    public function getResultRowObject($returnColumns = null, $omitColumns = null){
        // If no data is set, return false:
        if (!$this->_data) {
            return false;
        }
 
        $returnObject = new stdClass();
 
        if (null !== $returnColumns) {
            // Process only the specified columns:
 
            $availableColumns = array_keys($this->_data);
            foreach ( (array) $returnColumns as $returnColumn) {
                if (in_array($returnColumn, $availableColumns)) {
                    $returnObject->{$returnColumn} = $this->_data[$returnColumn];
                }
            }
            return $returnObject;
 
        } elseif (null !== $omitColumns) {
            // Process all columns, except the ones specified:
 
            $omitColumns = (array) $omitColumns;
            foreach ($this->_data as $resultColumn => $resultValue) {
                if (!in_array($resultColumn, $omitColumns)) {
                    $returnObject->{$resultColumn} = $resultValue;
                }
            }
            return $returnObject;
 
        } else {
            // Process all columns, without restrictions:
 
            foreach ($this->_data as $resultColumn => $resultValue) {
                $returnObject->{$resultColumn} = $resultValue;
            }
            return $returnObject;
        }
    }
}

The puzzle is almost complete now. Let’s put the pieces together in the CB_User class:

/**
 * CB_User class
 *
 * @author Tom Van Herreweghe
 */
 
class CB_User {
    private $data = array();
 
    /**
     * Getter magical function to get properties
     *
     * @param string $key
     * @return mixed
     */
    public function __get($key){
        if (array_key_exists($key,$this->data)){
            return $this->data[$key];
        }
        throw new Exception("Unknown property '{$key}'");
    }
 
    /**
     * Setter magical function to set properties
     *
     * @param string $key
     * @param mixed $value
     */
    public function __set($key,$value){
        $allowed = array("id","username","email");
        if (in_array($key,$allowed)){
            $this->data[$key] = $value;
        }
    }
 
    /**
     * Authenticate a user via credentials
     * The amount of parameters is variable, and are therefore not explicitly declared
     *
     * @return Zend_Auth_Result
     */
    public static function authenticate(){
        // Get all the arguments:
        $args = func_get_args();
        // Initialize an authAdapter, but set it to null
        $authAdapter = null;
 
        if (count($args) == 2){
            // 2 arguments: We assume they are username & password. This means regular authentication
            $authAdapter = new Zend_Auth_Adapter_DbTable(
                CB_application::getInstance()->getDbAdapter(), // This returns an instance of a database adapter
                CB_application::getInstance()->getTablePrefix("cb").'users', // this will return a prefix, defined in a config.
                'username',
                'password'
            );
            // set the necessary credentials
            $authAdapter
                ->setIdentity($args[0])
                ->setCredential($args[1])
                ->setCredentialTreatment("AES_ENCRYPT(?, '".CB_application::getInstance()->getKey()."') AND active = 1");
                // CredentialTreatment says we'll use AES_Encrypt to encrypt the given password, and the user must be active.
                // CB_application::getInstance()->getKey() returns a predefined key from a config file.
        }
        if (count($args) == 3){
            // 3 arguments: We assume username, challenge and signature. This means custom digest authentication
 
            // Here, we use our custom created CB_Auth_Adapter_DbTable:
            $authAdapter = new CB_Auth_Adapter_Chap(
                    CB_application::getInstance()->getDbAdapter(),
                    CB_application::getInstance()->getTablePrefix("cb").'users',
                    'username',
                    'password'
            );
            // set the necessary credentials:
            $authAdapter
                ->setUsername($args[0])
                ->setChallenge($args[1])
                ->setSignature($args[2]);
        }
 
        if (is_null($authAdapter)){
            throw new Exception("No authAdapter specified. Please check all the arguments");
        }
 
        // Zend_Auth will now try to authenticate, using our auth adapter:
        $result = Zend_Auth::getInstance()->authenticate($authAdapter);
 
        if($result->isValid()) {
            /*
             * When authentication is succesfull, we will write an instance of CB_User to the storage
             * Later on, when we want to know which user is authenticated, we will retrieve this CB_User from the storage
             * This all hapens quite transparently via regular Zend Framework code
             */
 
            // get the resulting line from the database, except the "password" column:
            $data = $authAdapter->getResultRowObject(null,'password');
            // retrieve an an instance of CB_User via the user id:
            $user = self::getInstance($data->id);
            // write our CB_User object to the storage of Zend_Auth
            // Zend_Auth::getInstance()->getIdentity() will return that CB_User object
            Zend_Auth::getInstance()->getStorage()->write($user);
        }
        // Return the Zend_Auth result:
        return $result;
    }
 
    /**
     * Return an instance of the CB_User class, via the user_id
     *
     * @param int $user_id
     * @return CB_User
     */
    public static function getInstance($user_id){
        $table = new CB_Table_Users();
        $select = $table->select()->from(CB_application::getInstance()->getTablePrefix("cb") . "users",array("id","username","email"));
        $select->where("id = ?",$user_id)->where("active = 1");
        $result = $table->fetchRow($select);
 
        if (!is_null($result)){
            return self::factory($result);
        }
        throw new Exception("No user found");
    }
 
    /**
     * Transform the raw data from the database into a CB_User object
     *
     * @param Zend_Db_Table_Row $data
     * @return CB_User
     */
    public static function factory($data){
        if (is_a($data,"Zend_Db_Table_Row")){
            $user = new CB_User();
            $user->id       = $data->id;
            $user->username = $data->username;
            $user->email    = $data->email;
            return $user;
        }
        throw new Exception("Wrong datatype for the CB_User::factory() method");
    }
}

Now that we have written our classes, we want to use them. Mind you, this is just a demonstration. We will first log the user in via the CHAP method, then we do a logout, and then we log the user in via the regular method.:

class IndexController extends Zend_Controller_Action{
    public function indexAction() {
        $this->getHelper("Layout")->disableLayout();
        $this->getHelper("ViewRenderer")->setNoRender();
 
        // Verify if we aren't logged in yet
        if (Zend_Auth::getInstance()->hasIdentity()){
            // Oops, we were still logged in from a previous demonstration..
            // Let's log out:
            Zend_Auth::getInstance()->clearIdentity();
        }
        // Alright, at this point we're not logged in.. Let's proceed:
 
        $challenge = 'challenge'; // <- this should be a random string
        $password = 'password';
        // Calculate the signature:
        $signature = md5(md5($password) . $challenge);
        // Authenticate via CHAP:
        $result = CB_User::authenticate('test',$challenge,$signature);
        // Show the result
        print_r(Zend_Auth::getInstance()->getIdentity());
 
        // Logout again for another demonstration:
        Zend_Auth::getInstance()->clearIdentity();
        // Lets try the other authentication method:
        $result = CB_User::authenticate('test',$password);
        print_r(Zend_Auth::getInstance()->getIdentity());
        die("demonstration is over !");
        }
}

That’s it for now. We have completed a first part of a framework: create a User class, and allow a user to authenticate. Right now, we can’t do much with it yet, but it will come in handy. We still need a way of registering users, but most importantly, we have to determine what that user can do now that he’s logged in. I will write about that in another blog post about Zend_ACL.

Security with Zend_AMF and Flex – Part 2: Practise

Tom — Wed, 15 Apr 2009 21:58:45 +0000

In my previous post “Security with Zend_AMF and Flex – Part 1: Theory“, I explained the theory behind securing your Flex-PHP calls. After the theory comes the practise. I will only provide snippets for the PHP side of this story, as I’m totally ignorant about Flex and ActionScript. I used Zend_AMF, written by Wade Arnold, to handle all the communications between Flex and PHP. If you need to know the basics, please read the documentation first.

Project structure

For the project, I used a slightly modified Conventional Modular layout. The basics remains the same, but I have added support for so called versions of the application. The layout I maintain is not really important for the project. However, at the end of the article, you’ll be able to download everything, and I thought it necessary to explain the structure first. All code is part of the default module, and all classes are located in the “models” directory.

Application layout

The remoting endpoint

Flex and AIR need to connect to an endpoint. A URL you provide, the point of access to your API. I have created only 1 endpoint. Ideally, you should make 2 endpoints: one for logging in, and one for the rest of the calls. That way, in the future, you could do the login process via an SSL secured line. The remoting endpoint is located in the IndexController.php, since that file is the door to the outside world for the project.

	// Explicitly loading dependencies
	Zend_Loader::loadClass("Proxies_AMF_System");
	Zend_Loader::loadClass("Proxies_AMF_Login");
	Zend_Loader::loadClass("AMF_VO_Error");
	Zend_Loader::loadClass("AMF_VO_Message");
 
	class IndexController extends Zend_Controller_Action{
		/**
		 * Everything to handle the rest of the API requests
		 */
		public function indexAction() {
			$this->getHelper("Layout")->disableLayout();
			$this->getHelper("ViewRenderer")->setNoRender();
 
			$server = new Zend_Amf_Server();
			$server->addDirectory(dirname(__FILE__) . "../models/");
			$server->setClassMap("MessageVO","AMF_VO_Message");
			$server->setClassMap("ErrorVO","AMF_VO_Error");
			$server->setProduction(false);
			$response = $server->handle();
			echo $response;
		}
	}

Basically, the indexAction is where my endpoints is. There I set up the Zend_AMF server. After setting it up, the server is ready to listen to requests. All requests are handled by the classes from the folder specified with addDirectory(). In our case, the Proxy classes will handle all the requests. After specifying the directory of the classes, we set up the class mapping. Here, we tell Zend_AMF, how the mapping should be between our PHP classes on the server side, and the Flex classes on the client side. In my case, all Flex classes end with “VO”. Once the mapping is complete, our server is ready to give a response to the client. Note that I explicitly load every class in IndexController.php. For some reason, Zend_AMF isn’t able to use autoloading, which is a too bad.

Responses to the client

To have a cleaner interface, I have opted to return structured responses to the Flex client. When the client makes a call, there are only 2 kind of responses he can expect. When the call is successful, the response will be an object of the type AMF_VO_Message that’s being mapped to a MessageVO in Flex. Or it will be an object of the type AMF_VO_Error, mapped to ErrorVO, when an exception is thrown. This is very easy for the client side coder. All other data is encapsulated in the AMF_VO_Message object, and that makes it easy to process everything. It’s also easier for the PHP developer, since it is now possible to return a lot more information to the client. If you decide to use caching, then the AMF_VO_Message response can contain information about that; timestamps; transaction id’s; … Anything you would expect an API to return.

    /**
     * This class is used to wrap data for output to the AMF adapter
     *
     */
    class AMF_VO_Message {
        // Properties:
        public $status		= null; // string
        public $timestamp	= null; // string
	public $transaction	= null; // string
	public $data		= null; // mixed
	public $cachehit	= false; // boolean
    }

The class is basically a collection of public properties, where I can add data that is useful for the client, like timestamps, transaction-id’s, …

    /**
     * This class is used to wrap data for output to the AMF adapter
     *
     */
    class AMF_VO_Error {
        // Properties:
        public $timestamp	= null; // string
        public $errorCode	= null; // string
        public $file		= null; // string
        public $linenumber	= null; // integer
        public $messages	= null; // string
    }

Again, some properties that are useful for error tracking.

First phase: login security

In the first part of this series, I explained how securing the login process works. To initialize a login procedure, the client first needs to notify the server that an authentication attempt will be made. This is done by means of the challenge() function from the Proxies_AMF_Login class. Remember, all available calls are exposed via my so called Proxy classes. This is not mandatory, but a convention I want to impose on the users of the API I’m going to write. In Flex, the front-end developer can specify the class (proxy) and method (API call) to invoke.

When a client request a challenge from the server, I don’t necessarily want to know. Therefore, I can keep this function clean and simple. All code is explained with in-line comments.

/**
 * Login proxy class to expose classes to AMF
 *
 */
class Proxies_AMF_Login extends Proxies_AMF {
        /**
         * Start the authentication session, by requesting a challenge.
         * The client then uses this challenge to calculate a signature (or digest)
         * When signing on, the username & signature is used, instead of the password
         *
         * @return AMF_VO_Message
         */
        public function challenge(){
		$arguments = func_get_args();
		return $this->processRequest(__FUNCTION__,$arguments);
	}
 
        protected function _challenge(){
            // 1. generate the challenge
            $challengeString = AMF_Login::generateChallenge();
            // 2. remember the challenge !
            $amfNamespace = new Zend_Session_Namespace('AMF_Auth');
            $amfNamespace->challenge = $challengeString;
            // 3. return the challenge to the client
            return $challengeString;
	}
 
        /**
         * Authenticate the user
         *
         * @param string $username
         * @param string $signature
         * @return bool|AMF_VO_Error
         */
        public function signOn($username,$signature){
			$arguments = func_get_args();
			return $this->processRequest(__FUNCTION__,$arguments);
	}
 
	protected function _signOn($username,$signature){
            // 1. Digg up the challenge from our memory:
            $amfNamespace = new Zend_Session_Namespace('AMF_Auth');
            $challenge = $amfNamespace->challenge;
 
            // 2. Verify if we can authenticate the user
            if (AMF_Login::signOn($username,$challenge,$signature)){
            	// User authenticated, let's forget the challenge
                unset($amfNamespace->challenge);
                return true;
            } else {
            	// Failed to authenticate the user
            	unset($amfNamespace->challenge);
		Zend_Auth::getInstance()->clearIdentity();
                throw new Exception("Could not authenticate the user");
            }
	}
}

A challenge() call is logically followed by a signOn() request (verifySignature() will be explained later on) to complete the login process. As you can see, for both challenge() and signOn(), there is a protected counterpart _challenge() and _signOn(). This has to do with the processRequest() call in both of the methods. More about that call later on, but for now, you only need to know that processRequest() does some administrative tasks, and wraps the responses for Zend_AMF server in a structured response.

An attentive reader will have noticed that I make use of an AMF_Login class, and I forgot to explain what and how. No wories, it’s not very mystical. I have just created that class to bundle some functionality. However, it does use some functionality from classes I have not written myself. They fall under the copyright of my employer, and I don’t want to infringe on that. So I won’t show you how AMF_User is written, but from the comments, it should be obvious what the purpose is. In the downloadable source, you’ll find a simplified version of AMF_User, which doesn’t really do anything, except illustrating the purpose.

class AMF_Login {
	/**
	 * Returns a random string, that is used as Challenge for the sign-on
	 * procedure
	 *
	 * @return string
	 */
	public static function generateChallenge(){
		return uniqid(rand()+time(),true);
	}
 
	public static function signOn($username,$challenge,$signature){
		$user = new AMF_User();
		$result = $user->authenticate($username, $challenge, $signature);
 
		return $result;
	}
 
	/**
	 * Method to verify if each signature for an API call is valid
	 */
	public static function verifySignature($challenge,$signature){
		// 1. get the password from the session (it was put there during authentication)
		$amfNamespace = new Zend_Session_Namespace('AMF_Auth');
 
		// 2. calculate signature with given $challenge
		$calcSignature = md5(md5($amfNamespace->password) . $challenge);
 
		// 3. compare calculated signature, with given signature
		if ($calcSignature == $signature){
			return true;
		}
		throw new Exception("Incorrect signature given");
		return false;
	}
}

This concludes the first part of the security measures: logging in. By now, a user can be logged in, and stay logged in, by means of the browser session.

Second phase: API call security

In the second phase, I want to verify all API calls. What we need, is a way to process every API call, and do a number of actions for each of those call. I want to be able to:

Log what calls were made, by what user, what arguments, and what was the response
Have the possibility to cache certain calls
Use Zend_ACL to verify if a certain user has access to certain functionality
Create my structured response
Trap errors (yes, they should be trapped in the ErrorController.php, but I haven’t fully investigated that option yet)
Write code once, use it multiple times

As you can see, there are plenty of requirements. There is one option to fulfil them all: Make sure all calls invoke 1 function that handles all the “administration”. The perfect solution exists, and is named __call() in PHP5. If a method does not exist in a class, then the __call() magic function is called with the exact same arguments. There, you can create all functionality for the administration, and then request another function that will do the real work. I know, quite the abstract explanation, but I won’t go any further on that road because using __call() fails. Zend_AMF uses Zend_Reflection, a PHP reflection class. Simply put, a reflection calls detects all publicly accessible methods and properties of a given class. Since __call() is a magic function, it isn’t publicly accessible. Hence, it won’t be discovered by Zend_Reflection, and consequently, Zend_AMF will not be able to make use of your functionality if you rely on __call(). That took me a while to discover, but if you think about it, it’s quite logical.

To work around this problem, I have decided to create my own __call() method, and I have named it processRequest(). This will do the exact same as I have explained in the paragraph above: handle the administration (logging, caching, …). The diagram below will immediately shine a light on the working.

Diagram: handle requests

A request comes in via helloWorld(). It then travels to processRequest() for the administrative tasks. Next it travels to _helloWorld() that will do what needs to be done and give a response. The response then will travel back from _helloWorld() to processRequest(). There, the response is packed in the structured response, and then that structured response is sent back to helloWorld(). There, it is delivered to the Zend_AMF server, and ultimately, to the Flex client.

To write the code only once and use it many times, the only option is to place the processRequest() function in the parent of our Proxy classes: Proxies_AMF.php. Below, you will find a simplified version of the code I have written for a recent project. Everything is explained in in-line comments.

/**
 * AMF proxy class to expose classes to AMF
 *
 */
class Proxies_AMF {
	// Some classes do not need verification, like Login. Only the other API calls should be verified
	private $exemptVerification = array("Proxies_AMF_Login");
 
	protected function processRequest($method,$arrArguments){
		try {
			$className = get_class($this);
 
			// Retrieve the parameters to verify the identiy of the requesting party:
			if (!in_array($className,$this->exemptVerification)){
				$challenge = array_shift($arrArguments);
				$signature = array_shift($arrArguments);
			}
 
			// Verify the given signature, before executing the call, unless the request is to a method from an exempt class:
			if (in_array($className,$this->exemptVerification) || AMF_Login::verifySignature($challenge,$signature)){
				// Verify if the requested method actually exists:
				if (method_exists($this,"_{$method}")){
					$cachehit = false; // support for caching could be added, but not now.
					$returnValue = call_user_func_array(array($this,"_{$method}"),$arrArguments);
 
					// wrap the response of the API call in an AMF_VO_Message object, and return it:
					return $this->wrapOutput($returnValue,$signature,$cachehit);
				} else {
					throw new Exception("The method '{$method}' does not exist.");
				}
			} else {
				// An exception is already thrown in AMF::Login::verifySignature()
			}
		} catch (Exception $e){
			$errorVO = new AMF_VO_Error();
 
			// Get the currently logged in user
			if (Zend_Auth::getInstance()->hasIdentity()) {
				$user = Zend_Auth::getInstance()->getIdentity();
				$errorVO->user_id = $user->user_id;
			} else {
				$errorVO->user_id = 0;
			}
			$errorVO->timestamp		= strftime("%Y-%m-%d %H:%M:%S");
			$errorVO->errorCode		= get_class($e); // usually, this will be "Exception", unless we introduce custom Exceptions
			$errorVO->file			= $e->getFile();
			$errorVO->linenumber		= $e->getLine();
			$errorVO->messages		= array($e->getMessage());
			$errorVO->stacktrace		= debug_backtrace();
 
			// return response
			return $errorVO;
		}
	}
 
	private function wrapOutput($data,$transaction,$cachehit){
		$message				= new AMF_VO_Message();
		$message->status		= "success";
		$message->timestamp	= strftime("%Y-%m-%d %H:%M:%S");
		$message->transaction	= (!is_null($transaction))?$transaction:"no_transaction";
		$message->data		= $data;
		$message->cachehit		= $cachehit;
 
		return $message;
	}
}

The function processRequest() is now very basic. It only incorporates authentication of the requester, and catching errors. I will not implement the rest of the requirements during this article, since that would lead me outside the scope of this article. If you feel like implementing extra functionality for all your calls, then processRequest() is the method to elaborate on. For a business project, I have extended it with everything mentioned in my list earlier.

The hard part is over now. Most of the functionality has been written. Each API call will be accompanied by 2 extra parameters: a client-chosen challenge, and a calculated signature. The given challenge and signature are just the first two parameters of each of our service calls. These parameters will be verified in processRequest().

The actual verification is done by the verifySignature() method in AMF_Login class. To make it easy, below is a copy of that method again. The parameters are the challenge, chosen by the client, and the signature, calculated by the client. The method will then use that same challenge, and the password stored in a session object, and then calculate the accompanying signature. If both signatures match, then we have a winner! If they don’t match, then somebody is trying to tamper with the requests. It’s up to you how you deal with that. I just output an AMF_VO_Error object.

/**
 * Method to verify if each signature for an API call is valid
 */
public static function verifySignature($challenge,$signature){
	// 1. get the password from the session (it was put there during authentication)
	$amfNamespace = new Zend_Session_Namespace('AMF_Auth');
 
	// 2. calculate signature with given $challenge
	$calcSignature = md5(md5($amfNamespace->password) . $challenge);
 
	// 3. compare calculated signature, with given signature
	if ($calcSignature == $signature){
		return true;
	}
	throw new Exception("Incorrect signature given");
	return false;
}

With this last functionality explained, I think I have everything covered. Only thing left, is giving you an example on how our processRequest() method is used:

/**
 * System proxy class to expose classes to AMF
 *
 */
class Proxies_AMF_System extends Proxies_AMF {
	/**
	 * Say hello to the world
	 *
	 * @return string
	 */
	public function helloWorld(){
		$args = func_get_args();
		return $this->processRequest(__FUNCTION__,$args);
	}
 
	protected function _helloWorld(){
		return "Hello World !!";
	}
}

I mentioned the helloWorld() function earlier, and here is the implementation. The Flex client calls the helloWorld() function (without the underscore !!), and gives as paremeters the challenge and signature. You’ll see that those parameters aren’t explicitly mentioned in the function definition, but that is no problem. In that function, processRequest() is called, which will do the verificiation and then call _helloWorld(). The response from _helloWorld() will then travel back, over processRequest(), via helloWorld(), to Zend_AMF, stopping at the Flex client.

Ammendments

This article is the fruit of a quite lengthy development process at the company I work at. During this period, I have made many mistakes regarding Zend_AMF, and learned from them. To illustrate that development is a continuous learning process, I will end with a “mistake” I just recently discovered:

When I first implemented these security measures, I used the URL to transport the challenge and signature for each API call. This way, could keep my interfaces clean, without having to take into account the challenge and signature. In Flex, I had the front-end developer create a dynamical endpoint for each call. Each endpoint connection only made one call, since in the URL, the challenge and signature were added. However, when an endpoint connection is established, it is standard procedure of AMF to ping the server once. As a result, each call generated a new ping request, effectively doubling the amount of calls. I don’t have to explain that this is quite a big overhead. To overcome this, I had to extend each API call to accept the challenge and signature as a parameter of that call. And the result of my mistake is that you can enjoy a “better” implementation than what I had 2 days ago.

What I want to say: This method is not fool proof, and I have no doubt that there are probably many ways of improving the security. If you happen to have suggestions, by all means, let me know. I’m happy to learn from you.

Download all files: zip | tar
Credits where credits are due: Thanks Vic Rau for the small UnitTest (yes, it is also in the downloadable archive)

Security with Zend_AMF and Flex – Part 1: Theory

Tom — Sun, 05 Apr 2009 18:24:46 +0000

In a series of two posts, I will explain how to secure the communication between a Flex client and PHP server architecture. The first part will explain how I envision that security, and in the second part I will show snippets of PHP code for the practical implementation.

I’m currently working together with my friend and colleague Vic on a client-server application that involves Flex on the front-side, and PHP (Zend Framework) on the back. Since I’m the PHP guy, I’m in charge of creating the API for his Flex application. For the moment, the project will only be accessed locally from the client’s network. But there is a possibility that in a later stage, it might open up to the general public. One of my main concerns was how we could make every API call as secure as possible. This without making it too complicated, or involve too many service calls that might slow everything down.

First login…

The first aspect of securing the application, is authentication of users. The application is accessible to multiple users, and those are grouped in various groups. Each group has different privileges. To identify every user, they are given usernames and passwords. My biggest concern with the login process, is that we didn’t have access to a secure connection via SSL. This means that every communication between the client and server software can be monitored by network sniffers. Flex uses the AMF protocol to communicate with Zend_AMF. Even though this is a binary stream, the specifications are open to everyone. So it is possible to intercept communications, and read everything out.

I wanted to make sure that during the login process, the password was not exposed to the outside world via the communications line. Because once the password and username are intercepted, then it would be easy to start making fake API calls to the server. The solution I had in mind, is called CHAP – Challenge-Handshake Authentication Protocol. Here’s a diagram about how this protocol works.

The main purpose of using CHAP, is to authenticate users, without having to transmit the password between the client and the server. To accomplish this, we have to replace the password with something else, but still allowing the server to recognize the correct user, and log him in. I call this password replacer the “signature”.

In the first phase of the diagram, the client requests a “challenge” from the server. A challenge is a key that will be used by both the client and the server, to create the signature. During the first phase, the client asks the server what key will be used to encrypt the password. It is important that both the client and the server use the exact same challenge.

During the second phase, an authentication request is being transmitted. Instead of using the username and password, we use the username and signature. The final signature is calculated like this:

$signature = MD5(MD5($password) . $challenge);

The client calculates this signature, and sends it to the server. The server then looks up the given username in the list of possible users. The corresponding password is then used by the server, to also calculate a signature, with the earlier given challenge. The outcome of that comparison is given during the third and last phase of the CHAP authentication.

This is not a very difficult protocol to implement, but according to me, it still provides a reasonable amount of security during the login process.

… and then the rest of the calls.

Now that a user is logged in, the most important stuff still has to happen: interactivity with the server by calling API methods. Once the user is authenticated, he remains authenticated by using the browser session. So now we have to keep in mind that other people might try to steal the session, and invoke undesired API calls. During each call, the server needs to be sure that it was still the same client making the call, and not some man-in-the-middle.

For this, I decided to elaborate on the CHAP protocol, used for authentication. How about we could keep the system, where the client signs its calls with a signature. For obvious reasons, it is not possible to recycle the signature from the login procedure. But I wanted to force the client to generate a new signature for each API request. There are still two variables to generate the signature: the password and the challenge.

It was not a practical option to ask the user’s password for each and every API call. Instead, once the user was authenticated, I had the Flex application remember the password in its memory. This way, it could be recycled to generate new signatures.

The only problem was the challenge. A first option was to have the client request a challenge from the server. But this would mean that for each regular API call, there would be an additional call for a challenge. That would double the amount of API calls. I was afraid that in the long run, this would generate too much calls for the server. That’s why I chose a second option: have the client generate the challenge and transmit it alongside the signature. The server would then use this challenge, and the password from the session, to calculate the signature. If they matched, then the call was permitted to be executed.

As far as I could see, this wouldn’t impose a bigger security risk. Of course, a man-in-the-middle could generate fake challenges. But he still wouldn’t be able to generate fake signatures. For that, he still needs the correct password. Once he has that, then well… all security checks fail. The responsibility of keeping the password secret lies entirely at the client now.

Garbage collector problem after foreach loop ?

admin — Wed, 01 Apr 2009 09:45:42 +0000

In PHP5, it is possible to loop through an array, and alter each item on the fly via the “by-reference” operator: the &-symbol. However, you should be on the lookout for unexpected behaviour.

First the initial data:

	$items = array(
		array(
			"field"		=> "field1",
			"value"		=> "value1",
			"options"	=> array(
				"operator"	=> "=",
				"logic"		=> "OR"
			)
		),
		array(
			"field"		=> "field2",
			"value"		=> "value2",
			"options"	=> array(
				"operator"	=> "=",
				"logic"		=> "OR"
			)
		),
		array(
			"field"		=> "field3",
			"value"		=> "value3",
			"options"	=> array(
				"operator"	=> "=",
				"logic"		=> "OR"
			)
		),
		array(
			"field"		=> "field4",
			"value"		=> "value4",
			"options"	=> array(
				"operator"	=> "=",
				"logic"		=> "OR"
			)
		)
	);

It is a simple array. It could be an array of where-clauses you wish to define for your own search module (like this example), or it could just simply be test data for a unit test.

In my case, this data comes into a function, and in that function I want to strip some of the data. The “logic” index under “options” is deprecated, and I want to remove it. To make my code compact, I have decided to alter each filter on the fly, by using the by-reference operator.

	foreach($items as &$item){
		unset($item["options"]["logic"]);
	}
 
	print_r($items);

This results in the following altered version of my initial data:

Array
(
    [0] => Array
        (
            [field] => field1
            [value] => value1
            [options] => Array
                (
                    [operator] => =
                )
 
        )
 
    [1] => Array
        (
            [field] => field2
            [value] => value2
            [options] => Array
                (
                    [operator] => =
                )
 
        )
 
    [2] => Array
        (
            [field] => field3
            [value] => value3
            [options] => Array
                (
                    [operator] => =
                )
 
        )
 
    [3] => Array
        (
            [field] => field4
            [value] => value4
            [options] => Array
                (
                    [operator] => =
                )
 
        )
 
)

As you can see, I have successfully removed the “logic” index by using an ampersand in my foreach loop. This is potentially very useful functionality. It’s very easy to manipulate your data like this. No more fiddling around with indexes, or even create temporary arrays to house your data. Just edit it on the fly. There is however a (big) flaw, and I’ll illustrate it below.

In the previous loop, I have altered my array, and now I want to loop through it again, and just output each item in the array. Nothing more. But things go awry quite unexpectantly.

	$counter = 1;
	foreach($items as $item){
		echo "Item {$counter}: ";
		print_r($item);
		$counter++;
	}

The output of this loop is:

Item 1:
 
Array
(
    [field] => field1
    [value] => value1
    [options] => Array
        (
            [operator] => =
        )
 
)
 
Item 2:
 
Array
(
    [field] => field2
    [value] => value2
    [options] => Array
        (
            [operator] => =
        )
 
)
 
Item 3:
 
Array
(
    [field] => field3
    [value] => value3
    [options] => Array
        (
            [operator] => =
        )
 
)
 
Item 4:
 
Array
(
    [field] => field3
    [value] => value3
    [options] => Array
        (
            [operator] => =
        )
 
)

The last output isn’t what I expected it to be. As you can see, it is the same output as the previous item “3″.

I have encountered this problem in production code, and only by rigorously outputting all my data, I have discovered that this was my problem. I am not a PHP expert by far. So I don’t always know how things technically work. My guess is that after a foreach loop, with the by-reference operator, the garbage collector doesn’t do its job well. But I’m not sure this is the correct explanation. I have found a workaround though. After the first foreach, where you use the & symbol, you just unset that variable. Like this:

	foreach($items as &$item){
		unset($item["options"]["logic"]);
	}
	unset($item);

Now the output should be what you expect it to be. Just to be sure, here’s what I got:

// ...
Item 4:
 
Array
(
    [field] => field4
    [value] => value4
    [options] => Array
        (
            [operator] => =
        )
 
)

If anyone has an idea to whether or not this is a bug, please let me know. I have so far not filed a bug (yet), as I feel I need to do a bit more research first. It could be something by-design, and then I’m making quite a fool of myself. I guess that’s just part of the learning process