Contact Us
31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39

Running Tests from The Commandline in CakePHP

Posted on 8/5/08 by Tim Koschützki

Hey folks,

I have contributed some new cool feature for you to the cake branch. A testsuite shell that allows you to run your tests from the commandline. It allows you to run all of the following:

  • All core testcases at once
  • All core test groups
  • Each core test case individually
  • All application-specific testcases at once
  • All application-specific test groups
  • Each core test case individually

It also supports plugins, which means you can run plugin cases and groups.

How does it work?

Actually it's very simple. You just call

./cake testsuite category test_type file

Category can either be "core", "app" or the name of a plugin.

The Test Type can be "case", "group" or "all".

The file is the name of the file with the folder prefix, but without the test.php / group.php suffix. Examples are: models/my_model, controllers/my_controller, etc.

Can you show me examples of valid calls to the testshell?

Yes, of course! Although I am going to directly copy this from the help screen from the testshell. Why should I reinvent the wheel? : P

./cake testsuite app all
./cake testsuite core all
./cake testsuite app case behaviors/debuggable
./cake testsuite app case models/my_model
./cake testsuite app case controllers/my_controller
./cake testsuite core case file
./cake testsuite core case router
./cake testsuite core case set
./cake testsuite app group mygroup
./cake testsuite core group acl
./cake testsuite core group socket
./cake testsuite bugs case models/bug  // for the plugin 'bugs' and its test case 'bug'
./cake testsuite bugs group bug  // for the plugin bugs and its test group 'bug'
./cake testsuite bugs_me case models/bug  // for the plugin 'bugs_me' and its test case 'bug'
./cake testsuite bugs_me group bug  // for the plugin bugs_me and its test group 'bug'

What does the output look like?

Welcome to CakePHP v1.2.0.6311 beta Console
App : app
Path: /Applications/MAMP/htdocs/cakephp/branch/app
CakePHP Test Shell
Running core case router
Individual test case: libs/router.test.php
1/1 test cases complete: 190 passes.

Macintosh-6:branch dakangelbge$ 
elcome to CakePHP v1.2.0.6311 beta Console
App : app
Path: /Applications/MAMP/htdocs/cakephp/branch/app
CakePHP Test Shell
Running app all
All App Tests
2/2 test cases complete: 6 passes.

Macintosh-6:branch dakangelbge$

Got any questions?

PS: There is some hidden feature in it, which I am not going to reveal until another thing got written about first. ; ) Anybody who knows already please do not spoil the fun. : P


Unit Testing in CakePHP Part 1 - Introduction to Unit Testing

Posted on 6/5/08 by Tim Koschützki

So you want to read up on Unit Testing in CakePHP? That is great, testing can be such a help in finding bugs. It's a shame that so few clients dedicate a budget to it and then expect their application that contains a ton of complicated code to be stable nevertheless.

Surprisingly, many people do not know yet what unit testing is. In this first part of a whole series you can get a good grasp of what it is and is not.

1.1 Introduction to Unit Testing

According to Wikipedia, unit testing is an automated procedure to ensure your software units work properly. Units are the smallest testable parts of an application. Meaning, they are procedures or functions in procedural programming and object methods in object oriented programming. Classes are also commonly referred to as being the units of the system. Quite a few JUnit (the first testing framework of the XUnit family - made for Java) folks do not like the term "Unit Testing" at all, because they think it is overused. They like to refer to Unit Testing as "Programmer Testing". No, we are not testing programmer here. But programmers are testing their own source code. To me it does not matter. Seriously, I could not care less. No matter what you think a unit is, you will understand what I am talking about here. For the sake of clarity, I will refer to it as "Unit Testing".

The idea behind unit testing is that once you have a unit that you think works, you set up a testcase where you specify some input to the unit and compare the result of your unit with your expected result. You know about the expected result, because you know what your unit is doing (or you should know it). Well, you better know what your unit is supposed to do, or else you should not do programming in the first place... ; ) Then you run the tests and the CakePHP/«insert your framework here» testsuite tells you if they passed or if not, with some graceful message where the error occured.

Now you have that testcase. Now you add testcases for t h a t input and this one. The advantage of this is that once you wrote the tests down they are there (cool, huh?) and you can hold on to them. There is no need anymore for you to open the browser and test everything manually again when you change your system. Instead, you add functionality, run the automated unit tests again, if they pass you are good to go, if they don't pass you broke something. Well, what if you broke something but your tests don't catch it? That's something that UT cannot do for you. You must make sure you have a good test coverage ( for UT to work very well for you. However, having not enough tests is still much better than having no tests, so....

Typically, the order of the running of the tests should not matter. There might be special cases, but in well over 90% it does not. This should also be your goal, too, to have two different problems if two test cases fail. Keep them all isolated and you will sleep well. For most tests there is also not much configuration to be done. You specify the input, your expected result, crank the handle and evaluate how well you have done. You should typically be able to group tests together, too. When you run these groups you can get a good overview over large components of your system.

If you use a framework - like CakePHP - you should also be able to run all tests to see how well your system works.

Unit Testing also tells you when you are done with your work. If you get a green bar (ie all tests are passing) you are done (except if there are more tests to add, heh). If you get a red bar, you got work to do. Simple.

1.2 So How Does It Work?

Unit Testing generally works with assertions that take two parameters - at least most of them do. The first one is your known expected result and the second one is the output of your unit based on your input. The testsuite then compares the expected result with the output of your unit. The CakePHP Testsuite in its current state strictly relies on and thereby inherits SimpleTest's assertions. Let's have a look at them:

So, with the custom assert() function, mentioned last in the table, you can build your own assertions keeping your code really clean. The others should be straightforward. If not, head on over to and read the documentation there.

When any of these assertions fail, you will be presented an error message telling you which test failed, on which line and what the error is. What if you want to supply your own error messages - for example to better mark an often failing test? You can do that in simpletest pretty easily. Just append your custom message as the last parameter to the assertion:

$this->assertTrue(1, 'This should pass');

You will notice that it replaces the automatic error message. If you want to embed your custom message within the automatic one, use %s:

$this->assertTrue(1, 'This should pass: %s');

1.2 A quick SimpleTest example

Okay so before we are going to jump onto the cake test wagon let's look at a real world simpletest example. That way it will be easier for us to deal with cake's testsuite later.

So here is the problem:

You want to calculate the costs for a given trip (to a cakefest). You get: $flightPrice, $hotelCosts and expenses for food and drinks ($otherExpenses). You know you will need more money since gwoo (the president of the Cake Software foundation) will come along persuading you to do sling shot (and yes we had a lot of fun at Cakefest in Orlando :]) although you are already a little drunk. So we take on a 10% buffer that we add to the sum. Our total expenses are:

Formula: $total = 110% of ($flightPrice + $hotelCosts + $otherExpenses)

So, we might end up with:

require_once('simpletest/unit_tester.php'); // install simpletest into /vendors after you downloaded it from

define('BUFFER_RATE', 0.1);
function calcTravelExpenses($flightPrice, $hotelCosts, $otherExpenses) {
  $sum = $flightPrice + $hotelCosts + $otherExpenses;
  $sum = (BUFFER_RATE) * $sum;

class TestOfTravelExpenses extends UnitTestCase {
  function testExceededTravelExpenses() {
    $this->assertEqual(1320, calcTravelExpenses(600, 400, 200), 'Okay so this is a pretty exciting test %s message end');

$test = &new TestOfTravelExpenses();
$test->run(new HtmlReporter());

When you run this you see you get a red bar with the message that "Integer" differs from "NUll". So 1320 is obviously an integer and the output of our function is NULL. Uh oh, okay, so there is no return statement! Let's add it:

function calcTravelExpenses($flightPrice, $hotelCosts, $otherExpenses) {
  $sum = $flightPrice + $hotelCosts + $otherExpenses;
  $sum = (BUFFER_RATE) * $sum;
  return $sum; // here

Now the value is not quite right. Ah okay, because we are taking 10% of the sum instead of 110%:

$sum = (1+BUFFER_RATE) * $sum;

Green bar! Now that's a great feeling right? We can do all sorts of crazy refactorings now to our little function and still we will see if it works or not. And all of that automagically! Wohoo! Right, so what are the benefits of all this?

1.3 Benefits of Unit Testing

1. Unit Testing allows us to refactor our code later with real confidence. If we break something and have plenty of tests there should be a test failing, we fix it up and the entire system should work again (that means all units independently of each other). Great!

2. It forces you to think about what your code is really supposed to do. You finally get rid of the script kiddy attitude that hacks something together fast and does not even lose a second on thinking if his unit even returns an integer or if there could not be a division by zero error. I am not sure how it happens, but you will think about the edge cases, the most important cases, automatically. It's so difficult to think about them without automated tests, but with UT you approach the problem from a more theoretical and mathematical side.

3. Automatic testing is faster than browser testing. Period.

4. UT provides living documentation. If you develop in a team with several people and someone wrote a unit without phpdoc comments and you have no idea what it does, you can frequent the tests for it. Understanding code from tests is ultra easy and fast and most of the time better than any discussion. You could even picture the code in your head (if you are smart enough) just looking at the tests.

5. It helps you separate interface from implementation. Code that uses your code works with your interface, that is the unit name and the parameter signature and your return type. If you don't change that you can do what you want in the unit implementation. It's so great because it gives you so much freedom. Also as long as the original tests still run, you ensure backwards compatibility of your code, too.

6. Make the CakePHP Core Team happy. Many people submit tickets to CakePHP to make us aware of bugs. That's a good thing. However, if more people submitted tests alongside their bugreports that would help even more.

There are plenty of other benefits..

1.4 Limitations of Unit Testing

1. UT as such does not prove there are no errors in your system. It is not a theoretical / mathematical prove that a particular unit is bug free. Yes it shows the presence of errors if you have good and enough tests. However, it does not show the absence of errors.

2. Besides that, unit testing tests the units of your system independently of each other (for the most part). So if your system suffers from performance problems or integration problems, unit testing will not catch them. Also it will not prove that your system is not vulnerable to any security attack from the outside.

3. What's more? Well, many people don't implement unit testing yet, because it takes a rigorous amount of discipline to do it consistently. Especially in a team environment with clients paying, tight deadlines and all sorts of other interruptions, testing seems to be the first thing to cut on. However, in the long term it will prove to be much more productive to do it. The drawback stays, though: You write extra code that does not add any "real" features to your application. Don't get me wrong - I love to do it, but I can partly understand the clients, too. Would be cool if you could throw in your two bits on this one.

Conclusion: Do it! No I mean try it out. And if you only implement ten tests in your application, you are still much better off than without any tests.

1.5 Test-driven Development

So, when do you write the tests? From what you have read so far, you must have the idea that you write the tests after you wrote the code. However, with that "interface-over-implementation" and that "think-about-it-before-you-start-it" benefits, wouldn't it be cool to write the tests before you write the code? Test-driven Development - also known as TDD - does that.

Essentially TDD is a software development technique consisting of short iterations where new tests covering the desired improvement or new functionality are written first. Then you implement the production code necessary to pass the tests. TDD helps a lot, to make your code design nice and to accommodate changes you refactor.

The availability of tests before actual development ensures rapid feedback after any change. Remember that green-bar-means-you-are-done and red-bar-means-work? TDD ultimately boils down to those.

TDD is actually a method of designing your software instead of testing it. Why? Because with TDD you REALLY think about the stuff you do before you do it. If you have a problem you cannot tackle, you write the simplest possible test, make it work, and go from there. As you write more tests and as you think about how people should use your code (ie, what your interface shall look like), you design your software well automatically. Yeah, it's not always that easy, but you get the idea.

In TDD you have a rough cycle that should take just minutes, if not seconds:

Add a test -> see it failing -> make it work -> see the green bar -> refactor -> still see the green bar - > add a test -> see it failing -> ...

1.6 Mock Objects

Mock Objects are objects that simulate real objects that would normally be difficult to construct or time consuming to set up for a test. The most common use of mock objects is the mocking of a database connection object.

Setting up a test database at the start of each test would slow testing to a crawl and would require the installation of the database engine and test data on the test machine. If the connection can be simulated you can return data of your choosing. By that you not only win on the pragmatics of testing, but can also feed your code spurious data to see how it responds. You could simulate databases being down or other extremes without having to create a broken database for real. In other words, you get greater control of the test environment.

However, the mock objects not only play a part (by supplying chosen return values on demand) they are also sensitive to the messages sent to them (via expectations). By setting expected parameters for a method call they act as a guard that the calls upon them are made correctly. If expectations are not met they save you the effort of writing a failed test assertion by performing that duty on our behalf.

This can be very useful, because with standard Unit Testing you just test the interfaces of your objects. Mock Objects give you a means to test the inner implementation of them.

1.7 Resources and Further Reading

Process is at least as important as tools. The type of process that makes the heaviest use of a developer's testing tool is of course Extreme Programming. You should also read about Agile Methodologies in general. If you want to read up on test-driven Development, please do so here.

The original Unit Test Framework was JUnit. Most people writing their own test tools seem to be cloning it in one way or the other. PHP Unit is the XUnit ambassador for the PHP world. Together with Simpletest it forms the top testing framework in the php world.

Wrap Up

By now you should have an idea of what you can get out of Unit Testing and automated tests in general. I hope I have raised some questions or made controversial statements. As we discuss this article, some good things can be added to it I am sure.

In the next part of the series we will look at Cake's Testsuite.


FixturesShell - Share your test data

Posted on 3/5/08 by Felix Geisendörfer

Hey folks,

for some of the projects Tim and I recently got involved with I noticed that I needed a way to share a set of test data with the other people working on the project. The result is a simple shell leveraging CakePHP 1.2's fixtures. Usually those are used for unit testing your Models and Behaviors, but they also do a great job at sharing a common set of data within a project.

So if you're interested to try it, check out the project page and let me know what you think.

-- Felix Geisendörfer aka the_undefined


XHTML is a joke

Posted on 1/5/08 by Felix Geisendörfer

Hey folks,

I don't know how often I had to discuss this with people over the past months, so let me say this once and for all:

XHTML is a joke. It is an elitist trend against the very nature of the web and offers no advantages over HTML 4.01 strict!

This is an old old discussion and there are great resources that list all of the problems associated with using XHTML. All I want to address in this post are two problems I have with XHTML:

1) Strict rendering is a STUPID idea

I don't know if I am the only one that realizes this. But if all your favourite web sites were to be converted to XHTML strict most of them would be unusable half of the time because of tiny errors in views. XHTML promotes that the slightest mistake in your markup will cause your page to NOT RENDER!

To me this is the most ridiculous concept ever. The view / presentation is the least important layer in any application and XHTML says it should be able to completely crash your page. Give me a break, but this is madness.

[Edit: 2008-05-02] When I say that the view is the least important layer than I mean that it should be most the forgiving layer because it does no business critical data processing.

Publishing content on the web should be easy for everybody. I am against anything raising the bar in that regard. Of course I wish MySpace never happened, but millions of people absolutely love the idea of expressing themselves in the most ridiculous ways. Heck, if I never had gotten into the web industry you can bet that I would have a 5mb MySpace page that brings any browser to its knees.

So let us come up with better standards, but please make them forgiving about mistakes within reasonable boundaries. The main reason we all use PHP (or any other dynamically typed language) is that we hate strict boundaries. We love flexibility in any form, and I strongly believe that flexible and forgiving standards are far superior in terms of enabling innovation compared to strict standards like XHTML.

2) Presentation agnostic XHTML is an ILLUSION

The biggest advantage people see in XHTML is that it is supposedly better at separating content from presentation. That is BS. XHTML pretty much supports the same set elements as HTML, including strictly presentational ones. But more importantly: If you ever tried to completely change the design of an XHTML page I know that you cheated and changed the markup.

[Edit: 2008-05-02] The above is meant in the context of real world situations with dynamic content. CSS Zen Garden and similar sites are not realistic web environments. See comment below.

So if you are under the impression that by using XHTML your pages automatically become presentation agnostic data sources that can be aggregated you are hallucinating. Microformats and web scraping is what you should look into.

Conclusion: I think XHTML did a ton of good for the web. It gave people a way to draw a line between traditional table driven HTML layouts and modern XHTML / CSS approaches. It spawned a wave of support for semantic markup and web standards. But looking at XHTML as a format by itself I see no advantages in it. I have big hopes for HTML5. Meanwhile I feel very happy with the feature set and philosophy provided by HTML 4.01 and will continue to use it.

Looking forward to feedback on this.

-- Felix Geisendörfer aka the_undefined


Code Coverage Analysis soon in CakePHP - Test How Well You Test

Posted on 27/4/08 by Tim Koschützki

Hey folks,

the last week I have been dealing with an implementation of code coverage analysis for the upcoming release 1.2 of CakePHP. For those of you who do not yet know what code coverage is, here is a short excerpt from Wikipedia:

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.

There are several different kinds of criteria to code coverage. The two most important ones are line coverage (or statement coverage as wikipedia puts it) and path coverage. Line coverage basically analyzes if each line of the subject-under-test has been executed during the test. Path coverage examines if every possible route through the code has been executed. Since I rely on Xdebug for the implementation we are talking line coverage here.

So how is it going to work? Pretty simple actually. Whenever you run a CakePHP test case Cake assembles information in the background about which lines of your subject-under-test are called. Then it will display all code lines from the subject-under-test and mark those that were "covered" (ie executed by the testcase) and which were not. Lines that play no role for the code coverage percentage, such as multiline comments, will be displayed in a different color with less contrast.

How could an example report look like? Here you go:

I have also implemented a diff-style report. It just lists all the lines that are not covered by your tests and then seven lines before and after each of those. With that view you can really focus on the uncovered lines, but see enough lines around them in order to be able to write a reasonable test case to get the uncovered lines covered.

This is what the diff view looks like for the cakephp core router.test.php:

Within the dev team we are currently discussing these two views. If you favor one of these, can recommend another one or just want to voice your opinion on the looks of them, please do so by commenting on this post.

For your information: Code coverage analysis currently only works for separate test cases only. However, the plan for this week is to get aggregate code coverage working. This means that your group tests are analyzed and the coverage percentage is displayed for each of the test subjects involved.

This is a very valuable tool when developing tests for your cakephp application. Although we are talking line coverage here and not path coverage it is still a pretty good indicator of how thorough your tests are. It might as well encourage some more people to start writing tests if they see a neat report after they have done so. :]

-- Tim Koschuetzki aka DarkAngelBGE

31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39
Assertion What it does Example
assertTrue($x) Fail if $x is false
$this->assertTrue(1 == true);
assertFalse($x) Fail if $x is true
$this->assertFalse(1 === true);
assertNull($x) Fail if $x is set
$variable = null;

assertNotNull($x) Fail if $x not set
$variable = 'something but not null';

assertIsA($x, $t) Fail if $x is not the class or type $t
$tim = new Person;
  $this->assertIsA($tim, 'Person');
assertNotA($x, $t) Fail if $x is of the class or type $t
$tim = new Person;
  $this->assertNotA($tim, 'Animal'); // or maybe I am? o_O
assertEqual($x, $y) Fail if $x == $y is false
$fahrenheit = 50;
  $celsius = (5/9)*($fahrenheit-32);
  $this->assertEqual(10, $celsius);
assertNotEqual($x, $y) Fail if $x == $y is true
$this->assertNotEqual('5', 5); // fails, they are equal but not identical
assertWithinMargin($x, $y, $m) Fail if abs($x - $y) < $m is false
$this->assertWithinMargin(10, 50, 60); // passes

  $this->assertWithinMargin(10, 50, 30); // fails
assertOutsideMargin($x, $y, $m) Fail if abs($x - $y) < $m is true
$this->assertOutsideMargin(10, 50, 60); // fails
  $this->assertOutsideMargin(10, 50, 30); // true
assertIdentical($x, $y) Fail if $x == $y is false or a type mismatch
$this->assertIdentical(0, false); // will fail since 0 is not false
assertNotIdentical($x, $y) Fail if $x == $y is true and types match
$this->assertIdentical(10, 100/10*5 - 40);
assertReference($x, $y) Fail unless $x and $y are the same variable
$a = 5;
  $b =& $a;
  $this->assertReference($a, $b);
assertClone($x, $y) Fail unless $x and $y are identical copies, that means they are identical but not referenced
$a = 5;
  $b = 5;
  $this->assertClone($a, $b); // passes, 5 equals 5, but $a and $b are not references to each other
assertPattern($p, $x) Fail unless the regex $p matches $x
$this->assertPattern('/hello/i', 'Hello world');
assertNoPattern($p, $x) Fail if the regex $p matches $x
$this->assertNoPattern('/heppo/i', 'Hello world'); // passes
expectError($x) Swallows any upcoming matching error

assert($e) Fail on failed expectation object $e; use SimpleTest's built in expectation objects to have fun here
class TestOfNetworking extends UnitTestCase {

      function assertValidIp($ip, $message = '%s') {

          $this->assert(new ValidIp(), $ip, $message); // uses validIp expectation

      function testGetValidIp() {

          $server = &new Server();



                  'Server IP address->%s');