debuggable

 
Contact Us
 

Unlimited Model fields - Expandable Behavior

Posted on 1/6/08 by Felix Geisendörfer

Hey folks,

lets say you have a model called Upload. Your Upload model has some generic fields like this:

The problem

However, you would like to store different types of meta information for different kinds of uploads. Examples:

Images:

  • Width
  • Height
  • Quality (if jpg)
  • Camera
  • Lens
  • Focus

Videos:

  • FPS
  • Bitrate

PDFs:

  • Author
  • Description

ZIPs:

  • Original Size
  • File Count
  • Compression Rate

The solution

So what are you going to do? Add 13 fields to your uploads table? Probably not. It is time to normalize things:

Ok nothing fancy so far. CakePHP's associations make it easy to deal with it. However, working with this setup can be a little inconvenient at times. Everytime you fetch a set of records from Upload, you will have to manually extract the meta information from the associated UploadField records:

-
    Upload:
        id: 1
        name: funny.mov
        type: video/quicktime
        bytes: 20480
        created: 2008-06-01 14:47:23
    UploadField:
        -
            id: 1
            upload_id: 1
            key: fps
            val: 26
        -
            id: 2
            upload_id: 1
            key: bitrate
            val: 376

So everytime you want to access your videos bitrate you will have to search your UploadField records for the 'bitrate' key. How annoying. But worry not, Expandable comes to rescue. With the Expandable behavior activated on your Upload model, your resultset will look like this:

-
    Upload:
        id: 1
        name: funny.mov
        type: video/quicktime
        bytes: 20480
        created: 2008-06-01 14:47:23
        fps: 26
        bitrate: 376
    UploadField:
        -
            id: 1
            upload_id: 1
            key: fps
            val: 26
        -
            id: 2
            upload_id: 1
            key: bitrate
            val: 376

But it comes even better. Expandable also makes it dead-simple to create / update UploadField records. This is how it works:

$this->Upload->save(array(
  'id' => 1,
  'fps' => 30,
  'rating'= > 7/10,
));

Without you having to do anything, the following happens to your uploads resultset:

-
    Upload:
        id: 1
        name: funny.mov
        type: video/quicktime
        bytes: 20480
        created: 2008-06-01 14:47:23
        fps: 30
        bitrate: 376
        rating: 0.7
    UploadField:
        -
            id: 1
            upload_id: 1
            key: fps
            val: 30
        -
            id: 2
            upload_id: 1
            key: bitrate
            val: 376
        -
            id: 3
            upload_id: 1
            key: rating
            val: 0.7

As you can see the fps UploadField value has been updated and a new record with the key rating has been created. So this means you can use the CakePHP form helper to create different editors for your uploads like this:

$form->input('Upload.fps')
$form->input('Upload.bitrate')
$form->input('Upload.rating', array('options' => array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)));

And even so none of those fields really exist on the Upload model, everything will work just as if they would ; ).

How to use it

  1. Download the behavior from: the debuggable Scraps repository at github
  2. Place the expandable.php file into /app/models/behaviors/expandable.php
  3. Optional: Place the expandable.test.php file into /app/tests/cases/behaviors/expandable.test.php
  4. Create a table and model for UploadField with at least the fields shown above. (replace Upload with the name of your base model)
  5. Setup a Upload hasMany UploadField and UploadField belongsTo Upload association
  6. Add this to your Upload model:
class Upload extends AppModel{
  var $actsAs = array('Expandable');
}

That is it. You are ready to go. Enjoy the magic ; ).

Pro Contra
  • Easy to use
  • Saves db space
  • Mostly leverages existing CakePHP magic
  • (Small) Performance hit while updating meta data

Please let me know what you think about this approach!

-- Felix Geisendörfer aka the_undefined

 
&nsbp;

You can skip to the end and add a comment.

nao  said on Jun 01, 2008:

I was confronted to similar problem and my first idea was to create more specific table which inherit main table, the all manage be a behavior.
But to complex to build for me!

Your solution is more flexible, so it accommodate me ^^

Useful contribution! Thanks!

GreyCells  said on Jun 01, 2008:

I like the concept, Felix, but the implementation appears to leave you a little exposed. The standard model validation would deal with many of the value issues, but only if the field is specified (you'd have to set allowEmpty for all expandable fields). So the conditional mandatory (i.e. if it is file type x, then fields a,b and c are required) validation would get a little interesting.

Of greater concern though is the ability to 'inject' a whole load of expandable fields direct from a form - i.e. if the fields are not specified in the base model, then they are automagically added to the expandable model - not sure this is a good thing. Whitelisting might help, but perhaps a mandatory 'expandableFields' array in the behavior setup would be better?

An alternative (for fields that will never be required in a where clause) is to serialize the expandable fields into a single text field - also eminently doable with a bit of cake magic.

Daniel Hofstetter said on Jun 01, 2008:

Hm, why don't you create a model for each type (PDF, Image, etc), each with a hasOne association to Upload (would probably need a different name in this new context)?

Felix Geisendörfer said on Jun 01, 2008:

GreyCells: Set up all your different rules for different upload types in an array and then apply the one you are interested in to Model::validates in the beforeValidate callback based on the data[Upload][type] key. I will add a key to limit fields at some point, or do you have a patch for me ; )?

Daniel: Simplicity and Flexibility. This behavior helped to simplify a 14 model app to a ~7 model one which had a very positive impact on the simplicity of things ; ).

leo  said on Jun 01, 2008:

Hey Felix,
think there was too much copy & paste with the test. ;) the class should be named ExpandableBehaviorTest instead of ContainableBehaviorTest, shouldn't it?

Thanks anyway!

Felix Geisendörfer said on Jun 01, 2008:

leo: There is no such thing as too much copy & paste ; p. Anyway, thanks for catching this - fixed.

GreyCells  said on Jun 01, 2008:

@Felix - yes, that would be the approach I would expect to use - despite all the advances in the model validation in 1.2, I still seem to have an awful lot of stuff in Model::before/validates()... (that's where I put a lot of my business validation) :)

Untested (shock, horror!), but this is what I had in mind:

--- expandable.php.felix 2008-06-01 19:20:58.000000000 +0100
+++ expandable.php 2008-06-01 19:18:57.000000000 +0100

@@ -75,21 +75,25 @@

$fields = array_diff_key($Model->data[$Model->alias], $schema);

$id = $Model->id;

foreach ($fields as $key => $val) {

- $field = $Model->{$with}->find('first', array(

- 'fields' => array($with.'.id'),

- 'conditions' => array($with.'.'.$foreignKey => $id, $with.'.key' => $key),

- 'recursive' => -1,

- ));

- $Model->{$with}->create(false);

- if ($field) {

- $Model->{$with}->set('id', $field[$with]['id']);

- } else {

- $Model->{$with}->set(array($foreignKey => $id, 'key' => $key));

- }

- $Model->{$with}->set('val', $val);

- $Model->{$with}->save();

+ if ( !isset($expandableFields)

+ || in_array($key, $expandableFields)

+ ) {

+ $field = $Model->{$with}->find('first', array(

+ 'fields' => array($with.'.id'),

+ 'conditions' => array($with.'.'.$foreignKey => $id, $with.'.key' => $key),

+ 'recursive' => -1,

+ ));

+ $Model->{$with}->create(false);

+ if ($field) {

+ $Model->{$with}->set('id', $field[$with]['id']);

+ } else {

+ $Model->{$with}->set(array($foreignKey => $id, 'key' => $key));

+ }

+ $Model->{$with}->set('val', $val);

+ $Model->{$with}->save();

+ }

}

}

}

i.e. if there is an array called 'expandableFields' (or an empty array?) in the behavior settings, then the field name must exist in that array, otherwise continue as before.

Brian D. said on Jun 01, 2008:

I've used this approach before (just not with CakePHP) and in retrospect I think that, unless it's absolutely necessary, it's usually VBI ("Very Bad Idea"). This approach works well with some cases, but it's definitely one of those approaches that one must use caution in implementation.

There's a great thread on this on the NYPHP mailing list (I asked a question about it) if anyone is interested in reading other opinions. The thread can be read here:

http://www.mail-archive.com/talk@lists.nyphp.org/msg02145.html

Ramon said on Jun 01, 2008:

It's like an EAV data model but with two tables.
http://en.wikipedia.org/wiki/Entity-Attribute-Value_model

Felix: does it delete the meta records automatically when you delete the entity?

Thanks

Felix Geisendörfer said on Jun 01, 2008:

Brian: As I said it works excellent for the scenario I'm describing in this post as well as for storing meta information for different types of Pages in a CMS I've worked on. But you are right, using this in the wrong context will not do you much good ; ).

Ramon: If you use foreign keys and have them setup that way, yes. If you mark the association as 'dependent' in CakePHP, yes. Otherwise, no. : )

Btw. does anybody like the pretty pictures and stuff?

Lars Strojny said on Jun 01, 2008:

One simple thing: consider using a join table ("UploadFieldKeys") between the upload table and the UploadField table. The join table would consist of an int primary key and a name. The UploadField table would be modified so that the key field is an integer and references the UploadFieldKeys-table to avoid key duplication.

Daniel Hofstetter said on Jun 02, 2008:

@Felix: Sure, your solution requires less models and is quite flexible. On the other hand: do you really need this flexibility? And is it worth to move "data logic" (e.g. that a PDF has author and description properties) from the database layer to the application layer?

And regarding your question about the images, I like them (with the exception of the first one, which I don't understand)

Christoph Tavan  said on Jun 02, 2008:

Felix, I think I don't have to repeat that I love you for that expandable behavior ;)

@Others: I agree that in many cases it might be against the principles of MVC or other noble concepts. But in some cases it's just soooooo damn f***in useful!
Take an example: You have a CMS that you sell to your clients which you always custom-fit to the needs of each client. Some will want to add videos to their websites, others PDFs, again others Word, Excel, whatever... Some months later a client wants to be able to handle additional filetypes. You don't even have to touch the DB, just add a new controller method and you're done. Isn't that convenient for you as a programmer?

thomas  said on Jun 02, 2008:

hi felix,

and thanks for this. I was using an 'extra data' approach to deal with special cases for categories when I found your Behavior. It helps a lot but ...

Seems like the behavior is not triggered when the model that has the 'extra data' is not populated with this extra data when it is part of an association.

Setting 'recursive' to 2 or more won't change anything.

Is it expectad behavior ?

Thanks in advance

thomas

Nils.r said on Jun 02, 2008:

This will definately come in handy in the next project! Had something alike in my head, but didn't yet have the time to think it out.

Thanks for saving my time! :D

Khaled  said on Jun 02, 2008:

Yeah the pictures are great , What is the program that you used to create them??

I saw expandable behavior two months ago, but I didn't pay attention to it ... or maybe I didn't understand it because there was no enough documentation !!! but with this post things changed it's pretty easy & effective .... thank you Felix ...

Felix Geisendörfer said on Jun 02, 2008:

Khaled: OmniGraffle 5

Where did you see the behavior two month ago? I don't recall publishing it anywhere up until now ; p.

Khaled  said on Jun 02, 2008:

maybe in the bakery ... or it's something similar, but I remember that the title looks like yours "~Expandable Behavior" ... or maybe I'm wrong ... you know how many things you get when you're surfing ;-)

Khaled  said on Jun 02, 2008:

BTW, nice application "OmniGraffle 5" , but I don't have MAC :( ... I know some similar programs but not as power as this app ...

Thank Felix

Dirk said on Jun 03, 2008:

I have a very similar approach to this. But My "UploadField" Table has 2 more fields: serialize and unserialize in which i store the name of a php-function executed as you would expect, to save even array-data and/or complex structures in the blob-field. This way, it is even more expandable.

I also created the modelclass in runtime (because they are pretty predictable in structure), so i do not need to create them manually.

Great Work, though. I really like your implementation.

Alexander  said on Jun 06, 2008:

Hey Felix,
thank you for that behaviour!! It's exactly what I need for my project, and what caused so much headaches on how maintaining such variable models...

Regards,
Alexander

Lucian Lature  said on Aug 07, 2008:

Hi Felix!...

There is a bug inside your code.
More precisely, you should modify the line $conventions = array('foreignKey', $Model->hasMany[$settings['with']]['foreignKey']); from setup() function into $conventions = array('foreignKey' => $Model->hasMany[$settings['with']]['foreignKey']);

That way it will be no problem using this behavior when you're using the "with" variabile inside your model's Expandable behavior, like var $actsAs = array ('Expandable' => array ('with' => 'Chichi'));

Brad  said on Sep 19, 2008:

Hi,

I would like to try the Expandable behavior, but the it doesn't seem to be there anymore. Is there another link?

Thanks

Felix Geisendörfer said on Sep 19, 2008:

Brad: Sorry about that : ). The behavior can now be found here: http://github.com/felixge/debuggable-scraps/tree/master/cakephp/behaviors/expandable

I also updated the link in the post. Thanks for reporting this issue!

Martin Westin  said on Sep 23, 2008:

Hi Felix,
Just ran into a link to this post today. I must say your behavior looks nice. I was wondering if there were any plans to support find operations. For me, that is one of the major reasons not to simply serialize things into a "properties" field.

What I mean is something like:
$this->Upload->find('first',array(

'conditions'=>array('Upload.width'=>'400')

));

and this condition would actually be converted to something like:

array('UploadField.key'=>'width', 'UploadField.value'=>'400')

I might make use of the behavior but I would probably have to write a beforeFind to do something like that first. If I go that way I will send you the results but it would be nice to know if something like that is already in the works.

Felix Geisendörfer said on Sep 23, 2008:

Martin Westin: nothing is in the works. Feel free to fork the github project and hack away, I'll gladly pull in any good changes you make!

Anthony  said on Nov 01, 2008:

The expandable behavior on the Github scraps is not correct. It doesn't match the older version I have.
1) note the explicit use of 'file_id' in the afterSave method

2) it doesn't create new Fields only tries to update pre-existing ones

Felix Geisendörfer said on Nov 03, 2008:

Anthony: Do you have a patch to fix this? Or can you email me the version you have?

Felix Geisendörfer said on Nov 04, 2008:

Anthony: Ok, I actually found the previously released version and pushed it to github. Sry for the inconvenience!

This post is too old. We do not allow comments here anymore in order to fight spam. If you have real feedback or questions for the post, please contact us.