Bug description
http://projects.blender.org/tracker/index.php?func=detail&aid=27510&group_id=9&atid=498

Patch revision: 36986
Tested in Ubuntu 10.10 64

Hi, could you clean this patch up a bit?, it seems to include unrelated changes.
I think the logic for this can be done with only minimal edits 2-5 lines.

I'll do it tomorrow, I just submitted my report to our professor.

I added lots of comments, I should remove it.

I thought It should be simpler too,

but it came up this complex logic probably because I have not enough programming experience

New Patch

Index: source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c
===================================================================
--- source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c (revision 37009)
+++ source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c (working copy)
@@ -55,7 +55,9 @@

VECCOPY(out, in);

- if(fabs(in[0]-c->key[0]) < c->t1 &&
+ if(( ( (c->key[0] + c->t1 * 0.5f >= 1 ) && (( 1 - in[0] <= (c->t1*0.5f + 1 - c->key[0])) || (in[0] <= (c->t1 * 0.5f - 1 + c->key[0]))) ) ||
+ ( (c->key[0] - c->t1 * 0.5f <= 0 ) && ((in[0] - c->key[0] - c->t1 *0.5f <= 0 ) || ((1 - in[0]) <= (c->t1 * 0.5f - c->key[0]))) ) ||
+ fabs(in[0]-c->key[0]) <= c->t1) &&
fabs(in[1]-c->key[1]) < c->t2 &&
fabs(in[2]-c->key[2]) < c->t3)
{

I can't use this system to upload file,

It said

"double submit" error.

Sorry about paste the content here

Thanks Tom for your fix, but I looked into this and found a simpler change, committed r37051.

Index: source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c
===================================================================
--- source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c (revision 37030)
+++ source/blender/nodes/intern/CMP_nodes/CMP_colorMatte.c (working copy)
@@ -49,16 +49,24 @@

static void do_color_key(bNode *node, float *out, float *in)
{
+ float h_wrap;
NodeChroma *c;
c=node->storage;


VECCOPY(out, in);

- if(fabs(in[0]-c->key[0]) < c->t1 &&
- fabs(in[1]-c->key[1]) < c->t2 &&
- fabs(in[2]-c->key[2]) < c->t3)
- {
+ if(
+ /* do hue last because it needs to wrap, and does some more checks */
+
+ /* sat */ (fabs(in[1]-c->key[1]) < c->t2) &&
+ /* val */ (fabs(in[2]-c->key[2]) < c->t3) &&
+
+ /* multiply by 2 because it wraps on both sides of the hue,
+ * otherwise 0.5 would key all hue's */
+
+ /* hue */ ((h_wrap= 2.0f * fabs(in[0]-c->key[0])) < c->t1 || (2.0f - h_wrap) < c->t1)
+ ) {
out[3]=0.0; /*make transparent*/
}

thanks, I hope mine provided clue for your patch.
I forgot there is a fabs(),
this one is better.