I was in the process of creating a custom 3x3 filter for the cmp nodes when I realised that theeths suggestion that I use the PROP_MATRIX flag to help layout the multi array in the form array[3][3] was not working.

I used the following RNA code:

int matrix_dimsize[]= {3, 3};

prop= RNA_def_property(srna, "matrix", PROP_FLOAT, PROP_MATRIX);
RNA_def_property_float_sdna(prop, NULL, "matrix");
RNA_def_property_multi_array(prop, 2, matrix_dimsize);
RNA_def_property_range(prop, -32.0f, 32.0f);
RNA_def_property_ui_text(prop, "Matrix", "Custom 3x3 matrix");
RNA_def_property_update(prop, NC_NODE|NA_EDITED, @rna_Node_update");

This initially displayed as shown in: 3x3_filter.png
After adding the PROP_MATRIX flag: 3x3_nearly.png
With patch: 3x3_fixed.png

Investigation:
Seems that the matrix_dimsize is only passed to find out the memory required and 'ui_item_array' assumes that the array is always square. When calculating the ui, the loop does not iterate enough. For a [3][3] array this only loops through three times.

Patch for 'ui_item_array' function
interface_layout.diff
This loops through again in order to make sure that enough uiDefAutoButR are displayed. E.g 9 instead of 3.

Caveat:
I'm not an experienced coder and I'm using Blender to learn C!! My patch may be naive and may be better implemented by an experienced coder.

RNA layout:
It seems that for the autolayout, buttons need displaying left-to-right. This is why I changed the order of the loops. I think that for an array[x][y] it will display the colums as x and rows as y but if this is important it will need testing.

Notes:
1)
'RNA_def_property_multi_array' function from rna_define.c contains the following note: /* TODO make sure arraylength values are sane */

2)
Other places where PROP_MATRIX is used in rna, however I could not find any use of thes for displaying in the gui.

./blender/makesrna/intern/rna_armature.c:552: prop= RNA_def_property(srna, "armature_matrix", PROP_FLOAT, PROP_MATRIX);
./blender/makesrna/intern/rna_scene.c:541: prop= RNA_def_property(srna, "matrix", PROP_FLOAT, PROP_MATRIX);
./blender/makesrna/intern/rna_object.c:1568: prop= RNA_def_property(srna, "matrix", PROP_FLOAT, PROP_MATRIX);
./blender/makesrna/intern/rna_pose.c:755: prop= RNA_def_property(srna, "constraint_inverse_matrix", PROP_FLOAT, PROP_MATRIX);


From #blendercoders

charlie: I'm having trouble getting the PROP_MATRIX to work properly. I don't think that the dimensions from the array are properly passed to 'ui_item_array' in 'interface_layout.c'. The rna property is as follows: int matrix_dimsize[]= {3, 3}; RNA_def_property_multi_array(prop, 2, matrix_dimsize); See the custom node in the screenshot shoyuld be 3x3 rather than as shown. http://homepage.ntlworld.com/charlie.jolly/blender/3x3_nearly.png
charlie: fix: http://homepage.ntlworld.com/charlie.jolly/blender/3x3_fixed.png
charlie: Updated section of code in 'ui_item_array' in 'interface_layout.c' http://www.pasteall.org/9471/c
matt_e: charlie1: can you put it in the tracker + send mail to ML?
charlie: Probably needs fixing properly!!!