It seems its not down to being the second call in a row, but both deselect=True and toggle=True do not work when used in combination with extend=True...

I found out when using toggle=True and extend=False it has the behaviour that is expected when extend would be True (old selection remains), so toggle=True and extend=True are two mutually exclusive settings, so also when setting extend=True, the deselect and toggle settings are not used.

Better behaviour should be allowing deselect and toggle to be used with/without extend...

If you set all the parameters in both cases:

bpy.ops.view3d.select(extend=True, deselect=False, toggle=False, location=(mx, my))
bpy.ops.view3d.select(extend=True, deselect=True, toggle=False, location=(mx, my))

Does that work?

It doesnt work that way either...

Behaviour is :
when extend=True, it always SELECTS, even if deselect or toggle are on
when (deselect=True or toggle=True) and extend=False, it will respectively deselect and toggle, with behaviour as if extend is on...

This is an old function that was converted to an operator that has internal properties exposed... but that didn't make the operator design itself to be good or better.
It's probably a direct mapping of the C code internally.

To me this isnt a pressing matter anymore, since I figured out how it DOES work and it does everything needed.
It could confuse other people that first give it a try though...

The main confusing thing here is that not all combinations of parameters make sense. Saying that you want to extend the selection and deselect at the same is not logical, it can only do one of them.

So is what is the bug here, that it doesn't give any warning/error message about invalid parameter combinations? Should the documentation be clarified?
http://www.blender.org/documentation/blender_python_api_2_68_2/bpy.ops.view3d.html?highlight=bpy.ops.view3d.select#bpy.ops.view3d.select

I wouldn't change the behavior because that's going to break custom key configurations, and this is not a good enough reason in my opinion.

No, the problem is that when extend=True it ignores deselect=True and toggle=True and always does a plain "select".
Those combinations do make a lot of sense and should work differently. This behaviour had me confused for quite some time...

How should they work differently, what do you expect extend=True and deselect=True together to do different than either extend=True or deselect=True individually?

They shouldnt (work differently) but for consistency they should do the same. One reads in documentation extend=True extends the selection instead of deselecting everything first, then one reads about the behaviour of deselect and toggle, and expects these two keywords to work together.
When you expect these two to work together, you dont think: since deselect and extend=False doesnt mean anything, it will do the same and this shouldnt be, and thus extend=True will not work together with deselect/toggle. No, you expect extend+toggle to extend the selection (instead of deselecting all ) AND toggling the element. If you start of with this assumption, you are wildly confused when extend=True always defaults to selection and think something is broken about this operator. Consider as proof this bug, and how I first formulated it, I thought the operator didnt work when used two times in a row, because it didnt toggle as expected. If its important for compatibilities sake to hold on to the old behaviour, at least the user should be informed of his choice, being eg. extend=True and toggle=True together is invalid, or documentation should clearly reflect extend=True will always set selection state to 1, as it is this is not clear.

Agree with Brecht here, the properties are mutually exclusive and should not be used together.
Campbell mentioned today that a better solution would be to have an enum instead of using bools.

That would break things so I propose the attached patch (select_mutual).
It would still break scripts. But only in cases where script writers were wrongly using the mutual exclusive options).

Thoughts?