Why combat maneuvers are often worthless

[sparkletwist]

I think we all have had the gut feeling that doing combat maneuvers to achieve some sort of bonus effect is, in many if not most systems, often not as worthwhile as just attacking all the time. So here's some math behind that feeling.

Essentially, to be "worth it," the damage you can do by performing the maneuver has to be greater than (or at least equal to) the amount of damage you could do just attacking twice. So we'll write that as an equation:
(p = base probability of hitting, pm = probability of a successful maneuver, pb = probability of hitting with a bonus granted by a maneuver, d = damage)



If we assume a best-case scenario that a manuever is always successful (i.e., pm = 1), and that upon performing a maneuver you always hit (i.e., pb = 1), then the best case result is that p <= 1 - p, which means that p <= 0.5. So if you have a greater than 50% chance of hitting, it's still not worth it to perform a maneuver, even in this best case scenario. If we assume a somewhat more realistic scenario where your combat maneuver roll is based on your normal attack roll (i.e., pm = p) and you still always hit, then the p's cancel and we end up with 1 <= 1 - p, which means that p <= 0, that is, it's never worthwhile to do a manuever at all. And that's if you always hit after doing one, which most systems don't even guarantee.

A big way to help this sad state of affairs is to let maneuvers cause some bonus damage in addition to just increasing chances of success-- in systems like FATE where greater degrees of success automatically add bonus damage this is automatic, but many other systems don't do this. So here's the new equation:



Assuming that db is just base damage plus some bonus amount z, we can simplify this to:



Returning to the "optimistically realistic" scenario, where pm = p and pb = 1, we get:


So the ratio between the damage increase and the base damage has to exceed the probability of hitting. This makes perfect sense-- all other things being equal, an attack should do double damage or it's not worth it to not attack twice. This also means that if the chances of a hit after a maneuver aren't 100%, or you don't want to double the damage, then there has to be some sort of bonus for doing a maneuver instead of just attacking.

If the system doesn't offer incentives like this... it's just not worth bothering. And, unfortunately, a lot of systems out there don't.

[ooc=But what about aid another?]
So, what about helping someone else? Let's see if it's worth it for player 1 to help player 2.



Optimistically assuming p1m is 1, then player 2's damage increase still has to exceed player 1's total potential damage output:



Aid another, with its buff of 0.1 (a +2 on a d20 adds 10%) is only worthwhile if your total expected damage output is less than 10% of the other guy's. That... isn't likely. And that's why aid another is mostly terrible.
[/ooc]

[Steerpike]

Interesting analysis.  I have no criticism of the math, but I do think the critique neglects context to a certain extent.  I'd maintain that the entire point of combat maneuvers is that they shouldn't be de facto more damaging than a regular attack, but when used tactically in the right situation they allow for a more optimal result (which may or may not be more damage).  Not to say that you're suggesting otherwise, I just think it bears saying.

I'll give an example or two to illustrate my point.  As usual my stand-by system will be Pathfinder because I'm most familiar with it.

Feinting in Pathfinder results in denying a target their Dexterity bonus from their AC.  On the face of it this may not be that useful.  In a one-on-one fight against a medium-armoured opponent in which both combatants are non-rogues, it's of minimal utility.  Likewise in a fight against a horde of enemies where allies are all attacking other bad-guys.  But if you're leading a group of 1st level rogues (or something) against a small number of heavily armoured foes, feint becomes hugely useful because now all of your allies can hit more easily and get sneak attacks.  It's still of considerable utility if you outnumber your enemy significantly and/or if your foe is heavily armoured, because you're decreasing the enemy's AC not only for your next attack but for everyone's.

Another example: Bull Rush.  Useless in most contexts, pivotal when you're fighting on the side of a cliff with a pit of acid/lava/flesh-eating monstrosities below.  Most of the time it's going to do pretty much nothing and won't be worth it, but when full immersion in lava deals 20d6 points of fire damage per round with no save, it becomes hugely effective.

Then take Disarm and Sunder, another classic combat manuever.  Pretty useless against most foes, but against those who draw much of their power from using a particular weapon - kensai, specialists, those with magical weapons, wand-users - they become much, much more useful.

Then there are all those situations where there are important goals other than dealing damage but that still take place in combat.  Say you need to subdue a foe rather than kill them (for questioning, kidnapping, etc) - or you need to steal an item from their person but don't want to harm them - or you need to stall them until reinforcements arrive - or you're performing for a crowd in an arena and you want to humiliate your opponent - or your goal is to get your enemy to surrender rather than slaughter them - or you're running away and you'd rather confuse/delay your opponent than try and kill them.  These are the situations when combat maneuvers become worthwhile.

In other words, I think context is key, and a strict mathematical analysis, though helpful when comparing the results to a default attack, doesn't necessarily invalidate a combat maneuver.  Most combat maneuvers shouldn't be a standard go-to in combat, I think; they should be employed tactically in key moments when appropriate.  Otherwise the reverse becomes true to an extent - if combat maneuvers give you a reliable way of dealing more damage regardless of context, why bother with a regular attack at all?

EDIT: Indeed, I would suggest that combat manuevers should be "often worthless," with the proviso that they should also be "occasionally pivitol."

[Lmns Crn]

Steerpike has said part of what I was trying to formulate, that context is key with what makes these things worthwhile (or not worthwhile).

The other idea I have is that the standard way a lot of systems have traditionally handled these types of maneuvers is pretty boring. And tied into that is the problem of fighters (or whatever melee combat specialists your system has) often being boring as well. I guess I'm talking about D&D/Pathfinder and their derivatives here.

I think what I'd like to see a virtuosic warrior-type character be able to do is, instead of facing the choice "do I attack for damage, or do I go for a non-damaging stun/disarm/trip/feint effect?", to have the chance to marry the two outcomes together.

Maybe the veteran gladiator has a certain number of times per day she can make an attack that, if it hits, has a stun effect in addition to its regular damage. D&D4E does something along these lines, in that a lot of your once/day or once/encounter attacks have secondary effects. You could easily have a once-per-fight attack that trips the opponent and does damage, a once-per-fight attack that dazes the opponent to reduce their attack and does damage, and a once-per-day attack that does extra damage and ruins the enemy's armor.

Perhaps when the tough old sensei lands an attack, he can spend an Overdrive Point (or whatever) to add the effect of throwing the opponent prone, or knocking the opponent across the street and into a fruit vendor's cart. That way, once the player successfully lands a hit, the option opens up: conserve my points, or use them to buy a special effect (but which one?)-- but damage isn't part of the tradeoff.

Maybe for the cruel duelist with the cruel blade, it's not a resource-driven system at all. Instead, each time the duelist hits, the opponent takes an ongoing -1 penalty to defense for the rest of the fight (so in a one-on-one fight, the duelist benefits from a slow and cautious battle of attrition, and in a large everybody-kill-the-ogre fight, the duelist is doing less damage than the barbarian but applying a debuff they both benefit from.)

[LordVreeg]

It was one of the reasons that most of the maneuvers in GS were done WITH an attack, not in place of.  Whether it be skill-aided attempted combat strategies, or purely skill based combat abilities, they go with normal combat, not instead of.

So an experienced combatant, whether PC or not, Normally has a 'static add on' skills like damage bonus or initiative bonus, but also a few others like 'multiple/chain attack' or "batter' that have a % of success that adds onto an attack, as opposed to being used instead of one.

[Steerpike]

I agree to an extent, LC, but what do you make the plethora of feats in Pathfinder that let fighters combine combat maneuvers and regular attacks?  Improved Feint.  Whirlwind Attack.  Bashing Finish.  Cleave. Greater Drag.  Bleeding Attack.  Parting Shot.  Passing Trick.  Blinding Flash.  Sliding Axe Throw.  There are tons and tons of moves that Fighters, with their abundance of Feats, can cultivate and specialize in till they have a repetoire of advanced moves that combine damage with some bonus ability or move.  Some require damage tradeoffs, some don't, but there has to be a tradeoff or there's no longer any point in a regular attack, unless you resort to the x/day model, which I really, really dislike for fighters (martial encounter powers may be the single biggest failure for me in 4E).  It's a badly dissociated mechanic, that, for me, makes no in-setting sense and shatters verisimiltude.

I'm not saying the Fighter in Pathfinder is perfect - god knows the 3.X power disparity with spellcasters has not been exorcised nearly thoroughly enough (hough mercifully it has been mitigated) - but do the Feats I mentioned and the numerous others like them not qualify?

[Lmns Crn]

I don't have any real familiarity with Pathfinder, so I don't know how those work precisely. Sounds like they're the sort of thing I was talking about, so, pretty cool that that already exists.

There are tons and tons of moves that Fighters, with their abundance of Feats, can cultivate and specialize in till they have a repetoire of advanced moves that combine damage with some bonus ability or move.  Some require damage tradeoffs, some don't, but there has to be a tradeoff or there's no longer any point in a regular attack,

In a sense, there's a tradeoff in that each feat has an opportunity cost associated with it. Maybe if I build my fighter with a certain highly restrictive combination of feats, it's fine if there's no point in ever making a regular attack again, because making special non-regular attacks is what I've specialized to do. Perhaps that type of character makes my playstyle more powerful and more satisfying, even if it doesn't give me a more interesting breadth of tactical options.

One discussion that I want to have at some point (although this is probably not going to be the best thread for it) is the interplay between the use of resources at character creation and the use of resources in play (that is to say, how choices made during character creation affect the array of available tactical choices, and the different currencies used in both situations).

[Steerpike]

Maybe if I build my fighter with a certain highly restrictive combination of feats, it's fine if there's no point in ever making a regular attack again, because making special non-regular attacks is what I've specialized to do.

That's true, and there are builds that make this so (particularly multishot-type builds or those structured around things like Power Attack).

Circling back to the more general topic as opposed to Pathfinder-specific things, I guess what I see as the most pertinent question isn't "Should combat maneuvers be 'cost-effective' damage-wise in relation to default attacks?" but "What is the purpose of combat maneuvers?"  It seems to me there're a lot of different answers to this question, some of which include:

1) To provide a repetoire of alternate ways of dealing damage.
2) To relieve the potential monotony of combat.
3) To provide a tactical advantage in specific combat contexts.
4) To accomplish things in combat that don't relate directly to damage-dealing.
5) To facilitate cooperation and teamwork between party members.

(and likely many more)

I'd suggest that answers 3 and 4 are the most important, followed by 5 and 2, and that 1 is proabably the least important from my point of view; I think the chief value of combat maneuvers is/should be derived from very particular contextual situations where they become appropriate, thus making tactical thinking - properly identifying when and where such mauevers are useful (and when, conversely, they are not useful)- a key part of combat.  If combat maneuvers become universally, statistically viable methods of damage-dealing regardless of context, they would actually flatten the tactical dimension of the game because they'd make different choices essentially meaningless.  Of course one could also imagine combat maneuvers so context-specific they're almost never utilized, which is also no good.  I think the key to designing good combat manuevers is to make them broad enough to be useful but narrow enough to necessitate careful tactical thinking.

EDIT: In this sense, then, I'm specifically challenging sparkletwist's thesis that "Essentially, to be 'worth it,' the damage you can do by performing the maneuver has to be greater than (or at least equal to) the amount of damage you could do just attacking twice," at least as the default state of affairs.

[sparkletwist]

If combat maneuvers become universally, statistically viable methods of damage-dealing regardless of context, they would actually flatten the tactical dimension of the game because they'd make different choices essentially meaningless.

I agree. Combat maneuvers that are almost always useful would be just as big of a game balance problem as combat maneuvers that are almost never useful, because there would be no point in making regular attacks. My issue is that by being often worthless and not really making any effort to rise above this, they are often trap options-- they give a numerical bonus that players are led to believe is beneficial, but is actually hardly ever a tactically smart move.

So let me go back to...

"What is the purpose of combat maneuvers?"

I should also point out (and perhaps clarify) the first sentence of my post: I am talking specifically about combat maneuvers whose main goal is to create an advantage by giving a numerical bonus. Many sorts of aspects in FATE, a feint or dirty trick in Pathfinder, or whatever. That is a large subset of the available combat maneuvers in RPGs, but far from all of them. Situationally appropriate combat maneuvers should be used when situationally appropriate, and if there are roleplay or other factors that make the maneuver worthwhile, then math doesn't even enter into the picture. So, I agree with you on item #4, definitely.

However, I contend that a "tactical advantage" as defined by #3 (and quite possibly #1's "alternate ways of dealing damage," especially if that alternate way involves more damage) is generally modeled in a RPG by giving you bigger numbers to attack, damage, or some other advantageous quality, or taking away numbers from your enemy. So, since it's all based on numbers, a detailed mathematical analysis is actually an excellent way of determining when that maneuver is appropriate or not, because mathematically speaking, a maneuver is clearly either worth or it isn't: it's worth it when the benefit of performing the maneuver outweighs the opportunity cost of whatever you don't get to do. So,  yes, you can say that the context is very important, and I'll agree with you, but in these cases, the context is quite simply whether or not your attack is improved.

Earlier, you said "a strict mathematical analysis... doesn't necessarily invalidate a combat maneuver" but you cited a bunch of cases where a strict mathematical analysis absolutely can tell you whether it's worthwhile to perform that maneuver or not. A group of rogues will get a sneak attack bonus that will or won't be worth the opportunity cost of one character's missed attack. Bull rushing a character off a cliff into lava is worthwhile because the average damage you will do per turn rises dramatically, whereas ordinary bull rushing grants no such huge bonus and is as such not useful. Disarming an enemy is a sizable debuff that affects the amount of damage you will suffer, and this too can be mathematically analyzed.

So, yes, I agree that maneuvers should be employed when appropriate, and I think we've both acknowledged there are times when strict numerical analysis isn't the whole story at all. On the other hand, I'm honestly not sure what your point even is, when your list of examples is full of maneuvers that do benefit from a mathematical analysis.

[Polycarp]

I should also point out (and perhaps clarify) the first sentence of my post: I am talking specifically about combat maneuvers whose main goal is to create an advantage by giving a numerical bonus. Many sorts of aspects in FATE, a feint or dirty trick in Pathfinder, or whatever.

I don't think your model is really complex enough to handle this.  Feint and Sneak Attack might be pretty straightforward, but what about Dirty Trick to lower someone's save for my wizard buddy about to cast Hold Person?  What about demoralizing knowing that my cleric buddy can also hit the target with fear to stack the effect?

I agree that a lot of these can be mathematically analyzed, but I guess I don't really see the point because of how many variables are involved.  If the basic point here is simply to say that "a maneuver isn't worth it unless it is better than a normal attack," then yes, I think we can find broad agreement on that; it's not a terribly controversial proposition.  Most maneuvers in PF, when optimally used, work out with your analysis just fine; Feint is not worth it unless you're sneak attacking, Intimidate is not worth it unless you're stacking fear, Steal is not worth it unless the enemy has a Wand of Killing You in his belt, and so on.

Maneuvers in Pathfinder are sometimes "trap options," but not for the reasons you're talking about.  They're not trap options because they're worse than attacking; when used optimally, they're usually not, and you could even make that argument for the lowly Dirty Trick.  They're trap options because maneuvers are niche by design, and if a fighter (or anyone else) spends feats on tripping only to be confronted by an adventure full of giants, oozes, and enormous centipedes, he's essentially wasted those feats.  Someone who specializes in demoralizing is going to have a bad time in a world of undead.  In PF, at least, the issue is not really the effectiveness of combat maneuvers so much as how combat maneuvers are accessed and used in the first place, but that's a whole different topic that touches on feat economy and the woes of martial classes in d20.

You could be absolutely right about maneuvers from aspects in FATE, I don't know enough to say.  PF, at least, doesn't seem very vulnerable to your critique.

[Gamer Printshop]

None of the combat maneuvers in Pathfinder are not intended to cause damage at all: grapple is for pinning opponents, while most others are for changing the tactical situation by moving one's opponent around on the battlefield to change the advantage in favor of the PCs. Steal is for taking something away, while sunder is for destroying non-living components. Your argument that attacking twice causes more damage than a single maneuver, I'd argue that a single attack causes more damage, since combat maneuvers, in general, don't cause any damage at all. That's not what they exist for (damage causing). If you want to cause damage use a weapon or cast an attack spell.

[sparkletwist]

I don't think your model is really complex enough to handle this.

Clearly what we need is a more complex model! :grin:
No, seriously, it can't handle everything, but if we remember what the goal is and try to think about things simply, we can usually produce some results that make sense.

what about Dirty Trick to lower someone's save for my wizard buddy about to cast Hold Person?

Since hold person is a save-or-lose, to keep things simple for the mathematical model, we'll consider its 'damage boost' to essentially be equal to 100% of the enemy's hit points. I know that is not strictly accurate and there are flavorful and tactical reasons that make it distinct from an attack, but from a pure numerical analysis, the question is simply "does doing this help to take the enemy out of the fight or not?" so it's not a bad way to go. So the question becomes, is doing a dirty trick to reduce the target's save worth more than making an attack?

Dirty trick imposes a variety of conditions, but the useful ones for this case all are -2 to saves, so that is, generally speaking, a 10% increase to the chances of succeeding at the hold person. When considering expected values, that makes a dirty trick in this combo "worth" 0.1 of a hold person. In other words, it means a dirty trick is essentially worth 10% of taking an enemy completely out of combat. Another way to gain 10% of taking an enemy completely out of combat is, of course, to damage the enemy for 10% of its hit points. The dirty trick + hold person combo is thus only worthwhile if the dirty trickster's average attack damage is less than 10% of an enemy's total hit points. The wizard might want to try the spell regardless, but I'm not going to worry about that part of it. I just want to know if we should attack or do a dirty trick.

Let's assume our dirty trickster is a level 4 fighter with an attack bonus of +10 (+4 BAB, +4 strength, +1 masterwork, +1 weapon focus) who is doing 2d6+6 damage (greatsword, +6 two-handed strength); against an AC 21 Drow Noble. With an average hit rate of 50% and an average damage of 13, the expected value for damage is 6.5, without even bothering to consider critical hits or power attacking or whatever. This is well over 10% of the Drow Noble's HP of 20. How about an AC 17 Derro? Now the hit rate is 70%, giving expected damage of 9.1, again well over 10% of the Derro's HP of 25. How about an Ogre? The Ogre is AC 17 also, and has 30 HP, so 9.1 is still well over 10%. An Advanced Ogre has AC 21 and 38 HP, meaning the expected damage of 6.5 HP is still over 10% as well. You get the idea. There may be some monster out there this works for, but for the most part, it seems like the numbers demonstrate that what the maneuver gains is most of the time not worth the opportunity cost of attacking.

What about demoralizing knowing that my cleric buddy can also hit the target with fear to stack the effect?

The spell fear takes the target straight to panicked if they fail their save, so the real issue is that demoralizing lowers the save, and causes the target to become frightened for one round even on a successful save. So, it has a somewhat narrow region of efficacy, but I spent enough time on the last one that for the time being whether or not this combo actually ever ends up being worthwhile is left as an exercise for the reader.

I agree that a lot of these can be mathematically analyzed, but I guess I don't really see the point because of how many variables are involved.  If the basic point here is simply to say that "a maneuver isn't worth it unless it is better than a normal attack," then yes, I think we can find broad agreement on that; it's not a terribly controversial proposition.

The point is to determine when a maneuver is worth it or not. In order to know if and when a maneuver is better than an attack, it requires analyzing both the maneuver and the attack and determining which is actually better. If the number of variables starts becoming unreasonable, we can throw in some best case common sense assumptions-- that's still a whole lot better and more likely to generate worthwhile outcomes than just throwing up our hands and not even bothering.

Most maneuvers in PF, when optimally used, work out with your analysis just fine; Feint is not worth it unless you're sneak attacking, Intimidate is not worth it unless you're stacking fear, Steal is not worth it unless the enemy has a Wand of Killing You in his belt, and so on.

Stealing a "Wand of Killing You" is a big situational debuff and I'd agree that it's useful. Feinting for sneak attacking and runs into the problem that you're giving up the chance to make two mediocre attacks for a chance to try to get a really powerful one-- figuring out when that is good and when it isn't is the entire point of my original post. For example, the formula clearly tells us that a Rogue with 2d6 of sneak attack against an enemy with a +2 Dex bonus to AC that he has a 70% chance of hitting and 90% chance of feinting should feint if he's doing d8+3 of damage or below, but shouldn't if he's doing d8+4 or above. If the chance of feinting drops to 80%, then don't bother doing it unless the weapon is only d8+2 or less. Our aforementioned Drow Noble has a lot of Sense Motive so feinting her is 50/50 shot at best and it's never worth it unless stuck swinging a weapon for d6 with no bonus whatsoever.

Sometimes it works, and sometimes it doesn't, and if I'm wrong about the efficacy of certain maneuvers in certain cases, then it would be enlightening to all involved to demonstrate that. Nothing is accomplished by simply declaring maneuvers "work out" when "optimally used" and leaving it at that.

They're not trap options because they're worse than attacking; when used optimally, they're usually not, and you could even make that argument for the lowly Dirty Trick.

Yes, you could make that argument. If you want to make that argument, feel free to do so. If you're just going to assert "when used optimally, they're usually not worse than attacking" without giving a precise definition of what "optimally" is and supporting that with a detailed analysis involving some hard numbers (or at least expressions involving variables) then I really have no reply, because you haven't actually presented anything concrete for me to reply to.

None of the combat maneuvers in Pathfinder are not intended to cause damage at all

I know. The point was that the bonuses granted by the maneuvers raised the expected value of the damage on the subsequent attack.

[Steerpike]

OK, I get your point sparkletwist.  You're saying - like in the rogues/feint example, or the bull rush example - that there's still a mathematical way of determining whether a maneuver is advantageous or not, once you factor in all of the variables.  I agree with that, I'm just pointing out that context is absolutely vital to both those scenarios, and that what seem like useless/sub-optimal maneuvers in most situations become vitally useful in others.

In most situations, the math quickly becomes complex enough and opaque enough that the situation becomes too unpredictable to crunch the numbers on the fly.  With feint, for example, you may not know your target's Sense Motive or AC or current hp, and so it becomes ambiguous (in practice, in the evolving sea of variables in a given session) whether or not feint is statistically useful in causing damage.  Sometimes - like when you have a bunch of sneak-attacking allies, a combatant who relies on agility rather than armour, when you're setting up an enemy for your 6d6 sneak attack, etc - it may be worth it, other times it's not.  But it shouldn't be reliably, predictably worth it in the majority of situations, I'd contend, because that would flatten the tactical landscape.

In essence, I'd suggest that you can't critique a combat maneuver fully until you've explored a diverse array of different contexts in which it might be useful, both in granting straight up numerical combat advantages and in pursuing the plethora of additional goals that may be relevant in a combat encounter.  If a maneuver's simply too niche to ever realistically come up (or be statistically effective) in a campaign, I'd agree it's poorly designed, but if one can make a case for its occasional usefulness (statistically or qualitatively) - as with bull rush, feint, grapple, disarm, dirty trick,* etc - I think it qualifies as a "good" or well-designed maneuver, even (especially!) if it's not statistically optimal in most situations.

*I consider blindness probably the most useful dirty trick condition overall, because it drastically reduces enemy chances of hitting unless they use up their movement, which with certain foes drastically decreases their potential for damage-dealing and general usefulness.

[Gamer Printshop]

Except for someone whose character is a grappler build, so they are doing grappling all the time, I look a combat maneuvers as being useful in with situational instances only. Using a combat maneuver to move a particular guardian from his position on the battlefield, so he's not in a guarding position in front of the enslaved princess, MacGuffin, avenue of escape. When actual combat becomes a secondary goal with a particular encounter, combat maneuvers are utilized to help accomplish that goal. In general combat maneuvers in combat are very situational - bullrushing an opponent off a cliff, tripping an opponent who is currently controlling the battlefield (in a place you can't charge), sundering or stealing the wand/staff/rod that is currently be used to kill off the party (removing an item to gain advantage).

As a standard tactic to use combat maneuvers in typical combat encounter, my group doesn't even consider it. Where the goal is to cause the most damage in the shortest amount of time, nothing beats pure combat, and in that situation combat maneuvers aren't even considered. (Note I'm saying this as it applies to me and my group, not necessarily everyone's situation.)

[Steerpike]

Since hold person is a save-or-lose, to keep things simple for the mathematical model, we'll consider its 'damage boost' to essentially be equal to 100% of the enemy's hit points. I know that is not strictly accurate and there are flavorful and tactical reasons that make it distinct from an attack, but from a pure numerical analysis, the question is simply "does doing this help to take the enemy out of the fight or not?" so it's not a bad way to go. So the question becomes, is doing a dirty trick to reduce the target's save worth more than making an attack?

Dirty trick imposes a variety of conditions, but the useful ones for this case all are -2 to saves, so that is, generally speaking, a 10% increase to the chances of succeeding at the hold person. When considering expected values, that makes a dirty trick in this combo "worth" 0.1 of a hold person. In other words, it means a dirty trick is essentially worth 10% of taking an enemy completely out of combat. Another way to gain 10% of taking an enemy completely out of combat is, of course, to damage the enemy for 10% of its hit points. The dirty trick + hold person combo is thus only worthwhile if the dirty trickster's average attack damage is less than 10% of an enemy's total hit points. The wizard might want to try the spell regardless, but I'm not going to worry about that part of it. I just want to know if we should attack or do a dirty trick.

I think there's a flaw here.  Unless I'm wrong you're equating the -2 penalty to saves with dealing 10% of an enemy's hit points worth of damage.  But the situations aren't actually comparable, because Hold Person - the spell your dirty trick is helping - takes the opponent out instantly, whereas chipping away at the opponent's hp could take many rounds of combat in which you risk further damage, reinforcements arriving, the enemy using healing potions/spells, etc.  Taking an opponent out instantly, in other words, is worth much, much more than 100% of an enemy's hit points.

In other words, I think the parts of your analysis I bolded above are incorrect.  Dirty trick increases the chance of instantly taking an opponent out of combat by 10%, which is not remotely the same as merely dealing 10% of their hit points worth of damage, leaving them with plenty of time to smash you to paste, regenerate, etc.

[Polycarp]

The dirty trick + hold person combo is thus only worthwhile if the dirty trickster's average attack damage is less than 10% of an enemy's total hit points.

Maybe, but maybe not.  Hold Person takes out a target immediately; beating an opponent to death generally takes multiple rounds.  You could certainly make the argument that hitting someone for 20% of their HP is better than adding a 10% chance to KO him right then, but that means 4-5 rounds of attacking - obviously less than that since you've got a wizard on your side potentially doing other things, but this may be a fight you need to (or would prefer to) end right now.  The medusa might petrify me next round - if I decide her odds are good enough, it might be rational to increase the wizard's chance of disintegrating her this round from 70% to 80% even if I would normally deal 60% of her HP in damage with an attack, because 60% damage will still allow her to gaze at me next round, and a successful disintegrate won't.

Now I have no doubt that you could figure out the odds of that, even with those added variables.  But in actual combat, you don't know the medusa's HP, AC, or the gaze save DC; in some combat encounters you may not even know what you're up against at all.  You are assuming perfect information when you make these calculations.  You also can't predict what your goals are in combat - maybe what I care about is not defeating an opponent as such, but keeping the enemy from attacking our cleric while he heals our dying friend.  That's a situation in which I would call tripping (in order to reduce his movement this turn) "optimal" regardless of how competitive it is, damage-wise, with a regular attack.  This kind of occurrence is probably not common, but we've already admitted that "uncommon" is the ideal frequency at which maneuvers should be useful anyway.  "Combat maneuvers are often worthless" is probably literally true, but "often" is not how you're supposed to use them.

I absolutely do not mean to suggest that what you are doing is pointless or not useful.  To illustrate that, let's talk about trap options.  The original use of the term, I think, is in regards to Toughness - Toughness isn't strictly worse than any other feat choice (I can imagine a contrived scenario in which a druid who took Toughness is better off than one who took Natural Spell instead), but it's 1) usually worse, and 2) a new or inexperienced player may not be aware that it's worse (and is thus "trapped").  By that standard, combat maneuvers in PF may indeed often be "trap options," if only because many players haven't done the math - math you have done, and thus math that can be useful in illustrating when one should and should not feint or demoralize or trip, if one cares to spend the time doing it.

But in practice, there are situations with unknown factors that make the math impossible, or situations (like the tripping scenario I mentioned) in which damage is irrelevant, or even situations in which a player may simply not care about what's superior - "I feint and sneak attack" might be a thematic choice just like "my fire mage casts fireball."  That doesn't mean that we should just throw up our hands and throw the math in the toilet, because it's worthwhile being able to show a new player that he probably shouldn't blow a standard action using demoralize every round when he would be better off just be hitting the guy.  "Trap options" are about information - that's what distinguishes "trap option" from "situationally useful option" - and if a formula is helpful in that regard then, well, it's helpful.  On the margins, however, and many times in actual play, a formula may be irrelevant or simply not possible to utilize because there are too many factors you don't know, and that's what I (and, perhaps, Steerpike) am getting at.

(Comedy option:  Every maneuver in Pathfinder is a trap option by definition because the fact that you are performing a maneuver strongly implies you chose a combat class instead of being a wizard, and as we all know, "being a Fighter" is the ultimate trap option.)