We could ask if it's okay to steal a loaf of bread to feed your starving family. But obviously the goal here isn't to help the starving person choose a moral outcome, but to choose the least immoral outcome. Both stealing and letting their family go underfed are bad.
So, when people ask the question, they don't mean "is it good?" – they know the answer is no – but "should we judge that person harshly, or negatively at all?"
I suspect most moral reasoning is actually trying to answer the second question, using the language of the first, but we might have to set that aside for the moment.
When we plead special cases, we're essentially talking about the converse of incentive systems. They aren't very strong, but they ought to impact people at the margin – that's their purpose, right? But who ends up being at that margin is very much in question. Stealing a loaf of bread to feed your starving family is more excusable when you're down on your luck through events out of your control. If you're a family of professional thieves who are down on their luck, of course it makes sense that you, on the margin, are going to steal a loaf of bread – the hunger pushed you to it, but many people are hungry and few steal their food. But, despite only adding details, it suddenly feels like the excuse doesn't hold a lot of water.
I think this is because it's common (and perhaps maximally polite) to assume people react to incentives somewhat randomly. There isn't a propensity to steal in those that end up actually doing it to feed their family, we might say – even though, you'd imagine they're stealing only marginally, just barely stealing when they so easily could have been in a different, no-stealing, situation – and those people at that margin probably do have some fractional tilt towards theft. They could beg or barter or just roam like a tramp trying to get work like desperate people used to on farms or something. There are shelters, out in the world, even if they aren't close to you. But instead of moving or begging or making a strange barter, they stole. Without any good decisions, that's the one they made – a strong hint to that slight inner tilt.
Nobody's perfect, and we probably shouldn't judge people for those inclinations too much. But maybe we shouldn't ignore them either?
Imagine a converse scenario, with a moral judgment you may or may not agree with – the condemnation of actors and actresses in the pornographic industry. The rule of politeness would encourage us, if we disapprove, to imagine them as totally non-inclined hapless people, faced with demands we would be outraged at. If I had to star in a pornographic movie to feed my family, I would think it was a moral outrage. It would be humiliating and violating and terrible.
Yet... I suspect that the people who take those deals at the margin have the inclinations needed to have a much, much more positive experience. The burden is less for them (for the most part) because in our free society they have access to all the other terrible options too. That isn't to say everyone is happy with their decision, or even that they'd prefer it to having had to beg. Merely that their slight internal intuitions and inclinations guided them from the things they were most worried about, to make that choice.
You can make moral judgments of people, assuming their feelings about behavior are precisely the same as yours. And you can excuse things, if you want, because of special circumstances that would make this totally normal person do something they otherwise wouldn't. But people aren't all the same. Their opinions about all topics exist on a continuous spectrum, and maybe we should acknowledge that the person to e.g. take the job of a polluting CEO is probably someone who, at the margin, isn't as bothered about that as you, and maybe not bothered by it at all. Margins incentivizing immoral behavior in a small number of people will always have a minor moral cost merely because the people who take the deals first will be the ones who care the least.