I think, in general, people don't want others involved in their affairs unless they need to be. There's a sense it which it would be weird if you have strong preferences about things unrelated to yourself. But this sort of "don't tell me what to do"-ism is (frequently) the weakest part of modern libertarian public perception, so much so that "classical liberal" is being used more and more frequently to evade this feeling. Maximizing liberty is good – but I'm concerned the "classical liberal"s aren't quite dealing with all of the practical concerns involved.
People want nosy preferences. They want to say it matters if there are billionaires, even if they themselves are not one. They want to say it matters if someone out there has some stupid belief (and they will devote infinite supplies of energy and attention to those nazis, or whatever, no matter how pathetic and substance-less they are). We are all incorrigible gossips, that's just human nature. We all care deeply about every little bit of people's lives, particularly when it's unusual.
But here's the thing: it turns out this may be a fundamentally destructive part of the world and you get impossibility results when you touch it. Or, people can make contracts to trade preferences – that breaks the impossibility result too. Non-nosyness and freedom to contract seem to be the only way to live in a sane society.
But this is all to say: there is a mathematical defense of libertarianism here, and of the liberal order more generically. And I don't think it's wise to say, well, free markets get me burritos, so those are okay, but billionaires are icky, so none of them. You've introduced nosy preferences again! I think it's worth confronting the fundamental mistake in this approach, that we ought to use government to mediate concerns that will very frequently bump into literal impossibility results. If you literally can't win, perhaps try another game (might I recommend, for all the classical liberals out there, libertarianism). Even the tic-tac-toe robot in WarGames figured that basic idea out.