It would be nice if HN had some way to say, "this whole sub-thread together tanks the SNR". Sometimes a thread will go wildly off topic, but there are so many replies off the topic that more interesting on-topic points can't be raised above the noise. Sometimes I wonder if this could be a deliberate astroturfing strategy.
One recent example was a useless conversation about whether the US was a democracy or a republic, with 10 different people saying the same thing. Those 10 comments with replies fill several scrolling pages, so the real topic of conversation is buried.
On the other hand, you could argue that if a comment generates substantive replies (as opposed to just replies that say "this is off topic") then it's actually pretty strong evidence that the community feels the comment is on topic.
In fact, I think it would be a good thing if it was made impossible for moderators to detach comments that receive enough up-votes, or have child comments (especially comments that disagree) with many up-votes.
In those cases, it's clear that community members get value from the comment, which should infinitely override bespoke opinions of whether it's "on topic enough" to remain -- those subjective opinions ought to count for way less than simply observing whether there are up votes or replies.
It only takes about ten redundant comments (wild guess number) to derail a thread for thousands of would-be participants. This doesn't indicate community value; at best it indicates accidental nerd sniping, and at worst could be a coordinated attack on what would otherwise be a productive discussion, from which more people could derive far greater value.
But the comments are drawn from the distribution of Hacker News readers. You can't presume that just because you see a thread of ten comments as an off-topic distraction that so must the other thousands of readers, especially if there are healthy replies and up votes.
Your comment that it "derail[s] a thread" makes the implicit assumption that the comments are actually off-topic. If they are on-topic, but you happen to have the opinion that they aren't (even thought they are) then it's not derailing anything, it's just a healthy reflection of the comment-generating process that is "the pool of all HN readers." I think of it like the John Lennon quote: "Time you enjoy wasting is not wasted."
I don't have a magic answer for how to accurately detect off-topic diversions in a way that satisfied most people most of the time. All I'm saying is that upon seeing a healthy back and forth, it should give me a lot of pause that if I merely think it's off topic, I am probably wrong, and it's just an opinion that should not be indulged by moderator activity.
Interesting points. I'll just add that I don't think that enjoyment is the best metric for evaluating a thread, but maximal enjoyment and/or maximal usefulness. Especially if what I'm calling off topic prevents the community from addressing a threat to its values. I could be wrong, but it seems that the HN team want HN to be more than just a fun place to hang out for hackers.
There was another example of derailment recently, with a dozen comments arguing over what a "middleman" is when the original comment was really about how best to support indie makers.
Anyway, thanks for the interesting discussion, and I hope I haven't made this thread less useful for someone else in the process.
One recent example was a useless conversation about whether the US was a democracy or a republic, with 10 different people saying the same thing. Those 10 comments with replies fill several scrolling pages, so the real topic of conversation is buried.