As an absolute 100% “live and let live’ policy, yes, you are correct. That is why I’m not in the right-libertarian side.
“let’s get along and be basically decent while letting everybody be themselves”
is less flippant than it might seem.
“Get along and be basically decent” is part of a shared value system.
Superimposed with shared value system, is “letting everybody be themselves”.
As a Venn diagram, they can mostly overlap and be joined: freedom to and freedom from.
But there are also things not mentioned that are outside.