It is not quite as simple as you seem to imagine. While an irrelevant backlink has no direct influence on webpage ranking, it can indirectly influence a page ranking using correctly applied techniques.
So it is true that an irrelevant backlink seems to have absolutely no direct ranking benefit, it still passes link juice which, when properly channeled through a subsequent relevant link, will influence rankings. That is one of the reasons that proper internal link structure is so effective in influencing your webpage’s ranking.
You must consider the above-mentioned influence when analyzing your competition else your conclusions may be invalid.
Furthermore, many folks seem to not understand how Google determines backlink relevance, they often confuse website topics with the page topic. Page topic is what Google uses and anchor text is the most influential signal in backlink relevance.