In the past, it did make a difference because it affected the initial QS when adding new keywords. Last year AdWords switched to giving every new keyword an initial QS of 6, so it is no longer as important as it used to be.
Having said that, you do not need to suffer poor QS for those terms that you are explicitly targeting. Instead of adding a lot of broad match terms to a single ad group, split those terms into age groups with small tightly focused keyword lists. Include both exact and phrase match, as well as broad match keywords to the same ad group. You will still trigger all those elusive long-tail terms, but your exact match keywords will typically have better QS and lower CPC, which will save you a lot of money over the long term.
In case you did not already know this, Google does not ding your QS for low CTR on search terms other than those that are considered an exact match. The poor account structure, putting a lot of keywords into a single ad group is what dings up your QS, because you simply cannot write ads that are specific enough to your ad group’s keyword list. Keep those ad groups small and tightly focused, write highly specific ad text, and save yourself a bunch of money.