QS is based primarily on your CTR as it compares to your competitors’ CTR. You also have to deal with your account’s historical CTR average which is apparently low. You need to focus on bringing up, not only your individual keywords QS, but your account’s historical QS.
I recommend that you temporarily pause every keyword that has a QS of less than 6/10. Write a number of ads for each of the remaining ad groups and split test to see which gets the best CTR. Be sure to use best practices for writing compelling ads to maximize the CTR. Once you have your winners, pause all of the lesser performing ads. Continue to focus on a small set of keywords until you are able to get improved QS for that smaller set.
You will need to run those top performers for a few days before your account’s historical QS will improve. Continue the process on the next group of keywords (QS 5/10), until you have most of them up to 7/10, if you are unable to get a keyword up to the 7/10 QS range then pause the keyword and come back to it later. You want to take care that you do not continue to run ads for keywords that are below average QS, else it will bring down your account’s overall QS.
Another thing to consider is how well you have chosen your keywords. If you are using broad match keywords, those will usually have a lower quality score as they tend to be poorly targeted. Try focusing exclusively on exact match keywords that are extremely well targeted (not ambiguous in any way). After you get your quality scores up, then you can add phrase match keywords, and once those are getting good QS, add your broad match modifier keywords, with a generous application of negative match keywords.
Personally, I like to separate keywords by match type, I will only include exact match keywords in an ad group with other, nearly identical, exact match keywords. I put phrase match keywords into separate ad groups or sometimes separate campaigns.
Avoid the use of broad match keywords as they can often be devastating to your account Quality Scores. Broad match modifier is a better option, when applied correctly, however, you need to use negative match keywords generously with those broad match + keywords.
Since broad match keywords tend to generate a lot more impressions, most of which may be irrelevant, they can quickly kill your account Quality Score. Pause them until you get some decent QS for your exact match and phrase match keywords. Generate a Search Terms Report to identify irrelevant keywords that you need to add to your campaigns.
You need better ad text. You need to discover what your unique value proposition is and include that in your ad text. In other words, why should someone do business with you, rather than someone else? I’m sure all the other mortgage brokers claim to have “great service and excellent rates”, that is not compelling enough to get superior CTR.
You need to be more specific with quantifiable value. Why is your service great? Tell me what, in quantifiable terms, makes that believable? If you have excellent rates then put that specific quantifiable rate in the ad text. Making those small changes will make a difference.
Your call to action may also be working against you. If people are reading your ad and following your CTA to telephone, that will lower your CTR and harm your Quality Scores. Use a CTA that encourages a click, that will increase your CTR and improve your QS.