Absolutely it matters. Google has went to great lengths to help webmasters resolve this issue because it can have a very detrimental effect on your search engine rankings.
What you have done, creating backlinks to one URL while forcing Google to rank a different canonical will certainly hurt your SEO efforts. You need to pick one way and one way only, and make sure that every link that you create in your internal navigation links, backlinks and Webmaster Tools settings always point to the same canonical. In addition you should use mod rewrite to 301 redirect all incorrect canonicals to your chosen canonical.
The guiding principal is to be consistent in your preferred canonical. I does not matter which version you use as long as you always use the same version and never deviate.
Watch this video to get a better idea of why this is a big issue and what you need to do to make sure that you are doing everything possible to minimize the damage you are doing to your rankings:
