Old Man Shouting at Clouds: Affording stuff

Many people, myself included, will often say no to things by saying "I can't afford this" even when that's not actually the truth. When is that the right thing to say and when is it not, and why does it even matter?