If you commonly feel guilt you might enjoy two of my favorite aha's: "They'll be OK." and "I'm not responsible for how you feel."
I used to get so worked up over saying "no" or being concerned I was disappointing someone by doing X, Y, Z. God forbid someone else feel inconvenienced, hurt, or uncomfortable. I'd go through hoops to avoid someone else feeling any stir but brightness.
But that did several things that suck:
1. I'd end up going against what I needed to actually do in order to dance around what I thought could hurt someone.
2. I'd diminish one's capacity to adapt by assuming they couldn't handle X. Who I am to judge what someone can or can't handle?
3. By not doing what served me best in the moment - the other party (whether they consciously got it or not) usually lost out. Because the world only benefits when you're your shining best and happy.
4. I'd withhold the opportunity for one to deal honestly with their own reality. That's juicy growth time!
If this tugs at you. Take a deep breath. Stop enabling. And do what's right for you. They'll be OK. And, you're not responsible for how they feel. You can't do anything about it. Only they can make a decision about how they handle and interpret their own reality.
This applies for silly little things like saying, "No, I don't feel like going out tonight," to "I no longer want to see you" to "You know, I'm really disappointed in X, that didn't sit well with me."
Disclaimer. This, like many wisdom wonders, has to be managed within the context of its antithesis - in this case, it's ultimate compassion and kindness towards the other. It's a funny tension - but holding it will set you free!