Hey LilWing,
You are right, there's a pretty good chance it will break..
I take the items off the bike before attempting to straighten them. Use a small hammer and only as much striking pressure as it takes and go slowly. Test fit and repeat until the brake no longer rubs.
If you have a small propane or mapp torch it will help if you warm up the piece before attempting to bend it.
You only want to get it about 200 deg. 2500 and it will melt!
If you have an old pie tin, or the like, clean off the grease from the pivot mount and put it in the oven at 200 for 15min.
Then hammer the bend out. Still have to take it easy, but the aluminum will be softer.
There is a member here who cut a small piece of diamond plate and epoxied it to the case where the brake arm would rub now and again.
It looked pretty good polished up, and when it gets too worn you just cut another piece and replace it. Good Luck,
Tuck\o/
OOps, Blaster and I were typing at the same time. Great minds think alike!