@fBoyle-- That does sound like an allergic reaction. I react the same way to foods I'm allergic to. You could try a piece of broccoli by itself and see if you really are allergic to it or to something else. This might be dangerous though, so ask your doctor about it first.
If you're allergic to either broccoli or cauliflower, you're probably allergic to the other because these belong to the same family. So I would suggest being careful with both.
Also, I know that some people are allergic not to vegetables per say, but to the pesticides and chemicals used while growing the vegetables. I have a cousin who has an allergic reaction to all non-organic broccoli. She has no problem with organic broccoli.
It can be tough to figure out what exactly is triggering the allergy but it's best to stay away from that food until you're sure. I recommend getting a food allergy test as soon as possible.