@burcidi-- I think saw palmetto also builds up in the body over time. 320mg is not a high dose at all. Most of my supplements are between 500mg and 1000mg but when it comes to saw palmetto, this dose is enough for me.
The issue is that it takes a while to take effect. Some people get impatient and build up the dose quickly. But it's better to stick to a shorter dose for a longer period because there are less side effects that way.
I've been taking saw palmetto for a year, but I don't take it every day and I take several weeks off every two months.