Frequentist statistics are the kind of statistics you're probably used to, where data points from past events, in sufficient numbers, are transformable into a distribution (and likely a measure of certainty) for predicting future events; one might attempt to improve one's models by inserting enough potential causes for variation into one's model and analysing across these (ANOVA, SVD, etc), ideally in proportion to the amount of data one has.
Bayesian statistics model the mental state of a certain kind of observer of events, and are potentially more powerful (or more error-prone, depending on how you see things) because they might let you do the kinds of reasoning where you're trying to figure out the likelihood of things that have not happened before and have not had close analogues.
I've never been very comfortable with Bayesian statistics; I recognise that most non-Bayesian models have deficiencies in weighing unknown factors, and that our ability to do so is an important human ability, but I prefer to consider those things beyond the reach of statistics-as-I-recognise-them and to reject formalisation of the realm they reside in. I'm unsure how to weigh this against Bayesian statistics though, and I'm not sure if I am more reluctant of Bayesian statistics specifically or the basic idea of trying to semi-formalise (which I think bayes is) things that can't be made truly formal without (presumed) departure from good judgement. Can Bayes capture human semi-formal reasoning? (I am not sure "formal" is quite the right word for what I'm getting at)
It's possible that someday I will accept Bayesian Statistics as an inferiour alternative to proper statistics when we don't have the data needed to do it right but really need something mechanisable. Particularly given that between-domain reasoning (which we do a lot of) means the first part of that "when" is pretty important.