When freshmen undergraduates have to take introductory physics perhaps the most common complaint, aside from the incredibly creative “this is hard,” is that the material is too mathematically involved. Classical mechanics, electrodynamics, and the other topics covered in introductory physics almost exclusively involve calculus (differential equations, really), elementary algebra, and some basic geometry and trigonometry. Traditionally, calculus is the mathematics course undergraduates at large struggle with. So, it appears these students think the calculus makes it hard. While certainly students who struggle with calculus will struggle with physics, those who are more mathematically inclined will probably argue quite the opposite: the math is the easy part! Quickly and accurately assessing physical situations irrespective of mathematical abstraction is where the difficulty lies.
Of course, before Newton and others applied mathematics to physics the subject was very limited. But, it is also true that the more math that has been introduced to physics, the easier it has become. The somewhat unfortunate part about freshmen physics is that oftentimes there is little mathematical complexity available that would both be reasonably tame as to be teachable and be beneficial for solving actual physics problems. Sure, we could all learn Lagrangian and Hamiltonian mechanics from the beginning, but only the former can help make some problems easier, and both are clearly much more mathematically complicated than the traditional approach. This is why no freshmen course I have yet seen has even attempted the approach; it is a horrible idea.
Inexperienced students often have the misconception that quantum mechanics or statistical mechanics, because of their now quite popular weirdness, will be the most difficult physics courses they will take. But for many, the mathematically inclined in particular, these courses will actually be quite easy! The same could be said for, say, relativity as well; although, the mathematical background required for general relativity is pretty substantial in and of itself. In much of the well-understood “modern physics” the physical situation is complicated and very weird, but we have enough mathematics at this point to do quite well in solving problems.
Now, quantum mechanics is not a generically easy course, but the point I am trying to make is that math is easy but physics is hard. Moreover, even potentially great physicists are quite susceptible to struggling in freshmen physics, because it is hard, period.