If you've studied physics, you might immediately balk: Tensors seem like the tools of the current/previous generation of physics. You're not wrong.
But what impresses me about the use of Tensors (or, developing even further on this, Einstein's use of otherwise obscure ideas from geometry – ideas that got developed much more after people realized they described reality) is the non-obviousness of it. Physics is just littered with them (and psuedo-tensors, but whatever). Because that's precisely the mathematical object you'd want to use to describe fields that are invariant over many transformations. They're peachy, and they give you the framework to hang all your ideas off of.
But here's something I've been noodling: most physical theories have a field, to which many things contribute, and then we predict how things behave by modelling them as test particles in that field. We do this despite knowing for certain that isn't a good representation of reality – we knew that already for things like the electric field, but that space itself doesn't work that way isn't even a new revelation.
When I think about what mathematical objects will be used to describe the next era of physics, it's an object that captures the idea that these aren't really two different steps – reality is a canvas that pushes the paint. It sounds impossible if you phrase it like that, but obviously the true theory will be 'timeless' in the sense that it's more describing fixed relationships in some 3+1 space. Since you'd also want to fold in quantum weirdness (to which I mostly refer to superposition and the universe's are-these-states-identical checker it implies, as well as making sure anti-commuting swaps on wave functions mean fields go from quantum increments to binary [the Pauli exclusion principle applying to fermions only is... if there's an understanding which makes that obvious, point me at it], which I can't wrap my head around at all), these objects are going to get weird. But a lot of these properties are so simple, like how Tensors work, that once you nail the basic structure, the properties might just fall out naturally.
Maybe it's the massively-higher-dimensional models of String Theory – but I suspect it's more that higher dimensions can accidentally model a totally different system that has fewer dimensions – in fact I suspect that this generally idea almost has to be true (what is a field constraint, if that's not the case?), which would a bit of a death knell for the geometric realism of String Theory, if it turns out there's some weird epicycle thing going on in the dimensional geometry.
I don't know enough physics to even imagine what these mathematical objects might be. Maybe they're resting in the realms of recreational math. I suspect, given how genius Einstein/Reimann/Lorentz/Minkowski were, geometry might not be the answer, or they would have jumped further than they did. Truly, I cannot express the level of genius required to jump between special and general relativity – were there a further jump in that skill tree, they would have made it, or hinted at it more strongly. One guy invented SR and GR pretty much one after the other. Bonkers. But that's just an outsider's guess – geometry has helped a lot, maybe further work is warranted.
[Edit: I forgot to mention that the first few major advances in science were conservation laws, and Noether's math supporting them blew that whole sector to smithereens – essentially totally finishing the scalar invariant mathematics ~while Einstein found some interesting 3+1 dimensional invariants.]
But this is why I think modern physics is maybe overly-conservative. All the ideas sort of look similar to each other, at least to me. None is a deeply alien mathematical structure in the way you might suspect. It's hard to imagine any string theory being as deeply strange as Special Relativity was at the time, just peculiar.