One must distinguish between signifier and signified.
The terminology is from Saussure, who distinguished both signifier and signified from the referent object.
The signifier "C-A-T", whether as a sound or as a written expression, is of course a real object in the physical world (maybe only in the form of bits and bytes, but nevertheless physically real).
The referent object, a cat or the set of all cats in general, is also something physically real.
In between the two, however, is the signified - the concept, rather than the object or objects, to which the signifier refers.
To switch to an example which I can more easily defend - one cannot, upon hearing or reading the signifier "table", locate a "table" in the physical world, without first making recourse to a generalized concept (whether consciously articulated or not) of what a "table" is.
I think it is empirically true that we do have such concepts, which we use, even before we have begun to analyze or define them. The process of making definitions is itself more complex and more problematic than we might immediately assume. It would be easy for instance to include in the definition of table that it has legs; but I could then show examples of unusual tables that do not have legs, which most people would nevertheless accept as being tables. Definitions of the "signified" of signs (the concepts to which signifiers refer) are, in general, also subject to revision for reasons such as this. The signified or concept, that is to say, is in some way prior to our making of specific attempts at definition, and should not be confused with a definition.
The crucial point is that the signifiers of mathematics which we handle very readily seem to have "signifieds" - to refer to concepts - which are not things in the physical world, and which cannot actually (it would seem) be derived from things that exist in the physical world. It seems more appropriate to say that some things or phenomena in the physical world approximate the nature of mathematical (geometrical, arithmetic, algrebraic, etc.) "objects", than to say that the mathematical concepts are actually derived empirically from our experience of the physical world.
Mathematical concepts or objects and our empirical experience of the physical world are related in intriguing ways. It does not seem immediately obvious that either is reducible to the other.
Thank you for your comment!