Indeed, if we can find the private key from examples of signatures and public key, then the signature system is insecure.
More precisely: if the "thousands of examples of signatures" are for messages not under control of the attacker, finding the key is impossible for a signature that meets the definition of secure under UF-KMA (Universal Forgery under Known-Message Attack). If they are for messages chosen by the attacker, that's UF-CMA (Universal Forgery under Chosen-Message Attack). Most common signature schemes (RSASSA-PKCS1-v1_5, RSASSA-PSS, DSA, ECDSA, EdDSA..) are believed secure under an even stronger attack model, EF-CMA (Existential Forgery under Chosen-Message Attack). See this for details on this terminology.
And yes, it's surprising that there are signature systems that are secure under these conditions. It's difficult to build one. The feasibility of the concept was only formulated circa 1976 (Martin Hellman's New directions in cryptography), and it took a few years to find practical systems like RSA and Lamport signature (the later can only sign a limited number of times). How this feat is achieved varies from one signature system to another.
In the particular case of textbook RSA signature (which I consider in the rest of this answer), we can give a justification that obtaining thousands of signatures of random messages $m_i\in[0,N)$ does not much help someone knowing the public key $(N,e)$, as follows:
- These signatures are $s_i\in[0,N)$ with ${s_i}^e\bmod N=m_i$.
- There's a one-one mapping on $[0,N)$ between messages and signature, thus the $s_i$ are random too.
- There's no way to tell from a pair $(m_i,s_i)$ if it was obtained by signing a random $m_i$, or by computing $m_i:={s_i}^e\bmod N$ from a random $s_i$.
- The public key $(N,e)$ allows to build an unlimited supply of $(m_i,s_i)$ pairs in this way, that are indistinguishable from those obtained by signing random messages, at low computational cost (for common $e$, much less than signing for the legitimate holder of the private key).
- Thus $(m_i,s_i)$ obtained by signing random messages can't help much.
This argument does not extend to signature of non-random messages. As far as we know, that still does not help obtaining a working private key or the signature of random messages, but can help obtaining the signature of some messages with some characteristics. E.g. given $(N,e)$ and the textbook RSA signature of (the ASCII representation of) 80% of the words in a large dictionary, it's plausible we can find the signature of one of the remaining words.
For this reason, textbook RSA signature is not secure under EF-CMA, and actual RSA signature signs messages that are internally transformed into a suitably random-like $m$ thru a padding mechanism.