In fact the amount of people that truly understand the cryptography of Bitcoin might be less than 20 people
i don't think so. Any one that has taken basic a cryptography course or read some intro textbooks (handbook of applied cryptography, foundations of cryptography, cryptography theory and practice, etc) knows very well the math behind bitcoin's cryptography; it is much easier than you think when compared to more complex mathematics.
besides, anyone with basic knowledge of computer science can read "Programming Bitcoin" or "Mastering Bitcoin" and understand the technical aspects of Bitcoin in detail.
Surely, it requires some technical knowledge of computer science, mathematics, programming. But it is pretty accessible for people who work/study on these fields and also related fields like engineering.