Why a Paradigm Shifts

Thomas Kuhn in his ground breaking book "The Structure of Scientific Revolutions" showed that science does not progress in a continuous series of small increments but periodically shifts from one paradigm to another. The previous paradigm is completely discarded.
This is a fundamental idea in understanding the development of scientific ideas.
I would only add that these paradigms often occur when a new technology such as the microscope or telescope etc. allows us to see another level of reality and begin to understand the effect that level has on what we thought we understood previously.