Optical coherence tomography (OCT) is a powerful biomedical imaging technology that relies on the coherent detection of backscattered light to image tissue morphology in vivo. As a consequence, OCT is susceptible to coherent noise, known as speckle noise, which imposes significant limitations on its diagnostic capabilities. Here we show Speckle- Modulating OCT (SM-OCT), a method based purely on light manipulation, which can remove speckle noise, including noise originating from sample multiple back-scattering. SM-OCT accomplishes this by creating and averaging an unlimited number of scans with uncorrelated speckle patterns, without compromising spatial resolution. The uncorrelated speckle patterns are created by scrambling the phase of the light with sub-resolution features using a moving ground-glass diffuser in the optical path of the sample arm. This method can be implemented in existing OCTs as a relatively low-cost add-on. SM-OCT speckle statistics follow the expected decrease in speckle contrast as the number of averaged scans increases. Within a scattering phantom, SM-OCT provides a 2.5-fold increase in effective resolution compared to conventional OCT. Using SM-OCT, we reveal small structures in the tissues of living animals, such as the inner stromal structure of a live mouse cornea, the fine structures inside the mouse pinna, and sweat ducts and Meissner's corpuscle in the human fingertip skin - features that are otherwise obscured by speckle noise when using conventional OCT or OCT with current state of the art speckle reduction methods. Our results indicate that SM-OCT has the potential to improve the current diagnostic and intra-operative capabilities of OCT.