Rheologic variations in the Earth's crust (like elastic plate thickness [EPT] or crustal rigidity) modulate the rate at which seismic moment accumulates for potentially hazardous faults of the San Andreas Fault System (SAFS). To quantify rates of seismic moment accumulation, Global Navigation Satellite Systems, and Interferometric Synthetic Aperture Radar data were used to constrain surface deformation rates of a four-dimensional viscoelastic deformation model that incorporates rheological variations spanning a 900 km section of the SAFS. Lateral variations in EPT, estimated from surface heat flow and seismic depth to the lithosphere-asthenosphere boundary, were converted to lateral variations in rigidity and then used to solve for seismic moment accumulation rates on 32 fault segments. We find a cluster of elevated seismic moment rates (11–20 × 1015 Nm year−1 km−1) along the main SAFS trace spanning the historical Mw 7.9 1857 Fort Tejon earthquake rupture length; present-day seismic moment Salton Trough is reduced to only 60% of the regional average, which results in a ∼60% decrease in moment magnitude on these segments ranges from Mw 7.2–7.6. We also find that the average plate thickness in the at least a ∼30% increase in moment rate and even larger increases are identified in regions of complex accumulation rate along the Imperial fault. Likewise, a 30% increase of average plate thickness results in plate heterogeneity. These results emphasize the importance of considering rheological variations when estimating seismic hazard, suggesting that meaningful changes in seismic moment accumulation are revealed when considering spatial variations in crustal rheology. Plain Language Summary The earthquake potential of faults (i.e., moment magnitude) is determined by a fault's deep slip rate, the area of the fault, the time since the last major rupture, and the average crustal rigidity surrounding the fault. Here, we explore how earthquake potential is affected by variations in crustal rheology along the San Andreas Fault System (SAFS). We use measurements of surface heat flow and seismic velocity to estimate lateral variations in crustal rigidity and tectonic plate thickness. We then estimate seismic potential rate for 32 segments of the SAFS using surface deformation measurements from Global Navigation Satellite Systems and radar interferometry as constraints. We find high seismic potential rates along several segments of the San Andreas fault that also participated in the Mw 7.9 1857 Fort Tejon earthquake and have not ruptured in over 160 years. We also find that the change in earthquake potential scales almost linearly with plate thickness variations surrounding the fault. For example, the Salton Trough region has a plate thickness that is only 60% of the regional average, which lowers the earthquake potential along the Imperial fault by roughly 60%. These findings suggest that variations in crustal rheology have an important impact on earthquake magnitude forecasts.