Changes in terrestrial water content cause elastic deformation of the Earth's crust. This deformation is thought to play a role in modulating crustal stress and seismicity in regions where large water storage fluctuations occur. Groundwater is an important component of total water storage change in California, helping to drive annual water storage fluctuations and loss during periods of drought. Here we use direct estimates of groundwater volume loss during the 2007–2010 drought in California's Central Valley obtained from high resolution Interferometric Synthetic Aperture Radar‐based vertical land motion data to investigate the effect of groundwater volume change on the evolution of the stress field. We show that GPS‐derived elastic load models may not capture the contribution of groundwater to terrestrial water loading, resulting in an underestimation of nontectonic crustal stress change. We find that groundwater unloading during the drought causes Coulomb stress change of up to 5.5 kPa and seasonal fluctuations of up to 2.6 kPa at seismogenic depth. We find that faults near the Valley show the largest stress change and the San Andreas fault experiences only ~40 Pa of Coulomb stress change over the course of a year from groundwater storage change. Annual Coulomb stress change peaks dominantly in the fall, when the groundwater level is low; however, some faults experience peak stress in the spring when groundwater levels are higher. Additionally, we find that periods of increased stress correlate with higher than average seismic moment release but are not correlated with an increase in the number of earthquakes. This indicates groundwater loading likely contributes to nontectonic loading of faults, especially near the Valley edge, but is not a dominant factor in modulation of seismicity in California because the amplitude of stress change declines rapidly with distance from the Valley. By carefully quantifying and spatially locating groundwater fluctuations, we will improve our understanding of what drives nontectonic stress and forces that modulate seismicity in California. Plain Language Summary The earth responds elastically to changes in surface mass such that when mass is added, there is regional sinking of the land surface and when mass is lost, there is regional uplift. These mass changes can disturb the regional stress field, which in turn, may influence earthquake activity. The pattern of these disturbances, though, is controlled by the weight and area of the mass that is applied. Here we consider the mass loss and gain associated with groundwater withdrawal and recharge in California's Central Valley. This allows us to isolate the loading contribution due to groundwater storage change, which is primarily driven by pumping activity in the Central Valley, from that of other hydrologic components and quantify its contribution to stress. In isolating the groundwater component, we can show to what extent human pumping activities on both seasonal and long‐term timescales are disturbing crustal stress and possibly influencing seismic hazard. We show that seasonal groundwater loss does increase stress along faults in California.