Subducted sediments have been suggested to be a possible cause for local Mg isotopic variation in the overall homogeneous mantle based on anomalous Mg isotopic compositions detected in some mantle-derived rocks. Here we examine the Mg isotopic systematics of 77 bulk marine sediment samples from ten DSDP and ODP drill sites adjacent to the world's major subduction zones to assess their potential to cause mantle heterogeneity. In contrast to the limited mantle range (δ26Mg = − 0.25 ± 0.07‰), δ26Mg of these sediments varies widely from − 1.34‰ to + 0.46‰, which is similar to values measured in continental sediments and appears to be linked to sediment lithology. Calcareous oozes are generally isotopically lighter than seawater; however, the presence of detrital impurities can shift their δ26Mg to higher values. In comparison to calcareous oozes, Mg-rich aluminosilicate sediments typically have higher δ26Mg values, although they may be either isotopically heavier or lighter than the mantle. Detrital sediments are the dominant sediment type and display large variation (− 0.90‰ to + 0.18‰) that reflects provenance heterogeneity and textural/mineralogical sorting. Sediments derived from juvenile terranes have δ26Mg similar to mantle values while those derived from mature, and thus more strongly weathered sources, often have higher δ26Mg values. On the other hand, turbidites may have lower δ26Mg values if containing isotopically light dolomite and/or garnet. In addition, clay-size fractions in turbidites commonly have higher δ26Mg than the associated sands. Hydrogenetic clays have a relatively high and uniform δ26Mg value of + 0.02 ± 0.14‰ (2SD, n = 14), suggesting preferential uptake of heavy Mg isotopes during authigenic clay formation while hydrothermal clays have highly variable δ26Mg values (− 0.57‰ to + 0.46‰) that likely reflect variable exchange with seawater/hydrothermal fluids. Overall, considerable variations exist in both regional Mg mass flux and the weighted δ26Mg average of sedimentary sections subducting at different trenches. The flux-weighted δ26Mg of − 0.336‰ is estimated for Global Subducting Sediments (GLOSS-II), which is isotopically lighter than the mantle average because of the presence of thick turbidite sequences with relatively low δ26Mg values and local carbonate enrichment in a few trenches near the equator. Therefore, it appears that subducting marine sediments are a source of heterogeneous δ26Mg values in the mantle, and they may be manifested locally in volcanic rocks that form in regions having a high flux of sediment melts to the mantle.