Since the current evidence of the existence of dark matter is revealed only through its gravitational influence, the way it couples to gravity must be then of primary importance. Here, unlike the standard model sector, which is typically coupled to a metric, dark matter is supposed to couple to a spacetime affine connection only through a Z(2)-symmetry breaking term. We show that this structure leads to a coupling between dark matter, which is considered scalar, and the standard model Higgs potential. This induces dark matter decays into standard model particles through the Higgs boson, which acts as a portal between the visible and the dark sectors. We study thoroughly the resulting decay modes for various mass ranges and provide relevant bounds on the nonminimal coupling to affine gravity in line with observational data. Moreover, we find that the coupling to a Higgs boson can be sufficiently large to facilitate the production of dark matter lighter than 10 GeV at current and future high energy colliders.