Understanding in-utero exposure to extreme weather events is key to mitigating climate change's impact on health at birth. Using detailed historic weather records and data on infants born in the US between 1989-2004, we investigate how in-utero exposure to weather events, such as heat and cold waves or rainfall, impacts infant's health at birth. We focus on the effects of heat shocks on birth outcomes and systematically investigate heterogeneity therein using the causal forest, a recently developed causal machine learning technique. Exposure to a heat shock significantly reduces birth weight by around 6 grams on average and increases the small for gestational age (SGA) birth rate. We find substantial heterogeneity in the effect of heat shock exposure on birth weight. Especially infants born to black, Mexican, or low-educated mothers are disproportionately prone to health risks from extreme heat exposure.