Emmy Miller was found unconscious at a neighbour’s home in Orange County, southern California, and doctors were unable to revive her.
In a statement posted on Instagram, Mr Miller said he and his family were “beyond devastated”.
Mr Miller is the most successful skier in US history, with six Olympic medals including gold at the 2010 Games.
The 40-year-old and his wife Morgan, a professional beach volleyball player, were at a party on Saturday night when the incident happened.
Their daughter somehow got away and was found in the pool by Mrs Miller, who jumped in and tried to resuscitate her, local authorities told ABC.
Paramedics took the baby girl to hospital but could not save her.
“Never in a million years did we think we would experience a pain like this. Her love, her light, her spirit will never be forgotten,” Mr Miller wrote.
The pair have a son and are expecting a third child.
The skiing world has been expressing its sympathy.