"WHO must revisit the question of Zika and postponing and/or moving the games," the letter stated.
The letter went on to cite the cancellation of the 1916, 1940 and 1944 Olympic games as well as the relocation of other sporting events as a result of an outbreak of a disease.
Earlier this year, Major League Baseball relocated a match in Puerto Rico due to the Zika outbreak, while the Africa Cup of Nations was relocated due to the Ebola epidemic, the letter noted.
More than 120 medical professionals signed the letter, which was also forwarded to the IOC.
In February, WHO declared the Zika virus outbreak a global emergency.
The Zika virus is spread by the Aedes mosquito common in tropical areas of the Americas, and can affect the brains of unborn babies.
The outbreak started in Brazil in the spring of 2015 and has since spread across Latin America and the Caribbean. Cases have also been reported in several European countries and Asia.