The 54-year-old music mogul and the mother of his 5-month-old son have reportedly rented a house in Bridgehampton for the season.
Last year Cowell rented a $26,500-a-month Upper East Side apartment, but the couple is reportedly still looking into buying a place in the city.
Cowell and Silverman welcomed their first child on Valentine's Day 2014. The two became parents after having an affair while Silverman was married to Cowell's best friend Andrew Silverman.
Cowell told The Mirror in April that while he regretted how things began between him and Silverman, his new relationship had put "things into perspective" and turned his house into a home.
Since welcoming baby Eric, the new dad has been sharing several photos of his new family on social media.