The purchase is AOL's biggest since becoming independent from Time Warner Inc. in 2009.
AOL will pay $322 million in cash and about $83 million in stock for Adap.tv, based in San Mateo, Calif., the company said in a statement.
AOL paid $315 million for The Huffington Post in 2011.
Adap.tv's platform lets publishers and advertisers trade video advertising space in real time.
"AOL is a leader in online video, and the combination of AOL and Adap.tv will create the leading video platform in the industry," AOL Chairman and Chief Executive Officer Tim Armstrong said in a statement.
"The Adap.tv founders and team are on a mission to make advertising as easy as e-commerce, and the two companies together will aggressively pursue that vision," he said.
Adap.tv CEO Amir Ashkenazi said: "We believe that most TV advertising will soon be traded programmatically on platforms like ours. The combination of AOL and Adap.tv accelerates our vision of efficient and effective TV and video advertising."
Privately held Adap.tv, founded in 2007, has about 210 employees. Besides in San Mateo, it has offices in New York, Los Angeles, Chicago ,London, Sydney and Hyderabad, India.
The deal is expected to close by the end of September.