The outcome brought about a successful debut for Brian Bliss, who was named interim coach of the Crew this week after Robert Warzycha was dismissed by the team.
Warzycha had been in his fifth year as coach, but was let go after the Crew had lost five out of seven games.
Columbus (9-13-5) is five points out of a playoff position in the Eastern Conference.
Finley scored in the 47th minute after taking a pass from across the penalty area from Justin Meram.
Meram then beat Houston goalkeeper Tally Hall in the 74th minute.
Houston (10-9-7) remained tied with New England for fifth place in the East. Houston has three losses and a tie in its last five outings.
Workers accuse National Zoo of animal mismanagement
Video of Victoria’s Secret models trying to 'twerk' hits Instagram