An encounter is an enemy or group of enemies which must be defeated together, sometimes in a particular order. Typically, once a character engages an encounter, they will remain in combat until all enemies in the encounter have been killed. Some encounters, in particular boss encounters, also have scripted events associated with them.
The concept is not (to our knowledge) used internally in the game engine, instead stemming from the way Blizzard has chosen to design most dungeons and boss monsters throughout the game. As such, the term is mainly a theoretical construct which makes it easier to discuss the structure of some reoccurring challenges in the game. Most instances feature several major encounters throughout their course.