Vendor trash is a term for items which will most likely not be sold to any player and have little or no function in the game except to be sold to an NPC vendor for money, hence the term vendor trash, also known as "Junk" to some players. The term is somewhat subjective, but generally, most players agree that poor quality or "gray" items fall into this category. Common quality or "white" weapons and armor are often considered to be vendor trash as well.
The key factors here are useless items have no demand and should be vended (non-equipable poor quality items are almost all useless) and low quality weapons and armor have low demand, but have high markup in the auction house, and are too expensive and require too much effort to sell, thus should be vended. Common items (with a white name) other than weapons and armor do have some purpose to someone; trade item, consumable, quest item, etc. These have demand and so these are not generally considered to be vendor trash. Uncommon items and better have better properties and are rarely trash; uncommon or better weapons and armor that are not bound can be disenchanted if they are not needed. Still, the effort to sell an item defines the subjective definition of what is trash. Some items are for quests available only to the opposing faction, and so on, and are too much effort to sell.
Generally, as a character progresses, the subjective definition of what is trash broadens. It becomes too costly in terms of time to deal with lesser items when there are better items available. For instance, players with the appropriate skills can make use of the meat dropped by defeated animals (Hunters or Cooks especially), the cloth dropped by defeated humanoids (Blacksmiths, Doctors or Tailors especially), or other items. Players without those skills, however, may consider these items vendor trash. It becomes an issue of relative worth of items you can acquire and carry.
To those that are not enchanters, who cannot disenchant, most soulbound items are vendor trash, unless they have a specific need for the item. Even a rare item with great stats that do not suit your character (+int for a rogue) can be useless, and may even have a low vend value. If you aren't going to use it and you can't sell it to another player, and you can't make it something else, it is clutter in your inventory, and you might as well vend it for what you can get, regardless of what it is hypothetically worth to the right character type. A great weapon or armor of a type you can never equip, earned in a quest, would be an example of this. Hence, think before you roll on an item that someone else can use. Boss drops tend to bind, and many vend for very little relative to their real worth.
Some items do not even have a sell price. Among these are average drops, but also there are quest-only items that you might get too many of, and since you can't sell them, you have to trash them. There are a few 4-slot quest reward bags that have no vend value when you outgrow them. These items are technically not vendor trash, since you can't even sell them, but can be classified as junk. These must be destroyed when you need to clear them from your inventory.
One use of vendor trash is by enchanters who apply level-inspecific enchants such as Fiery Weapon to them. Since they are rarely BoP items, they can be used repeatedly by alts or sold at the auction house. They are mainly useful for allowing new or low-level alts to quickly run through early content.