The solution presented below is possibly the simplest. It is not the only solution, but it perhaps demands the least mental effort from the magicians.
In any group of five cards, there must be at least two of the same suit. A selects one of the cards from a duplicate suit and hands it back to the audience member. The other card (or one of the others, if there is more than one) is placed first in the set of four, and will indicate the suit.
Next, think of the remaining three cards as Low, Medium, and High values (their actual values don't matter.) Any pre-agreed order will suffice; for example, ascending face value (Ace to King), with the suit used as a tie-breaker, if necessary. We could use an alphabetical suit order: Clubs, Diamonds, Hearts, Spades. So, for example, 3S would come before 7C; and, using the tie-breaker, 7C would come before 7H. Hence the Low-Medium-High ordering in this case would be: 3S, 7C, 7H.
Given Low, Medium, and High, we can encode a number between one and six as follows:
LMH = 1, LHM = 2, MLH = 3, MHL = 4, HLM = 5, HML = 6.
This still leaves us short, as the hidden card could be one of 12. However, there is one opportunity to impart information we have not yet used: A gets to decide which of the cards from the duplicate suit to retain, and which to hand back to the audience member. How can the retained card be used to indicate more than just the suit?
Imagine the 13 face values (Ace to King) arranged in a circle. The shorter path between two cards, counting forward from one card to the other, is never more than six places. Therefore, A chooses to retain a card that begins the shorter path, and hands the other card to the audience member. B uses the encoded number to count forward from the first card in the hand of four.
Example: the audience member selects the following cards - 2C, 5D, JC, 5H, KS - and passes them to A. The only duplicate suit is clubs. Counting forward from JC to 2C is four places, so A retains JC and hands back 2C to the audience member. The first card in A's hand will therefore be JC. Of the other three cards, 5D is Low, 5H is Medium (using the suit tie-breaker), and KS is High. To represent four, A uses the ordering MHL. So the four cards are: JC, 5H, KS, 5D.
To decode, B notes that he must count forward from JC. He notes that the natural ordering of the other three cards is: 5D, 5H, KS, and so the cards 5H, KS, 5D represent ordering MHL, which encodes the number four. He therefore counts forward four places from JC and announces the two of clubs!
With a little practice, this trick can be made to flow quite smoothly. If performed repeatedly before the same audience, it is advisable to permute the position of the suit card, to make the trick harder to read. For example, on the nth performance, place the suit card in the (n modulo 4)th position.
Source: See credit for puzzle 20