It's a little worse than just being in an unknown state, or toggling needlessly. Digital circuits nowadays are mostly of CMOS type, with transistors switching both high and low sides; when we have clear 1s and 0s, they are either off or saturated, the two most efficient states for the transistors to be in. In between, however, is a region of linear operation; it's used for analog amplifiers, but it is not as efficient as the extremes - meaning more power is wasted as heat in the transistor. In the worst case, both the high and low side transistors leak thus (because the pin is in fact neither high nor low), and they can then combine to cause a notable current within the chip as they try to drive the internal state both high and low - possibly doing the same to the next gate in a chain reaction. The heat could become a problem even if the power isn't. IntelliChick's solutions still apply.
For pins also connected to ADCs, some microcontrollers offer the function to disable the digital input buffer, to prevent both this problem and leakage distorting the signal.