There are many people who (for reasons that make no scientific sense) think that capacitors are somehow 'evil', and mess up the sound. While this can happen if a part is selected for a completely inappropriate task, in general, capacitors are actually fairly benign. Using bipolar aluminium electrolytic caps in a crossover is (IMO) an inappropriate use, but the vast majority of film+foil caps are very well behaved, and contribute negligible distortion. However ...
It is worth pointing out that inductors are, in general, the worst passive component imaginable. This is particularly true for use in crossover networks. Because it is not usually practical to use a core material without introducing audible distortion, the coils used for crossovers are nearly always air-cored. This means that many more turns than might otherwise be needed must be used to get the needed inductance, and that means either a very large, heavy and expensive coil, or a smaller and lighter coil with significant resistance.
Because inductor coils use magnetic coupling, they are sensitive to stray magnetic fields, or any source of variable magnetic flux. This includes cross-coupling from other inductors in the network, or even speaker magnet flux modulated by cabinet vibration. The latter is unlikely in a well constructed enclosure, but it is sensible to keep the coils well away from strong magnets.
Coils also have inter-winding capacitance, and this causes them to have a self resonant frequency that may fall within an amplifier's pass-band ... although rarely if ever within the audio band. The possibility cannot be discounted that an amplifier may be confronted by a very low impedance at some frequency determined by the inductors in the system.
The biggest problem is resistance. Some people will spend a huge amount on 'special' low resistance cables, and/ or an amplifier with a very high damping factor. A typical loudspeaker crossover inductor will undo all of that instantly, adding perhaps 0.1 to 0.5 Ohm resistance in series with the bass driver (which is the very one that supposedly needs maximum damping). Resistance also causes power loss, and heat. For a crossover inside the cabinet, the last thing needed is another heat source!
So, the world's worst component is the inductor, with those used in crossover networks generally being the worst of the worst. Adding a core to reduce losses simply increases distortion (usually dramatically), so there is no easy way out (other than to use a fully active system, of course ... hint, hint
).
