1. [scientific computation] Used of a data set that is grossly atypical of normal expected input, especially one that exposes a weakness or bug in whatever algorithm one is using. An algorithm that can be broken by pathological inputs may still be useful if such inputs are very unlikely to occur in practice.
2. When used of test input, implies that it was purposefully engineered as a worst case. The implication in both senses is that the data is spectacularly ill-conditioned or that someone had to explicitly set out to break the algorithm in order to come up with such a crazy example.
3. Also said of an unlikely collection of circumstances. "If the network is down and comes up halfway through the execution of that command by root, the system may just crash." "Yes, but that's a pathological case." Often used to dismiss the case from discussion, with the implication that the consequences are acceptable, since they will happen so infrequently (if at all) that it doesn't seem worth going to the extra trouble to handle that case (see sense 1).