N mutually orthogonal vectors span an N-dimensional vector space, meaning that, any vector in the space can be expressed as a linear combination of the vectors. This is true of any set of N linearly independent vectors.
The term is used loosely to mean mutually independent or well separated. It is used to describe sets of primitives or capabilities that, like linearly independent vectors in geometry, span the entire "capability space" and are in some sense non-overlapping or mutually independent. For example, in logic, the set of operators "not" and "or" is described as orthogonal, but the set "nand", "or", and "not" is not (because any one of these can be expressed in terms of the others).
Also used loosely to mean "irrelevant to", e.g. "This may be orthogonal to the discussion, but ...", similar to "going off at a tangent".