1. A piece of hardware that encapsulates some function but can't be used without some kind of front end. Today we have, especially, "print engine": the guts of a laser printer.
2. An analogous piece of software; notionally, one that does a lot of noisy crunching, such as a "database engine", or "search engine".
The hackish senses of "engine" are actually close to its original, pre-Industrial-Revolution sense of a skill, clever device, or instrument (the word is cognate to "ingenuity"). This sense had not been completely eclipsed by the modern connotation of power-transducing machinery in Charles Babbage's time, which explains why he named the stored-program computer that he designed in 1844 the "Analytical Engine".