An injury-hit Magpies were second best for much of the game as goals as Niclas Fuellkrug and Julian Brandt secured back-to-back wins for the Germans over their English counterparts.
Newcastle must now likely have to win both of their remaining games, away at Paris Saint-Germain before welcoming AC Milan to St James' Park.
The match was briefly interrupted in the second half by fans throwing fake gold bars onto the pitch in protest at European governing body UEFA's planned competition reforms.
Newcastle, looking more aggressive in the second half, missed a golden chance to level through Joelinton's close range header in the 56th minute.
Dortmund are top of the group on seven points. Newcastle, who were missing injured key players including Sven Botman, Alexander Isak, Harvey Barnes and Elliot Anderson, have four.
Additional reporting by Reuters.