Goals from Leon Goretzka, Guido Burgstaller and Daniel Caligiuri looked to be sending Schalke through to the semifinals. But Ajax hit back twice in extra time and progressed on aggregate.
Schalke 3-2 Ajax AET. (Ajax win 4-3 on aggregate)
(Goretzka 53', Burgstaller 56', Caliguiri 101' - Viergever 111', Younes 120')
Heartbreak for Schalke. They produced a fantastic fightback and were in touching distance of reaching the semifinals before their dream died in a flash as Ajax netted a crucial away goal in extra time and then added another late on.
Markus Weinzierl's side had powered their way back into the tie after an insipid 2-0 defeat at Ajax last week thanks to a magical three-minute burst after the break.
First Goretzka thrashed home on 53 minutes after storming up field and getting a cute return pass from Burgstaller and soon after, the Austria striker rammed in the aggregate leveler.
Caliguiri thought he had won it for the hosts in the extra period but a flukey goal from Nick Viergever put Ajax ahead again in a rollercoaster encounter and to add insult to injury for the home side, German Amin Younes added a second for the visitors right at the end.
Schalke had been boosted before kickoff by news that Benedikt Höwedes and Arsenal transfer target Sead Kolasinac were fit to return in defence with right back Sascha Riether also returning to the starting line-up after a long absence.
Indeed, the hosts started the game in confident mood with Max Meyer cutting inside and striking the post after just three minutes.
Suddenly a memorable comeback looked on as Schalke dominated the opening 10 minutes, roared on by an expectant crowd.
Suddenly the game changed
But Ajax, who so comfortably overran the Germans last week, eased their way into the game and had a good half chance when Hakim Ziyech tested Ralf Fährmann with a free kick.
The Schalke goalkeeper was never likely to be as busy as he was in Amsterdam, where he saved his side from a more embarrassing deficit, but he was there when he needed to be as Ajax upped their threat.
Schalke still carried their own though, and the lively Meyer blasted over when well-placed after a quick break.
The home side just about edged the first half but it was the Dutchmen in fluorescent green who had the first big opportunities in the second period. Schalke were glad that faithful Fährmann was on hand to beat away Lasse Schöne's flying set-piece and save Bertrand Traore's effort.
Then suddenly the game changed. Burgstaller had endured a frustrating evening up to that point but had the vision to play a one-two with the charging Goretzka, who shot hard and low to make sure he scored despite a deflection. He later went off injured and was taken to hospital.
Weinzierl then threw on veteran Klaas-Jan Huntelaar but the Dutchman, heavily linked with a return to former club Ajax in the post season, could only watch as strike partner Burgstaller bagged the second from a typically bustling Kolasinac cross.
Ajax quickly looked a shadow of last week's side, with Joel Veltman sent off on 80 minutes for a second booking after a cynical foul on Nabil Bentaleb. The man advantage should have spurred Schalke on but they only mustered one very good chance from Höwedes in the final moments of the 90 minutes.
It looked like being the same story in extra time but Caliguiri popped up to expertly head home after 101 minutes despite the attention of two defenders, with Kolasinac again instrumental.
The Veltins Arena has seen few nights like this in recent years but such is the side's inconsistency, the atmosphere actually went a little quiet as time wore on, fans petrified that Ajax would hit back with a crucial away goal.
And that is exactly what happened. It would not be Schalke without more drama.
Out of the blue, Viergever grabbed the goal Ajax needed and it was as scrappy as can be. The Ajax player miscontrolled a cross but as Matija Nastasic tried to clear, the ball cannoned off Viergever and into the net.
Schalke desperately tried to hit back but gaps were inevitable and Younes completed the job.
"Compliments to the team for a good performance. It's an extremely bitter feeling to have gone out today. It is a bitter defeat as we were brutally unlucky,” Weinzierl told Sky, referring to Ajax’s first goal.
"Now we have to absorb the result. We deserved to qualify for the semifinals."
It had looked like being a famous triumph for Weinzierl but now he faces the prospect of being the first Schalke boss since 2009 not to qualify for Europe next term.
In the other ties, two more games went to extra time.
Manchester United beat Anderlecht 2-1 on the night and 3-2 overall after Marcus Rashford’s extra-time winner. But United lost Zlatan Ibrahimovic to a serious-looking knee injury.
Lyon also went through to the semifinals after winning 7-6 on penalties after a 2-1 loss on the night at Besiktas.
Spain’s Celta Vigo progressed with a 1-1 draw at Genk sealing a 4-3 aggregate victory.
The semifinal draw is on Friday with the first legs of the last four on May 4 and the second legs on May 11.
Re-live the Ajax v Schalke action as it happened: