Did you destroy a tower at the same time you destroy the castle gate? This could explain the problem.
I destroyed everything. I went to the gate with a huge army, elite ogre and cannons. I think it was a bug. Saddly I only got almost half of the skulls because of this.