This was one of the problems I had with the story as well. And all the super-weapon stuff that suddenly popped out of nowhere! And everything being Dark Side-enhanced.the silly part is the utter ease with which Luke goes from hero, to villian, and then hero again.
Btw. I don´t think those droid fighters contained the pilot´s essence, just their brains or something. Could be wrong, though...