Internal storage will read/write faster than the SD card, so yes it will slow down your phone.
Now, whether or not a human will be able to tell the difference, that probably depends on the app. In many instances I suspect the time difference will be measured in milliseconds, not seconds.
On all other phones I've had you could not move widgets (or apps with widgets) to the SD card without breaking the widget. I suspect that will be true for the S4 also.