I checked the schematic here, and USB GND and other GNDs are already connected on the board. So, you don’t need to worry abouthave any isolation between grounds. Note that you will only have a "ground loop" if your power supply and USB devices have another common ground (e.g. earth) As long as your "external power source" is isolated (transfo, battery, etc.), there should be no loop.
correct the misconception, the OP wants to avoid ground loops.