Try `dmesg -Tw` on the command line while swapping cables out, that may give you more information around the Pi actually detecting / allocating the USB port correctly (or not) at least.
I also recall running into weird issues where just having the cables plugged in during boot versus plugging in after would sometimes cause problems, but I wasn't able to delve into that further to figure out if that was actually the cause or not =(