Hey, I wanted to provide an update as it looks like I've solved the issue (Thanks to Alex at FLO).
So it seems maybe the Astroberry and my Pi had some weird udev rules for MTP errors. The rules clashed or I believe it considered the mount as something else than the FTDI.
I was having errors randomly, if it was 5mins, 10mins or hour or two. I was LOSING my mind and thinking it was the mount...