Good point. I am indeed synchronizing the borg file. I have however my script run borg check before it synchronizes to the cloud. It would be better to synchronize the source files but the encryption and deduplication are what's keeping me from doing that.
Otherwise it's still a SPOF for your backups (the borg files might corrupt, and you might be syncing that corruption to the cloud)