After quite a bit of debate with the wife, I just attached the holders to the saddle bag lids.
I epoxy'd a piece of aluminum to the inside for extra support and screwed through it to secure the holders.

Flag Holder 2.jpg

Flag Holder 1.jpg

The holders and flags are from https://www.flagsoncars.com/motorcycle-section.html

These have worked well for the past 3 years.