Unfortunately Affinity Designer’s SVG export doesn’t export using real units; it always uses pixel units instead. It may be that you’re not actually getting 96 DPI, in which case it will not be scaled properly in the GF UI. (The settings for this are not quite as straightforward as they ought to be.)
As a quick test, try exporting to PDF and see if that loads into the GF UI at the correct size. PDFs always use real units, not the arbitrary pixel units that SVGs default to, so you shouldn’t see scaling problems with PDF.
If you can’t use PDFs due to errors regarding image masks, clipping paths, or other PDF features that the GF doesn’t yet support, there’s one other thing you can try. Set your page size to 20"×12" (50.8cm × 30.48cm). When the GF loads a file with a 5:3 aspect ratio it always interprets it as 20"×12", so if your original document is that size everything magically works regardless of DPI or other settings. It’s a weird hack but it’s useful for dealing with software such as Affinity Designer that won’t export SVGs with proper units.