I used ~/.config/xfce4 instead of the entire .config directory
The result was that my main user can now change the font and it sticks between logins.
A later test of moving the xfce4-bup directory back to xfce4, has the main user able to retain the font settings. I'll have to do a fresh install and nvidia-driver install in order to get the issue to test more some other time.
No idea what happened this time, I couldn't reproduce it. I installed the nvidia-driver only, rebooted and the font was retained.
I do note that 16pt on a fresh install is twice the size of what 16pt is when nvidia-driver is installed. But that was not the issue. I do have a script that installs a few dozen apps and does a basic setup for me on fresh installs. The issue must be somewhere in there, yet running that script after the nvidia-driver caused no issue with the font. Running it before does.
Way too many variables for me to track down this issue. Sorry I couldn't solve it. Should someone else get this issue, I was going to see if just deleting ~/.config/xfce4 and logout/in would clear up the issue, but was unable to reproduce it this time.
Don't miss the donut by looking through the hole.