I run my 6/1 at 650 rpm, with the spoked flywheels and an ST-3.
Without my homebrew AVR, the flicker was unbearable. With my AVR, the flicker is bearable on incandescent lights of 100 watts or more. (My shop heat lamps are 250 watts and are not bothersome at all.) Lower wattage bulbs do noticeably flicker a little even with the AVR. This is not a problem for me as I use 120VDC for my home and shop lighting, except for the 2000 watts of heat lamps which I sometimes use in the shop in the winter. I did notice the voltage variation problem on my custom designed 120VDC battery bank charger; the voltage variation showed up as DC charge current variation which I had to filter out to avoid confusing my Picaxe controller.
In comparisons using a circuit I made to look just at the low frequency AC voltage variation, my 6/1 with AVR performed comparably to a 6/1 with SOM flywheels and no AVR. You will get a bit more flicker with the ST-5 head, despite the extra rotating mass, as the larger rotor inductance makes for slower changes in output voltage. A simple AVR can only do so much, since it cannot anticipate the 10Hz power pulse, and the delay caused by the rotor inductance is significant. In theory, a microcontroller should adaptively be able to anticipate the power stroke effects and provide some of the correction early enough to allow for the lag in rotor inductance. No one makes an AVR that does this for slow speed engines, since it is such a tiny market.
More rotating weight and higher rpms are definitely a good idea, so I would vote YES, on bumping up the speed, yes on adding an AVR and yes to adding more mass to the generator shaft (1800 rpm makes that quite useful). I'd also be watching for SOM or heavier flywheels.
Some of the better quality compact fluorescent bulbs will do electronic regulation such that incoming voltage variation is of no concern, likewise some electronic ballasts and some LED lights. A design which advertises a good power factor will more likely be one which does a good job of regulation. I can't tolerate fluorescent or LED lighting so can't help much there.
Best Wishes,
Bruce