This is a great question and there's a few things you can try that have worked for me: 1. Keep the refining denoise low enough to retain the facial character but high enough to add some realistic details. This can be some tricky trial and error and you don't always get the best result but it's the first thing I always try because it's so easy to do. 2. Take the same dataset that was used to create the Flux lora and create a SDXL one too. Then you can attach that lora to the refiner and it should help you retain the face better while still keeping the denoise high enough to do effective refining. If you don't have access to the Flux dataset then you can use that lora to create about 20 face portraits to use as the SDXL lora dataset. 3. If option #1 didn't work well then you can duplicate the refiner so that the image gets refined twice at a low denoise. So you'll do img2img from the Flux generation to the first refiner and then do img2img again from the first refiner to the second refiner. I would keep the denoise below 0.25 for both refiners so that they don't change the face much. As far as using Flux for the face detailer, it's certainly worth a try but I wouldn't have high hopes that you'll get a good result. It might change the face back to be closer to the character but then it will probably also lose some of the detail added by the SDXL refiner.