🎭 Emotion Puppeteer - SDXL LoRA 🎭

Use this model to win a 20K buzz bounty! (Ends August 31, 2024)

How to use

• Use this visual cheats sheet showing possible 96 eye-mouth combinations

  • 👁️ grieving, glaring, gazing, seething, beaming, eyes wide, winking, squinting

  • 👄 frowning, pouting, smiling, laughing, singing, screaming, sneering, snarling, grimacing, breathing, gasping, puckering, resting, relaxing

• Add one more 👁️ words and/or 👄 mouth words to your prompt

• For a stronger effect, increase the weight of the 👁️ / 👄 words, and/or increase the lora weight up to 1.2

Suggestions

• Feedback and questions welcome. Please share your results!

• Best for closeups. Most training images head and shoulders or closer

• Best with square aspect ratio. All training images were square

• Best for realism. Most training images were photos

• For character consistency, use a combination of controlnets

• Keep your negative prompt short

Other optional prompts

These words were all used in training, but their effect is weaker:

gnashing, cackling, flirting, glowering, leering, kissing, shrieking, shouting, smooching, sniffling, threatening, tisking, taunting, vomiting, wincing, yelling, accusatory, agony, amused, angry, anguish, annoyed, appalled, approving, aroused, ashamed, astonished, awestruck, bemused, bewildered, bitter, blame, certain, cheerful, cocky, concentrating, concerned, confident, confusion, contempt, contemptuous, content, crazed, crestfallen, crushed, curious, dazed, defiant, delighted, depressed, despairing, determined, disappointed, disapproving, disbelief, disgusted, dismayed, distrust, disturbed, dominant, ecstatic, embarrassed, enraged, envious, euphoric, evil grin, exasperated, excited, flirtatious, frightened, frustrated, furious, gentle, gleeful, happily, happy, hate, heartache, heartbroken, helpless, horrified, hurt, hurting, impatient, impressed, in love, incredulous, insane, insolent, intrigued, jubilant, judgemental, loathing, lustful, merry, misery, mistrust, mocking, nauseated, nonplussed, outraged, overjoyed, passionate, perplexed, petulant, pissed off, pitying, playful, puzzled, regretful, repulsed, revolted, rude, sad, scared, scorn, seductive, seizure, shaken, shocked, sickened, silenced, skeptical, smarmy, sour, startled, stunned, submissive, sultry, surprised, suspicious, terrified, thinking, ticked off, uncertain, unconvinced, unsettled, vexed, violent, wonder

How it was trained

I used ~300 high-resolution images, mostly photos, curated and captioned by hand. Sorry, I can't share the training images due to the need to protect the privacy of the subjects.

Each training image's captions included one word for the mouth type and one for the eye type. Some combinations had much higher representation in the data that others. All captions also included 2 to 4 specific matching emotions. The captions didn't include any other words. No other descriptions of the subject, physical parts of the face, background, style, or quality. Caption shuffling was used. Prior preservation wasn't. ~10K steps with default LR.