im actually trying to do this on a shitty game in unity. Using a custom "shader" to generate textures based on gamestate. But its all just to strange. The entire development workflow makes no sense.

I think we need some kind of "neural game engine" to make it work with this. Where all activity in the game is based on probabilistic outcomes rather than deterministic ones. Even probabalistic shaders. Technically, stable diffusion is a probabalistic shader, with the masks being polys on a flat surface. But overall, making games with this is really trippy.

We probably need to develop a model that takes "json" objects rather than text too. Its just too difficult generating pictures for a game with natural text..