ggufy updates, TensorPencil actually in beta, chat!
Hello! It's been a little while since my last post... As a quick introduction, I work on some programs related to diffusion. ggufy is a format conversion tool, allowing you to convert a bf16 source file to W4A8 ConvRot Safetensors or q4_k gguf, for example. TensorPencil is a more early-stage experimental desktop program that generates images / chats with LLM's locally.
Both ggufy and TensorPencil have been migrated to Zig 0.17.
ggufy
ggufy has been through several versions, adding support for converting Qwen Image 2.1, Flux 3 Klein, and Moss-Image, as well as the W6A8 Safetensors format.
Additionally, ggufy now has support for ComfyUI's "INT8 Attention" feature. This doesn't actually change the way the weight data is stored, it simply includes some directions to have ComfyUI use INT8 attention when generating an image. This is separate from INT8 activations (the "A" in W4A8-style naming). Note that INT8 attention, while it does speed up generation, has a pretty severe precision loss.
I've also added the concept of a "target" to make the different compatibility settings easier to use. When converting, you select a target such as ComfyUI or sd.cpp, and then your file and data formats. These targets will automatically select the best options for your file. You can also use "Custom" which works the same as before, allowing you to set the options as you please. There's also a line now that informs you where your output file will work well, and any caveats. Hopefully this makes it easier to work with the increasing number of options and types.
TensorPencil
TensorPencil has had a lot of work put into it. Here's the big ones.
LoRA support
Qwen Image 2.1 support
Moss-Image support
Vastly improved Vulkan support (i.e. AMD support)
MageFlow / MageFlow Compressed support
A lot of background work to decrease code duplication and make future additions easier and more safe
Windows support (I work on Linux)
And finally, TensorPencil is officially in beta! Now, let me take a little sidebar here on why TensorPencil took so long to get to beta.
When I was going to college, I took a class that was about discovering your personality. It was very in depth, and took an entire semester. The purpose was to find out what type of a person you were so you could be more self-aware. At the end of the class, you would take a test that would define three major character traits. Mine looked like this:
Perfectionist, Perfectionist, Perfectionist.
As you can see, I have a bit of a problem with wanting things to be perfect. This manifests itself in various ways in my life, some good, some bad. Knowing it's part of me helps me to understand myself and work to overcome the downsides.
This is why I've had a hard time getting TensorPencil into beta. I want it to be perfect! And, well, it's not. I have to keep reminding myself that "beta" explicitly does not mean "perfect".
So, here it is, with all its warts.
Some notes here. TensorPencil is an extremely ambitious project, that I never would have been able to pull off without the assistance of AI. Writing performant graphics kernels is not an easy task. Writing performant graphics kernels that work across multiple systems and cards is much harder. TensorPencil has been tested on a limited subset of cards because I only have a few cards available to test on. I've tried to make the code flexible and self-routing to support as many cards as possible, but I can't test it because I don't have those cards.
Some parts of the GUI are not very refined yet. Weird options are in convenient places, often-used options are in inconvenient places. Some things are unintuitive.
I guess my point is, you will almost certainly run into issues. It's very probable that your system is not the same as any of mine, and because of that you will run into problems that I didn't. If you do, let me know, and I will try to fix it. And you can do that in...
Chat!
I've wanted to have something like this for a while so people can talk and ask questions / report issues more casually about the programs that I work on, or AI generation generally. However, I wanted to stay away from Discord; I've stopped using it entirely as the platform has gotten increasingly more invasive. I've had my eye on Fluxer for a while, which is an open-source self-hostable Discord alternative. They've recently completed support for self-hosted instances in the desktop application, so I've set one up and put it through its paces. I like it quite a bit, the interface is very familiar and easy to use.
There are a handful of people in the chat already. If you'd like to hang out or have questions, feel free to join us! https://chat.syf.me/invite/T2JOigIZ