Convert flat videos to VR workflow: here
PLEASE READ: for 2.0: all you have to do is upscale and apply depth via Custome nodes
for 1.0: Download my Custome nodes and use the workflow thats in it to fix the video so its usable in VR. Despite my best efforts this dosnt work right out of the box but the good news is you can just drop it in the video cropper and then apply the Depth. you'll see the nodes in my workflow.
also i created a git because i plan to update it with essential tools to make VR videos so follow me on here or my patreon for updates.
TL/DR:
my custom nodes in repo
use workflow to fix video in said repo
crop in half then apply depth.
meta data already applied if used SBS
????
profit in VR
Anyway, as always, if you like what I do and want to support the work, feel free to buy me a coffee ☕
https://ko-fi.com/aidepository35
or support me on patreon
https://www.patreon.com/c/aitrepreneur/posts
So many of you have been asking for a 180 degree version of this lora so i made one. i had a difficult time finding flat panoramas to use for the training data but there was an abundance of them in cross eyed format.
for the videos made you can just crop the video in half, upscale it, then re-apply the cross eyed depth. I'm sorry i couldn't make this an easy one click but i do plan on improving this lora in the future if it dose well.
Actually from testing it out i kind of realized i can literally just crop half of it and train another model on that. maybe i'll give it a shot later.
important links:
https://github.com/google/spatial-media/releases
https://github.com/Ragamuffin20/MuffinsVRFixes/tree/main
Rant:
Dude wtf man this lora took 4 god dam days to train and i almost gave up because my run pod timed out and i thought i lost all the data for a second. Like you guys have no idea how mad i was. 4 days in and $50 almost down the drain but luckily everything was saved on the volume disc. i really whish it would automatically create identical cross eyed videos and tbh it probably would have with more training but i cant afford it tbh. 0.40 cents an hour adds up to alot if your training something for days straight. actually funny thing, i was using the A40 to trian it and still had to toggle low vram and offloading. but yeah here yall go, hope you can make some cool things.
Description
FAQ
Comments (30)
Could you share a workflow?
its from somones patreon and i dont really wanna give it out https://www.patreon.com/c/aitrepreneur/posts
@Ragamuffin20 i watch him
i updated the description
this is sus.
does this produce actual depth parallax between the two eyes or is it flat 180 video?
I zoomed out to 50% and crossed my eyes, and didn't see any effect. When my eyes combined the two it was as if I was looking at a flat picture. Usually this is enough to see some kind of 3D effect for me, and I didn't see it.
I generated an image, duplicated it to the right, ran through fp8 distilled using this lora, opened it on my quest 2 using virtual desktop video sbs preview and it is pretty much 3D. It may go out of sync time to time and corners may not have as much depth but it works.
@iLegoLoon this node can ad the 3D depth https://github.com/SamSeenX/ComfyUI_SSStereoscope?tab=readme-ov-file i'll update the workflow later to include it
@Ragamuffin20 I think its good enough to use without a custom node.
Again;
- I'm generating a 640x1024 image
- duplicating it to the right and scaling their width down by 2
- running the sampling with the lora set to 1
It just works. Shared an example. Tho it looks way better in vr.
@iLegoLoon oh ok mb, i misunderstood the first time i read your comment and wanted to test things out before i gave a response so sorry for the delay. So ive created a workflow with variouse nodes including SBS to fix these vr videos. the workflow and some essential nodes are in my repo. one of those essential nodes can crop videos exactly in half then you can use the SBS Nodes to apply cross eyed depth or parallel. which i recommend because however you did it works but because the base images were not stereoscopic and my lora didnt convert it to be stereoscopic it just ends up being one big oversized image. like dont get me wrong the depth is there but the video dosnt look like it was meant for 180 degree if you get what i mean. but anyway ive updated the workflow in my repo so you can just Cut the video generated from my lora in half and then re-apply the cross eyed depth. i plan to make a better lora in the future so follow me on here or on patreon for more updates. thanks for the feedback
i updated the description
@Ragamuffin20 I see. Could you by any chance train a z image turbo/base lora for sbs 180vr as well? I know its way easier than training one for ltx2 and I could do it myself but you probably have enough training data already.
@iLegoLoon i actually plan on training it on mutliple other models. i only liked LTX because of its 4k capabilitys but TBH wan2.2 out performs it by miles when it comes to physics and in some cases prompt adherance
Do I understand correctly that this makes LTX generate both eyes? Wouldn’t it make more sense to generate just one eye in a 1:1 format, and then add the second eye later using something like this: https://github.com/PointerSoftware/2D-to-3D-SBS-Converter
This would also 2x the resolution of the ltx generation.
yeah i realised that halfway through training lol but in the repo that ive attached theres custome nodes u can youse to crop the video in 1:1 format (if its original in a 2:1 format) then you can take it and upscale it and apply depth to it. the only data set i had access to was the cross eyed flat panoramic but dumbass me didnt realise i couldove just cropped the videos in half and trained it on that. luckily there is a fix though.
i updated the description
Does it actually work in a VR headset?
read the discription
this is sus af, pyc files, no real info or details, I wouldn't touch this with a 50 ft pole.
pyc files? bro i dont even know what that is
@Ragamuffin20 I assume he's talking about this page https://github.com/Ragamuffin20/MuffinsVRFixes/tree/main/__pycache__
@WaifuRegen oh shit.. they where .py files when i uplaoded them. idk what happened but dont blame you. ill try to convert them to re-up them later into .py files if that helps
@Ragamuffin20 But you somehow were able to figure all this out? and you don't know what a pyc file is? lol that alone is a red flag of a claim, given the context. Use if you want, I am just saying I think it's best advice to stay clear of files that you can't check lol
@Ragamuffin20 that would be fine too, just saying lol, if it's nothing no biggie right. lol
@navyguy757 im very new to making nodes and i changed the files to .py . also i get were your coming from. if it looks sus id stay away to
@Ragamuffin20 I don't think you can just change bytecode to .py lol, I think you would need a decompiler. Idk if just changing the extension will suddenly allow you to view the code before downloading lol.
@navyguy757 no like the original files on my computer are .py i literly have no idea how they changed to .pyc
@navyguy757 also you have to use my workflow for this version. you just need to upscale and apply depth which you can do a multitude of ways
pyc are just bytecode cache files, usually created in the the folder as the executed py files, if you ran it and then uploaded the whole folder without deleting it, it's obvious they will be in there. not that sus, but still good to fix.