Add support for previewing images in room scrollback

[benjajaja]

An image says more than a thousand words:

There has been some work to get some small change into ratatui that allows for "skipping" of rendering some cells (characters). This in turn allowed to make a crate ratatui-image that provides a image widgets for ratatui, taking care of the terminal's graphics protocol automatically. Currently the supported "protocols" are sixels, kitty, and unicode-halfblocks. iTerm2's protocol should be easy to add, I just don't have access to a mac.

This PR uses ratatui-image widget's render() method to render images in the ScrollBack widget, in the chat lines. This seems hacky, but this kind of usage was actually intended and recommended for these kind of use cases in the discussion on the ratatui PR.

Currently, the download and image processing happens in one spawned tokio task per attachment. I also have a branch where only one "worker" is created, but I don't actually know if that is an actual improvement. I mainly made that branch/version to learn a bit more about tokio/tasks/threads in rust.

[justchokingaround]

really looking forward to this!

[benjajaja]

It really only works with this patch on Alacritty, which seems to work because it simply never draws over image graphics. I'm working on fixing this in tui-rs-revival/ratatui in the linked PR, by setting some area to be "skipped".

[r3k2]

This will be great. I am using foot on wayland/sway should work. please release! 👍

[r3k2]

I was happy to see movement in this PR today, how is going? do you think if I patch this myself will it be somehow a bit stable?

[benjajaja]

@r3k2 I don't know what you mean by "patch". I can give you a status update though:

• Graphics support merged in ratatui and graphics related code has been extracted to new crate ratatui-image.
• Iamb uses ratatui-image to render images with whatever protocol is detected (sixel, kitty, or unicode).
• The attachments which are "autodownloaded" for rendering are not cached in any way (this should be configurable).
• Windows builds won't work because sixel lib probably won't compile / link correctly and is missing terminal-font-size detection (rustix). We should be able to make use of the relevant feature in ratatui for font-size, but protocol-guessing is a whole 'nother can of worms. Maybe it should guess on linux, and just have a configuration setting for windows users to e.g. force sixels.

To maybe answer your question, if you want to tackle something, you could try with the cache so that images aren't downloaded over and over. There should probably be a config option to specificy a cache dir, similar to download_dir but maybe in the image_preview section.

I am trying to fix the windows builds.

[r3k2]

@benjajaja cool, who cares about windows? :D :D just release it and say only works on Linux/Nix systems for now and that windows will come.

[r3k2]

can't wait for this, thanks @benjajaja for all your work ❤️ this will prob be the first CLI/TUI matrix client with sixel and other image previews, gomuks only shows a ascii representation, it will def bump to the top "iamb" in our TUI/low/simple resources/vim-keys/ community when it comes to matrix clients

[r3k2]

I see all check passed and such, why we are not adding this already to the main IAMB? 🤔

[craftyguy]

Probably because it takes time to review complex contributions?

…

On November 13, 2023 8:00:09 AM PST, ReK2 ***@***.***> wrote: I see all check passed and such, why we are not adding this already to the main IAMB? :thinking:

[ulyssa]

As @craftyguy said, this is a big feature, and I like to make sure I understand and try out PRs locally before merging, so that I can document them correctly on the website and answer questions in #iamb-users when they come up.

I've been playing around with this locally and it's really cool! 🎉 This is amazing stuff @benjajaja ! Thank you for doing the halfblocks work, too, it'll make it possible for people without sixel-aware terminals or kitty to still get image previews.

I made a couple small changes in 164503b: I moved the picker setup code into src/base.rs so that it can run as part of ChatStore::new, and fixed a couple issues I ran into where the image_previews dir didn't exist yet, and images were missing the x offset for placing them in the right vertical split.

I see that you just pushed the updates to use ratatui-image@0.4.1 yesterday. If you have anything else you want to add to the PR, let me know. but otherwise I'm down to merge this, and take care of anything else in a follow-up PR.

[ulyssa]

@benjajaja I've pushed some example docs to ulyssa/iamb.chat#18 . Let me know if you think that anything else should be mentioned there.

[benjajaja]

@ulyssa There is nothing else I want to add for now. At some point later rataui-image will get support for iTerm2's graphics protocol (I can't test it because I don't have any mac around), but that should just be version bump.

Thank you for moving the setup code!