Feature Request: sixel graphics support

Would like to see Sixel support in the Terminal, this is the standard used to show graphics in the console.
Sixel is part of the original DEC specification for doing graphics in terminals and has been re-popularized in recent years for doing graphics on the command line, in particular by Pythonistas doing data science.
The libsixel library provides an encoder but is also a great introduction to the subject (better than the Wikipedia page):
https://github.com/saitoha/libsixel

While implementing Sixel, it is important to test with images that contain transparency.
Transparency can be achieved by drawing pixels of different colors but not drawing some pixels in any of the Sixel colors, leaving the background color as it.
I believe this is the only way to properly draw non-rectangular Sixels, and would be especially nice with the background acrylic transparency in the new Windows Terminal.
Testing using WSL with Ubuntu for example, in mlterm such images are properly rendered as having a transparency mask and the background color is kept, while in xterm -ti vt340, untouched pixels are drawn black, even though the background is white, which seems to imply they render sixels on a memory bitmap initialized as black without transparency mask or alpha.

OOh. Sixel is very cool stuff.
I've decided that I need that. NEED.

I'll happily review a PR :)

Caught the Build 2019 interview today that mentioning this request. I still maintain that Xorg on sixel is just wrong. So very very wrong.
The ffmpeg-sixel "Steve Ballmer Sells CS50" demo never gets tired tho. Gotta say, it is a little disappointing the video lacks sound (sound really makes the video). Consoles already have sound, naturally. They totally beep. Precedent set. What we really need is a new CSI sequence for the opus clips interleaved with the frames, amirite?

Ken, I truly deserve this for mentioning Sixels ;)

From: therealkenc<EMAIL_ADDRESS>Sent: Wednesday, May 8, 2019 4:31:31 PM
To: microsoft/Terminal
Cc: Subscribed
Subject: Re: [microsoft/Terminal] Sixel graphics support (#448)
Caught the Build 2019https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmybuild.techcommunity.microsoft.com%2Fhome%23top-anchor&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=i8rfPCaN%2FxqdF%2F4qRtdN2Py4%2BVRlbPgpwJWtPZSGGHc%3D&reserved=0 interview today that mentioning this request. I still maintain that Xorg on sixel is just wronghttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FWSL%2Fissues%2F1099%23issuecomment-248513013&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=J%2BwCnn0z70FkI9lDcus1nMXcKz1P0ArL%2Bmdz5oi9uDo%3D&reserved=0. So very very wrong.
The ffmpeg-sixelhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsaitoha%2FFFmpeg-SIXEL&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=G%2F9mvw1EdADkwChSbHZ%2FI54k9xvXagV%2FxD9VbJtyw7g%3D&reserved=0 "Steve Ballmer Sells CS50" demohttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7z6lo4aq6zc%26feature%3Dyoutu.be&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=6IVwBHs6%2F43rXdk6GabiSUpTFS86xUGB6bubfkS3ea0%3D&reserved=0 never gets tired tho. Gotta say, it is a little disappointing the video lacks sound (sound really makes the videohttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEl2mr5aS8y0&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=Mm1ICN5KcgrP5YmdAZsUCzUKbVQDtxFE1qAEpkhKiZk%3D&reserved=0). Consoles already have sound, naturally. They totally beep. Precedent set. What we really need is a new CSI sequencehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FANSI_escape_code%23CSI_sequences&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=29pJq5661TXtnn2huLyUMgebTyYMEhTKXpAm19jzqHU%3D&reserved=0 for the opushttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FOpus_(audio_format)&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=XOq6Acz4%2B7gQeTKQBQ2fYJPnoLvx6vUjmLRhgOX1eDo%3D&reserved=0 clips interleaved with the frames, amirite?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FTerminal%2Fissues%2F448%23issuecomment-490688164&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=pnXPvsuGF7l5mQfU2htzFwJnqZjEuW4zNuh1HaBJnKM%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADNHLGXQOYKINZMIBKTB4LTPUNPFHANCNFSM4HLENFOQ&data=01|01|duhowett%40microsoft.com|81f48be19f374665cd3408d6d40d4dc6|72f988bf86f141af91ab2d7cd011db47|1&sdata=%2F4pMmm7bvPa%2BbFmE1gyN8%2BoTZDKJyRksBrkJpDh%2BLug%3D&reserved=0.

Related: #120

Need.

LOL I was watching the stream and I just thought to myself "here's my boss assigning me work live in front of a studio audience".

Please make this a priority for v1.0!

3d animations can be v1.5 😛

OMG

Upvoting this request, Sixels would be such an amazing thing to have in the Terminal.

This weekend I finished implementing sixel read support for my MIT-licensed Java-based TUI library, and it was surprisingly straightforward.  The code to convert a string of sixel data to a bitmap image is here, and the client code for the Sixel class is here.
I have done very little for performance on the decoder.  But when using the Swing backend, performance is still OK, as seen here.  (The snake image looks bad only because byzanz used a poor palette creating the demo gif.)  I was a bit taken aback how quickly it came together.  It's very fair to say that the "decode sixel into bitmap" part is the easy bit, the hard bit is the "stick image data into a text cell, and when that is present blit the image to screen rather than the character".
Just want to mention it to other folks interested in terminal support for sixel, and hoping it could help you out.

I'll upvote if someone else writes a Jupyter notebook client ;)

We already have an example of Sixel support in mintty which is written in C (vice java). Only thing needed is a refactor to C++ (at least for initial support). Still always good to see how it's been implemented in other projects.

We already have an example of Sixel support in mintty which is written in C (vice java). Only thing needed is a refactor to C++ (at least for initial support). Still always good to see how it's been implemented in other projects.

Any issues with mintty's license (GPLv3 or later)?

https://github.com/mintty/mintty/blob/master/LICENSE

https://github.com/mintty/mintty/blob/master/LICENSE

From that link:

Sixel code (sixel.c) is relicensed under GPL like mintty with the
permission of its author (kmiya@culti)

If you transliterate that exact code to C++, the derivative work would need to be licensed GPLv3 or later, as per its terms, or not distributed at all.  (One could also ask kmiya@culti if they are willing to offer sixel.c under a different license, or if it was once available under something else find a copy from that source.)
I don't know what is acceptable or not for inclusion in Windows Terminal -- my quick glance at Windows Terminal says it is MIT licensed, so depending on how it is linked/loaded using a direct descendant of mintty's GPLv3+ sixel.c could lead to a license issue.
Anyway, sorry to be bugging someone else's project here, heading back to the cave now...

There is a sixel capable, humble terminal emulator widget written in C/C++ for Windows/Linux, and it has a SixelRenderer class  which you can use,  and it has a BSD-3 license. Arguably its biggest downside is that it is written for a specific C++ framework. Still, IMO the SixelRenderer's code is translatable with little effort. (I know this because I am its author. :) )
https://github.com/ismail-yilmaz/upp-components/tree/master/CtrlLib/Terminal

Hey here are some relevant links for research:

"Basics for a Good Image Protocol" on terminal-wg (and a linked earlier discussion)
this massive thread about sixel support

We could add support for iTerm2 Inline Images Protocol once we are there... At least it should be easier to implement, it just only need a path to the image and does everything on its own.

That probably should be a different task. Sixel and ReGIS are explicitly for in-band graphical or character data. I'm not saying it's a bad idea, I'm just saying it should be treated as a different feature.

One doubt I have with both systems is, what happens with aligment? If images width or height are a multiple of chars width or height everything is ok, but if not, should a padding be added only in lower and right sides, or should image be centered adding padding to all sides?

Alignment of Sixel and ReGIS graphical data is described (poorly) in various manuals. Sixel images are aligned on character cell boundaries.  Each line of sixel data describes a stripe six pixels high. If you're trying to align sixel image data with text characters on a terminal emulator, this can be frustrating as the software generating the sixel data may not know how many pixels high each character glyph is. If you have an old-school xterm handy, you can see this by starting it up in vt340 mode, specifying different font sizes (to give you different character cell sizes) and then printing out some sixel data that tries to align image data with text data. (Here's a simple test file that looks correct when I tell the font server to use 96DPI and I specify a 15 point font. Modifying the font size causes images to increasingly come out of alignment with the text. https://gist.github.com/OhMeadhbh/3d63f8b8aa4080d4de40586ffff819de )
The original vt340s didn't have this problem because (of course) you didn't get to specify a font size when turning the terminal on.
The other thing you can see from that image, that isn't well described in the sixel documentation is that printing a line of sixel data establishes a "virtual left margin" for the image data. If you do the moral equivalent of a CR or CRLF using the '$' or '-' characters, the next line is printed relative to this virtual left margin, not the real left margin at the left side of the terminal.
Hope this helps.

Finally scrolling back to read this. Sorry for the tardy reply.

Testing using WSL with Ubuntu for example, in mlterm such images are properly rendered as having a transparency mask and the background color is kept, while in xterm -ti vt340, untouched pixels are drawn black, even though the background is white, which seems to imply they render sixels on a memory bitmap initialized as black without transparency mask or alpha before blitting them into the terminal window.

It shouldn't be too hard to support transparency in xterm. I've been digging around in the code for other reasons. I fear that someone, somewhere is depending on this behaviour of Xterm so would recommend putting it behind a compatibility flag, which also should be straight-forward. But then there's the question of the default value. What should be the default? Black or transparent.
Do we know what the original VT240, 241, 330 and 340's did? Could I suggest trying to faithfully represent the experience of an actual VT as the default behaviour? You could test this by printing inverted space characters, then layering sixel graphics above them and seeing what color unspecified pixels render as.
I don't know that I care too much what the default is for the msft terminal as long as there's the capability of behaving like Xterm emulating a VT340. The code I've written to do loglines over ssh in the terminal sort of assumes the behaviour above. I'd have to rewrite that code if we make this change.

If you're trying to align sixel image data with text characters on a terminal emulator, this can be frustrating as the software generating the sixel data may not know how many pixels high each character glyph is.
The original vt340s didn't have this problem because (of course) you didn't get to specify a font size when turning the terminal on.

Is there any reason why a terminal emulator couldn't just scale the image to exactly match the behaviour of the original DEC terminals? So if the line height on a VT340 was 20 pixels, then a image that is 200px in height should cover exactly 10 lines, regardless of the font size. That seems to me the only way you could remain reasonably compatible with legacy software, which is kind of the point of a terminal emulator.
I can understand wanting to extend that behaviour to render images at a higher resolution, but that should be an optional extension I think (or just use one of the existing proprietary formats). So ideally I'd like the default for Sixel to be as close as possible to what you would have gotten on an actual DEC terminal.

Hey here are some relevant links for research:
"Basics for a Good Image Protocol" on terminal-wg

It took some work (actually a lot of work), but with sixel one can perform nearly all of the "images in a terminal" tricks one can image:

Layered per-cell-masked images in a terminal: https://jexer.sourceforge.io/images/sixel_many_images.png

A floating (multiplexed) terminal window in a terminal that is using sixel for VT100-style double-width support: https://jexer.sourceforge.io/screenshots/jexer_sixel_in_sixel.png

"tmux-style" tiled terminals with images: https://gitlab.com/klamonte/jexer/-/wikis/uploads/7603381f82414ef9ae214bfcf759c064/example_tilingwm2_1.png

Multi-headed shared terminal session with differing text cell sizes showing the same plot: https://jexer.sourceforge.io/screenshots/multiscreen_2b.png

The use of sixel to render CJK and emoji that are not present in the main terminal's font: https://jexer.sourceforge.io/screenshots/xterm_sixel_cjk.png

I have included some other remarks at the referenced "good" protocol thread that might be of interest.
If nothing else, sixel is a good stepping stone to working out the terminal side infrastructure of mixed pictures-and-text.  Speaking from direct experience, the terminal side (storing/displaying images) is about 1/4 as hard as the multiplexer side (tmux et al).

sixels are indeed the ideal solution for in-band graphics (for example over ssh): as they are supported by many existing tools, they are ready to use for practical purposes like plotting timestamp sync issues on the go.
As illustrated by therealkenc and further explained by klamonte in 640292222 everything can be handled with sixels, even side-by-side images, but it requires some work.
A while ago I was working with a few other people on a fallback mode for tmux, using (advanced unicode graphics) [https://github.com/jart/derasterize] to represent sixel images in terminals that do not support sixel.
It is a bit like automated ANSII art, taking advantage of special block characters that are present in most fonts: this equivalent color unicode representation could be substituted for the sixels, then later overwritten by the actual sixel image (or not!). It would also solve the problem of keeping all the sixel pictures for scrolling back, by substituting them with low fidelity unicode placeholders (for ex to save memory), and having placeholders for sixel images when they can't be displayed for whatever reason.
The code was public domain. It could be usable immediately as a first step towards sixel support:

detect when sixels sequence are transmitted, then compute the unicode text replacement

diplay this unicode sequence, which is already supported by Windows Terminal

later, when sixels are implemented, render on top the sixel sequence.

Would you be interested?
BTW I recognize here my familiar gnuplot x^2 sin and 10 sin(x) plots I'm happy it provided some inspiration 😄

Please.

@DHowett  Is acac350 a first step toward actually rendering sixel graphics? I'm getting requests for sixel support in Microsoft Terminal from folks using ssh and wanting to view directories of images using my lsix program.

Here's some updates. I have a working branch here. An early screenshot looks like this:

Contrary to what I originally thought, the most difficult part of rendering sixel images is actually the conpty layer. Sixel images are supposed to be inline objects. The rendering of sixel images depends on the rendering size of a character. However due to the extra conpty layer we actually can not get the rendering size of a character when processing sixel sequences. This sounds very abstract and vague. Anyone who's interested in this can checkout my branch and see how it's done.
Overall, the conpty layer makes it very difficult to handle scrolling and resizing of sixel images. In my branch it works if you only need to display it. But both scrolling and resizing are completely broken.

Didn't look yet but can you use pass-through mode to implement in Terminal itself? I would still add it in OpenConsole but sounds like sharing code isn't possible. Since Windows Terminal needs to be decoupled from OpenConsole at some point,  you're best off simply duplicating the code for both. Also are you basing it on yours and j4james PRs for parameters? That would likely help as well.

@WSLUser Thanks for the attention. This screenshot is actually from about a month ago, when the fantastic parameters PR from j4james does not even exists. My work is entirely inside Windows Terminal, not conhost. I showed this PR to the Console team internally and made some progress since then. But I'm stuck because of the conpty problem.

Yeah I'd rebase off of master and add https://github.com/microsoft/terminal/pull/7578 and https://github.com/microsoft/terminal/pull/7799. From there, maybe see what's missing in ConPTY for pass-through mode. I wonder Mintty is using pass-through for ConPTY mode.

I wonder Mintty is using pass-through for ConPTY mode.

Pretty sure mintty isn't using conpty at all 😜

The trick here with conpty is that the console (conpty) will need to know about the cells that are filled with sixel contents, as to not accidentally clear that content out from the connected Terminal. Maybe conpty could be enlightened to ignore painting cells with sizel graphics, and just assume that the connected Terminal will leave those cells alone.
That might mess up some of our optimizations (like we can't EraseLine rows that have sixel data), but it might be a good enough start
</showerthought>

Maybe conpty could be enlightened to ignore painting cells with sizel graphics, and just assume that the connected Terminal will leave those cells alone.

This had been my original plan as well, and it may well be the best solution with the current conpty architecture, but there are a number of complications.

How would this interact with DCS streaming (which I don't think we've even got a solution for yet). I'm assuming we'd need some kind of split stream concept that passed the byte stream through to conpty at the same time as it's sent to the conhost buffer, but that seems like it would add a lot of unnecessary overhead to the process.
This would only work if you know the pixel cell size of the conpty terminal. I've mentioned before I think the best solution for Sixel is to match the cell size of the original VT terminals, and if we were doing that this wouldn't be an issue. However, as far as I'm aware, no other terminal emulators do that, so it wouldn't work with anyone else.

The second issue @j4james brought up becomes even more complicated with the consideration of different font, different font size and font resizing. So generally I think there's 3 aspects of the issue:

First conpty will need to know about the cells that are filled with sixel contents, Without this, the backing buffer in conpty and the drawing buffer in WT will be inevitably out of sync.
In order to do that, conpty will need to know pixel cell size in the drawing context, which is handled by the drawing layer in WT.  There is a huge gap between conpty and the actual DXRenderer, which makes this a difficult task.
Besides, when the font or the font size changes, ideally the sixel image should change correspondingly.
And finally deal with other things like pane, alternative buffer, differential drawing, scrolling, etc.

The second issue @j4james brought up becomes even more complicated with the consideration of different font, different font size and font resizing. So generally I think there's 3 aspects of the issue:

Just to be clear, my point was that none of that would be a problem if we exactly matched the behaviour of a VT340, so a 10x20 pixel image would occupy exactly one character cell, regardless of font size. It's only an issue if we want to match the behaviour of other terminal emulators, and that could always be an option that is left for later. There would still be complications with this approach, but I personally think they're less of a concern.
My bigger concern is that you seem to be ignoring the DCS streaming issue, which I expect could fundamentally change the architecture of the solution. The steps I would like to have seen are: 1. Resolve #7316; 2. Agree on a solution for cell pixel size; 3. Get something working in conhost; 4. Once all the complications are worked out in conhost, only then consider how we make it work over conpty.

Sorry for leaving the DCS streaming issue. In my current implementation I just store the entire string and pass it to the engine. This introduces performance issue when the sequence is larger. But at least it works. So my comments above are largely based on it.
But you are right. The DCS streaming issue is actually the top priority if someone else want to get their hands dirty on this.
获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: James Holderness<EMAIL_ADDRESS>发送时间: Saturday, October 10, 2020 8:47:15 PM
收件人: microsoft/terminal<EMAIL_ADDRESS>抄送: Chester Liu<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>主题: Re: [microsoft/terminal] Feature Request: sixel graphics support (#448)
The second issue @j4jameshttps://github.com/j4james brought up becomes even more complicated with the consideration of different font, different font size and font resizing. So generally I think there's 3 aspects of the issue:
Just to be clear, my point was that none of that would be a problem if we exactly matched the behaviour of a VT340, so a 10x20 pixel image would occupy exactly one character cell, regardless of font size. It's only an issue if we want to match the behaviour of other terminal emulators, and that could always be an option that is left for later. There would still be complications with this approach, but I personally think they're less of a concern.
My bigger concern is that you seem to be ignoring the DCS streaming issue, which I expect could fundamentally change the architecture of the solution. The steps I would like to have seen are: 1. Resolve #7316https://github.com/microsoft/terminal/issues/7316; 2. Agree on a solution for cell pixel size; 3. Get something working in conhost; 4. Once all the complications are worked out in conhost, only then consider how we make it work over conpty.
―
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/terminal/issues/448#issuecomment-706544319, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABD6BLZMHQZOSSVYVLQMYITSKBJVHANCNFSM4HLENFOQ.

Per discussion in https://github.com/microsoft/terminal/issues/57, I thought conpty doesn't care about fonts at all?

@yatli Yes. That's also what makes the issue tricky.

10x20 pixel image would occupy exactly one character cell

This is unfortunately wrong, at least for my current font setting.
Correct me if I'm wrong, but for pixel perfect image display, I think we do need to care about fonts.

@skyline75489 pls see my updated comment about the "anchor"

The cell data structure needs to be updated as char | sixel anchor

It's a good idea but the implementation details were killing me, due to the extra translation in conpty layer. To avoid spamming people with email, feel free to reach me on Teams @yatli if you're interested.

10x20 pixel image would occupy exactly one character cell

This is unfortunately wrong, at least for my current font setting.

What I'm suggesting is that you should make that the case. If you create a 10x20 pixel image and output it on a real DEC VT320 terminal, it's going to take exactly one character (at least in 80 column mode). So if we're trying to emulate that terminal, then we should be doing the same thing. If your current font happens to be 30x60, then you need to scale the image up. If your font is smaller, then you scale the image down.
This guarantees that you can output a Sixel image at any font size and always get the same layout. If you want it to cover a certain area of the screen, or you want to draw a border around it with text characters, you know exactly how much space the image will occupy.

Correct me if I'm wrong, but for pixel perfect image display, I think we do need to care about fonts.

It's true that you're not going to get "pixel perfect" images this way, but I don't think that should be the primary goal. Many modern computers have high dpi displays where it's routine for images to be scaled up, so it's not like this is a strange concept. And if we want to keep the layout consistent when the user changes their font size, we're going to have to scale the image at some point anyway, so you might as well do it from the start and get all the benefits of a predictable size.
And of course the other benefit of doing things this way is that it could feasibly be implemented over conpty. I don't see how you can make conpty work if the area occupied by the image is dependent on the font size, which you can't possibly know.
I'm not going to pretend this approach won't have any downsides, but I think the positives outweigh the negatives.

What if the font has a different aspect ratio than 10:20?

What if the font has a different aspect ratio than 10:20?

May I suggest reading this long - and somewhat "brutal"- discussion about the general problems regarding the inline images in terminal emulators.
It can give you the general idea.
Best regards

What if the font has a different aspect ratio than 10:20?

The image may be a bit stretched or squished, but I don't think that's the end of the world.
Let me demonstrate with a real world example. Imagine I'm a Bond villain, and I've got an old security system using a VT320 as the frontend. Now because of the coronavirus, I'm in lockdown and working from home, so I'm logging into the system remotely with Windows Terminal. If we exactly match the VT320 this is no problem - the terminal looks like this:

But maybe I prefer fonts with a weird aspect ratio. So let's see what it would look like with Miriam Fixed, which is wider than most. The image of Bond now looks a bit squished, but he is still easily recognisable.

The alternative would be to go with a pixel perfect image (not currently feasible with conpty, but let's pretend for a second). Bond no longer looks squished, but now the image is only a fraction of the size it was expected to be. And the higher the resolution of your monitor, the worse this is going to look.

Maybe this is a matter of personal preference, but I know I'd definitely choose option 1 over option 2.
Also note that there is no reason we couldn't have options to tweak the exact behaviour when the font aspect ratio isn't 1:2. One option could be to center the image within the cells it was expected to occupy. Or we could expand the image so it covers the full area, but clip the edges that overflow the boundaries. Any of these choices would be better than an exact pixel rendering in my opinion.

Maybe this is a matter of personal preference, but I know I'd definitely choose option 1 over option 2.

Me too, just only it would be better to know the font has a different aspect ratio, so image can adjust itself and keep the correct one.

One option could be to center the image within the cells it was expected to occupy. Or we could expand the image so it covers the full area, but clip the edges that overflow the boundaries

I think it's better to center them.

Maybe I'm misreading this thread. Are we actually talking about the terminal faking 10:20 characters for sixel image? I think that will cause many problems like the Bond distortion. Doing it the right way may be more difficult, but, in my humble opinion, a modern terminal should be font agnostic and leave it up to application programmers to deal with sixels and character cells.
Using escape sequences a user run program can determine the character cell size in pixels and decide how to intelligently deal with distortion for that application. The image viewing program I use works exactly like that. As I change font family or size, the displayed thumbnail updates to always be precisely five text lines high. The width is scaled proportionally for the image, unless it would be larger than a certain (in this case, rather large) maximum. By basing the image size on the character cell, it works automatically on high-DPI screens.
While the VT340 is a noble goal to emulate, fixing character cell resolution at 10:20 (and thus limiting resolution for the entire screen) is a mistake. The VT340 was only one of several sixel implementations, so its font size isn't necessarily more correct.
 (E.g., how to respond to a request for the size of the terminal window in pixels. Tell the truth, presuming they'll be positioning windows on the screen? Or, always return 800x480, presuming the user is scaling images for sixel output?)

Are we actually talking about the terminal faking 10:20 characters for sixel image?

Yes.

a modern terminal should be font agnostic

This proposal is font agnostic. The application doesn't need to know anything about the font. That's the whole point.

Using escape sequences a user run program can determine the character cell size in pixels and decide how to intelligently deal with distortion for that application.

I'm not exactly sure what method you're using, but the way I've seen this done before is with a proprietary XTerm query to get the window pixel size, and another query to get the window cell size, and then using that data to calculate the actual cell pixel size. The downsides of such an approach are:

It's proprietary, so wouldn't work on a real terminal, or any terminal emulator that exactly matched a real terminal.
If the user changes their font size while your application is running, then your calculations will no longer be correct, and images will be rendered at the wrong size (unless you're continuously recalculating the font size which seems impractical).
If the user has a high resolution display, and/or large font size, you're forced to send through a massive image to try and match that resolution. Considering how inefficient Sixel is to start with, that can amount to a lot of bandwidth.

That said, I understand that this is a mode that some people may wish to use, and I think we should at least have an option to support it one day (for reasons discussed above, this just isn't possible at the moment). But in my opinion, this is not the best approach for Sixel.

I have 300+ VT340's in nuclear power plants that I would like to eventually
replace.
There are commercial terminal emulation packages we could use, but I think
all but one have been EoL'd.
We have replaced some of them with Linux PCs running XTerm (or less
frequently, Win10 + Hummingbird + WSL running XTerm), because it has a
half-way decent open source sixel implementation and a sort of bad, but
open sourced ReGIS implementation.
The likelihood that we will be writing new software for the part of this
system that generates the sixel octet stream is NIL.
If your objective is to send graphics over an inline octet stream, there
are other options. But if you want to support sixel graphics, you should
support sixel graphics in a way that is halfway similar to previous
implementations. This, unfortunately, means you should emulate the
behaviour of exemplar systems (i.e. VT240, VT241, VT330 and VT340
terminals) even when it comes to integrating graphics with text.
This is a mock-up of the kind of thing I'm talking about. It would be very
nice if any new Sixel implementation maintains compatibility with existing
implementations so images do not run off the edge of the screen or only
fill half the screen.
https://vimeo.com/user32814426/review/467991744/ac5892fa7e
meadhbh hamrick * it's pronounced "maeve"
@OhMeadhbh * http://meadhbh.hamrick.rocks/ *<EMAIL_ADDRESS>Sent from my TRS-80 Model 102
On Tue, Oct 13, 2020 at 2:29 PM hackerb9<EMAIL_ADDRESS>wrote:

Maybe I'm misreading this thread. Are we actually talking about the
terminal faking 10:20 characters for sixel image? I think that will cause
many problems like the Bond distortion. Doing it the right way may be more
difficult, but, in my humble opinion, a modern terminal should be font
agnostic and leave it up to application programmers to deal with sixels and
character cells.
Using escape sequences a user run program can determine the character cell
size in pixels and decide how to intelligently deal with distortion for
that application. The image viewing program I use works exactly like that.
As I change font family or size, the displayed thumbnail updates to always
be precisely five text lines high. The width is scaled proportionally for
the image, unless it would be larger than a certain (in this case, rather
large) maximum. By basing the image size on the character cell, it works
automatically on high-DPI screens.
While the VT340 is a noble goal to emulate, fixing character cell
resolution at 10:20 (and thus limiting resolution for the entire screen) is
a mistake. The VT340 was only one of several sixel implementations, so its
font size isn't necessarily more correct.
 (E.g., how to respond to a
request for the size of the terminal window in pixels. Tell the truth,
presuming they'll be positioning windows on the screen? Or, always return
800x480, presuming the user is scaling images for sixel output?)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/terminal/issues/448#issuecomment-708019977,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE2756OOELGSJWOXZ435J3SKTBFJANCNFSM4HLENFOQ
.

a modern terminal should be font agnostic

This proposal is font agnostic. The application doesn't need to know anything about the font. That's the whole point.

I meant the terminal should be font agnostic instead of imposing 10:20 on every font. The application should be able to know the actual font size, if it wishes, since it's the application that knows the domain of what it is trying to show and can figure out the best way to present text and graphics together.

Using escape sequences a user run program can determine the character cell size in pixels and decide how to intelligently deal with distortion for that application.

I'm not exactly sure what method you're using, but the way I've seen this done before is with a proprietary XTerm query to get the window pixel size, and another query to get the window cell size, and then using that data to calculate the actual cell pixel size.

Yup, that's about right. There's also a query to directly get the character cell size, but I don't think that's as widely supported as just getting the screen size and dividing by ROWS and COLUMNS.

The downsides of such an approach are:
1. It's proprietary, so wouldn't work on a real terminal, or any terminal emulator that exactly matched a real terminal.

That's not a downside. It only means the program has to fall back on doing what it would have done anyway: presume $TERM=="VT340" means character cells are 10:20, "VT240" means 10:10, "mskermit" means 8:8, and so on.
Also, it's not an xterm proprietary sequence. Getting the screen size is called a "dtterm" escape sequence, but it was actually first implemented in SunView (SunOS, 1986). I believe it was later documented in the PHIGS Programming Manual (1992). Try sending "\e[14t" to a few terminal emulators and you'll see it is widely implemented.

2. If the user changes their font size while your application is running, then your calculations will no longer be correct, and images will be rendered at the wrong size (unless you're continuously recalculating the font size which seems impractical).

This is not a problem. The program simply traps SIGWINCH and only recalculates if the window has actually changed.

3. If the user has a high resolution display, and/or large font size, you're forced to send through a massive image to try and match that resolution. Considering how inefficient Sixel is to start with, that can amount to a lot of bandwidth.

Yes, sixel is extremely inefficient. But on modern computers, sending full screen images is quite usable, even over ssh. Does the Microsoft Terminal have some sort of baudrate limitation?
By the way, I believe sixel does have a "high DPI" mode where every dot is doubled in width and height. I've never used it and I don't think xterm even implements it, but perhaps that would alleviate concerns about bandwidth.

That said, I understand that this is a mode that some people may wish to use, and I think we should at least have an option to support it one day (for reasons discussed above, this just isn't possible at the moment).

This "mode" is simply having characters and graphics aligned just like the various historical sixel terminals did and current emulators do. I admit, I don't understand why it is not possible to do the same in Microsoft Terminal. If you say this 10:20 kludge is the best that can be done, I will trust that you are correct and thank you for doing it. A distorted picture is much better than nothing.

Using escape sequences a user run program can determine the character cell size in pixels and decide how to intelligently deal with distortion for that application.

@hackerb9, what's the actual escape sequence to get the font dimensions?

The relevant XTerm sequences can be found here: https://invisible-island.net/xterm/ctlseqs/ctlseqs.html -- look for XTWINOPS.
Additionally, on Unix you can typically get the terminal's internal pixel size along with the cell size using the TIOCGWINSZ ioctl. With openssh this works remotely too.
Just as a data point, the sixel branch for libvte is taking the cell size-agnostic route @hackerb9 is talking about. It treats incoming sixel data as "pixel perfect" and rescales previously received images across zoom levels and font sizes to cover a consistent cell extent. When merged, this implementation will be available to a large share of Linux terminal emulators, including GNOME Terminal, the XFCE Terminal, Terminator, etc. Superficially this seems to be interoperable with at least XTerm and mlterm.
Since libvte records a per-image virtual cell size, it'd be trivial to make this work with a fixed virtual 10x20 cell size too for interoperation. However, we'd need a way for programs to communicate their expected pixel:cell ratios to the terminal (e.g. by extending the DCS parameters). That could be very useful in general, since it'd also provide a form of pixel density control in bandwidth-constrained environments, as you touched on above.

Additionally, on Unix you can typically get the terminal's internal pixel size along with the cell size using the TIOCGWINSZ ioctl. With openssh this works remotely too.

Linux console returns always 0... they should fix that, though, but seems are not willing too :-/

What if the font has a different aspect ratio than 10:20?

The image may be a bit stretched or squished,
(...)
The alternative would be to go with a pixel perfect image (not currently feasible with conpty, but let's pretend for a second). Bond no longer looks squished, but now the image is only a fraction of the size it was expected to be. And the higher the resolution of your monitor, the worse this is going to look.
Also note that there is no reason we couldn't have options to tweak the exact behaviour when the font aspect ratio isn't 1:2.

Actually, there are 2 reasons: Windows Terminal supports neither TIOCGWINSZ nor OSC 14

One option could be to center the image within the cells it was expected to occupy. Or we could expand the image so it covers the full area, but clip the edges that overflow the boundaries. Any of these choices would be better than an exact pixel rendering in my opinion.

This should not be left solely under the control of Windows Terminal: applications have ways to introspect the terminal properties, and adapt their behaviours. If they can't do that, the terminal implementation is broken.
Currently, any software outputting sixels can't introspect the size on Windows Terminal, so it can't adapt the size of the sixel images it sends to the font being used by the terminal. However, with the number of rows, columns, and the x,y size of the terminal in pixel, this is easy to do - and I would be surprised if the power plant monitoring software used as an example in https://github.com/microsoft/Terminal/issues/448#issuecomment-708127508 by @OhMeadhbh didn't already do that.
Problem is Windows Terminal doesn't return the correct values through TIOCGWINSZ, and doesn't support either the OSC14t query, so there's no way to make t work.
I opened a separate issue and referenced #448, as properly returning the windows size would help a lot.
As @hackerb9 pointed out in https://github.com/microsoft/Terminal/issues/448#issuecomment-708184522 :

Try sending "\e[14t" to a few terminal emulators and you'll see it is widely implemented.

Indeed, XTWINOPS is extremely basic and expected to be working.
As @hpjansson mentionned in https://github.com/microsoft/Terminal/issues/448#issuecomment-708433602

Additionally, on Unix you can typically get the terminal's internal pixel size along with the cell size using the TIOCGWINSZ ioctl. With openssh this works remotely too.

I know, for I use that in production, and Windows Terminal not supporting either method breaks things.

I have 300+ VT340's in nuclear power plants that I would like to eventually replace. There are commercial terminal emulation packages we could use, but I think all but one have been EoL'd.

I don't know if it's the right place to say that, but I offer this kind of services.
So would you like to get in touch?

It would be very nice if any new Sixel implementation maintains compatibility with existing implementations so images do not run off the edge of the screen or only fill half the screen. https://vimeo.com/user32814426/review/467991744/ac5892fa7e

I have specifically written software that does just that: making sure whatever sixel sequence is sent will be correctly displayed, resizing it if needed to make the sixels fit properly.
If your budget is limited, I will soon release a new version of tmux-sixel that includes some of these features.
For example, it supports sixels in multiple panes (of course):

But it also supports scrolling, with textmode renditions to make it fast:

These renditions also let tmux-sixel provide a failback mode, to allows terminals that are not sixel aware (like current Windows Terminal and the various libvte terminals) to still "see" what the sixels represent:

It's multi user, because it's tmux: both the user of the sixel-aware terminals and of the "regular" terminals are attached to the same session, which is shared by tux: they can all work cooperatively, and they see the same thing.
Of course, text mode can't perfectly replace images, so it's not perfect, but still it can go quite far: this is what just 161 columns and 41 rows can give - check the result of TIOCGWINSZ below

If you are uncertain it may fit your need, and first want to test the performance on your images, that part is open source, you can try it on https://github.com/csdvrx/derasterize

@csdvrx oh dear, that looks awesome. You have done some really great work! I'd like to avoid some too early advertisement here, but how can I get in touch with you on very similar matters? You can email me<EMAIL_ADDRESS>with a very short "Hi" or so, so I can click on Reply. Sorry for the interference, but Github doesn't have private messaging yet (does it?) :-)

I think It has, but if not, you can always look for the other user email on their profile page :-)

@csdvrx looks like you've made major breakthrough on this -- congratulations!
Haven't revisited my branch since 2017 but I think now is the time!

@csdvrx looks like you've made major breakthrough on this -- congratulations!

Thanks! For the fallback mode, the next step is adding back fine details that have been stripped.
For example, you can infer the presence of the lemur hair under its chin, but you can't see them nicely. This is due to the loss of what are low frequencies in the spectral domain (FFT), as can be seen more easily if you run derasterize on https://github.com/csdvrx/derasterize/blob/master/samples/wave.png which I use for tests: it's simply a simulated collimator interference pattern, but it saves you the trouble of comparing spectrograms of inputs and outputs images.

Then it's obvious that the loss in more present at some frequency bands and following some vectors, and of course more extreme in lower resolution (Shannon-Nyquist, duh!)

My goal is to improve the result mostly in the diagonal and vertical, as the loss there is due to the set of characters chosen.
But you can't just try every Unicode glyph while looking for a better fit, as the problem is combinatorial explosion: if you want to keep some applications (ex: playing videos), you can't simply multiply the combining characters and the test character set, then test everything for every frame. Even with serious optimization, it's too slow.
The 2 best approaches seems to be 1) a band pass filter to concurrently select the character and the combining character 2) another pass sequentially, testing fitness improvement of just adding combining characters.
The difference in thinness between combining characters and regular characters make that perfect for a band pass approach, but it will require some calibration, so I'm partial to (2) as this would guarantee there's no regression: either the combining character improves the fitness score and it's added, or it doesn't and the current results remain.
Anyway, I will also release new versions of derasterize soon, with a new set of glyphs and a better optimized color picker. Ascii Art is becoming my new favorite Christmas activity :-)

Haven't revisited my branch since 2017 but I think now is the time!

Thanks a lot for your great work on tmux, which inspired me a lot!
But before revisiting your branch, could you please wait a bit to allow me to be done with the code review on mine?
This way, you can fork it!
Another thing someone may want to do is to release another https://github.com/saitoha/xserver-sixel : when sixel support comes to the Windows Terminal, it will be possible to run X inside, on vanilla Windows 10. This would be a major advancement for WSL.

But you can't just try every Unicode glyph while looking for a better fit, as the problem is combinatorial explosion: if you want to keep some applications (ex: playing videos), you can't simply multiply the combining characters and the test character set, then test everything for every frame. Even with serious optimization, it's too slow.

Sounds like a small scale convolution neural network would do...
Get some ground truth pics. Chop it into character-sized tiles. For each tile, search for the best representation (slow is ok for training). Throw a tile into the NN and force it to generate the good representation. Done (
(Just my random thoughts at 3AM)

But before revisiting your branch, could you please wait a bit to allow me to be done with the code review on mine?

Sure please go ahead!

It's a bit crazy that you are trying to get high resolution images on the terminal using unicode characters, and in https://piranna.github.io/tty.css/ the most difficult thing I'm facing off is to get images to look low-res like they are shown in a terminal with block characters 😅

Sounds like a small scale convolution neural network would do...
Get some ground truth pics. Chop it into character-sized tiles. For each tile, search for the best representation (slow is ok for training). Throw a tile into the NN and force it to generate the good representation. Done

It's am interesting idea: basically, selecting the most advantageous 'base blocks' using the equivalent of a LUT/rainbow table - except it will be embodied in code.
However, it would require a separate training step, which would have to be redone whenever new glyphs are added.  More on that below.

(Just my random thoughts at 3AM)

These are good thoughts!
The core problem of encoding images is not new. What's new is the specific limitations that are imposed by going to a text format:

the set of glyphs is limited (depending on the terminal limits: unicode>ascii),

the resolution of each axis is asymmetric (as most characters are taller than wider, unless we go to CJK double width, but this piggybacks on the previous limitation)

each glyph can only take 2 colors, selected in a colorspace also dependent on the terminal limitations (24bpp > 256 color > 16 color > no color)

In a way, the problem reminds me of the history of TV image encoding: since some information is more important (luminance), it historically constrained the design: black and white TV was first, just like ASCII art was black and white first. In a way YPbPr is analogous to ANSI colors: you "spice" the most important signal with some extra eye candy, that can be safely ignored by limited decoders, so that at least you can get the basic signal, the most important part.
I've thought about different approaches, they all have their drawbacks:

black box approaches may limit future evolutions,

look up tables/rainbow dictionary are big,

some algorithm while nice on paper are not suited to the problem (more on that below)

On top of that, quality and speed optimize in opposite directions, that may limit some usecase (ex: playing a youtube video in unicode in real time). Simple solutions have their places, but they may divert us away from the bigger picture. The ideal approach should be versatile or at least flexible.
A memoization like approach is currently being introduced, but in a different way: the public version of derasterize uses a 4x8 block. Unicode characters were selected based on this limit. The version that will soon be released uses a 128 bit for the 8x16 block decomposition canvas, to introduce more glyphs.
Obviously, extending to 8x16 increases the runtime needlessly for glyphs that are equivalent in 4x8 and 8x16: a typical example is the half-blocks, like U+2584 lower half block and U+258C left half block.
The approach selected is not really a memoization: it works the other way around! The input image is downsampled to 2 different resolutions, and the algorithm uses both downsampled version for computing the fitness of the individual blocks as it iterates through the glyphs: when it hits those that are amenable to a simpler computation (like the halfblocks), it uses the downsampled version for computing the fitness score on 32 bit, as it will be faster than doing that on 128 bits!
In a way, it's like a binary decision diagram, except it's at the algorithm level.
I've been considering introducing other refinements. Fine lines using combining characters are not just for lemur hairs (!!), but for a big use case: graphs. This is because you don't want the thin lines to become blurry and be lost in the details of block decomposition.
But there are many other low hanging fruits!
Here's another simple example: circles may be worth preserving, because they are extremely important in human perception. And it can be fast too, if you use the right tools from historical computer vision: a few Hough transforms could quickly isolate the circles.
And I don't just want to only add the combining characters on the premade blocks in a second step (on top of my head U+0307, U+0308, U+030A, U+0323, U+0324, U+0325, U+0359, U+035A, U+0360 cf https://en.wikipedia.org/wiki/Combining_character) like for the fine details of the lemur hair. I mean introducing the set of round glyphs:   ⴰ  ⸰ ° ᴼ o ᴑ ○ O O ◯ ⚪
The NN approach could certainly dealt with that, except if circles fall on the border between blocks during the decomposition step... then, they would be ignored.
However, maybe it's worth tolerating a few pixels of difference in the circle center? And that's even if that would totally ruin the fitness score when it's computed at the pixel level. Maybe that's true even if the circle is put on the wrong 8x16 block.
It's not clear cut, so some advanced rule could be made to reserve that to the most important features (ex: preserve circles when they stand out in luminance) - and unfortunately all this would be lost in a NN approach.
If you find such problems interesting, would you be interesting in working on derasterize?
It's the testing ground for tmux-sixels, by working on standalone images first.

But before revisiting your branch, could you please wait a bit to allow me to be done with the code review on mine?

Sure please go ahead!

Thanks!
I will try to get the important feature upstreamed in tmux, but I fear it may be refused, as the author doesn't see the usecase for sixels and expressed his opposition in the past.
Yet sixels are so practical once you get a taste of them that I found it worth forking tmux. So did you, and a few others.
BTW I see you've cloned wemux. Apparently, you are inconvenienced by tmux limitations. If the upstreaming fails, what about joining our effort?
We could maintain our fork tmux-sixel, and add all that's missing in regular tmux!

@csdvrx good idea. I'm interested in co-op these topics. Let's move our discussions to tmux/derasterize issues and give room for windows terminal discussion here 😅😅

Is there anyone here that has access to an actual VT340 terminal that is willing to run some compatibility tests? You'd just need to be able to output a text file to the device and photograph the result. I have about a dozen such tests I'd like to run, but depending on the results there may be some follow-ups. No need to commit to anything, though - any help would be appreciated.
And if nobody has a VT340, a VT330, or even a VT382 would be OK.

And if nobody has a VT340, a VT330, or even a VT382 would be OK.

I have a question about ECMA-48 here.  That specification (and its ISO counterpart) was last updated in 1991.  Is there a provision in ECMA-48 that accomplishes this device-specific feature?

@j4james, would a VT420 be suitable?

@j4james, would a VT420 be suitable?

https://en.wikipedia.org/wiki/VT420

There were no color or graphics-capable 400 series terminals; the VT340 remained in production for those requiring ReGIS and Sixel graphics and color support.

So... no, seems a VT420 is not suitable, but thanks anyway :-)

@orcmid, the only ECMA-48 reference I can find in the console documentation is in Classic Console APIs versus Virtual Terminal Sequences:

These sequences are rooted in an ECMA Standard and series of extensions by many vendors tracing back to Digital Equipment Corporation and Tektronix terminals, through to more modern and common software terminals, like xterm.

AFAICT, ECMA-48 does not define sixel graphics or any similar feature. However, it defines the opening and terminating delimiters of the device control string that DEC uses for sixel graphics, so a terminal that does not support sixels can at least recognise the delimiters and avoid displaying the bytes between them as text. The data formats defined in ECMA-48 leave a lot of space for vendor-specific extensions.

Is there anyone here that has access to an actual VT340 terminal that is willing to run some compatibility tests? You'd just need to be able to output a text file to the device and photograph the result. I have about a dozen such tests I'd like to run, but depending on the results there may be some follow-ups. No need to commit to anything, though - any help would be appreciated.

And if nobody has a VT340, a VT330, or even a VT382 would be OK.

Instructions for emulating VT-series terminals via MAME are here. .
A dump of some of the VT340 ROMs is here, along with a pointer to the discussion around getting those.
I have done the VT102 MAME, it worked alright.  I haven't tried to get a VT340 going, and do not know if MAME got that capability or not.  But perhaps there is enough out there to now to do it.

@j4james, would a VT420 be suitable?

@KalleOlaviNiemitalo Unfortunately that won't do. But thanks for the offer. That would definitely be useful once I get around to working on some of the VT420 functionality.

I haven't tried to get a VT340 going, and do not know if MAME got that capability or not.

@klamonte Unfortunately the MAME VT3xx driver is just a skeleton - it's not functional. Their VT240 implementation is OK, and I've been testing with that as much as I can, but it doesn't have all the features of the VT330/340.

. The data formats defined in ECMA-48 leave a lot of space for vendor-specific extensions.

It appears that DEC used the extension provisions as required, the usual agreement among implementations proviso entering into it.
It strikes me that Windows Terminal is not ncurses and I don't see any provision for device-specific selections that embrace extensions..
Another concern, however, is regarding accessibility.  That has come up in issue #7766.  I have no idea how there are requirements with respect to that for WT.  I had not thought about that wrinkle and it impacts my ideas about demonstrating CUA display of the MS-DOS variety.

I had not thought about that wrinkle and it impacts my ideas about demonstrating CUA display of the MS-DOS variety.

Do you mean what is commonly called TUI today, i.e. mouse-driven textmode windowing?  If so, you may find this to be of interest.  I do not have a Windows system with Terminal to test it on, but someone else told me that mouse was working as of version 1.4.3243.0 on Windows 10.0.19041.1 (using a Debian WSL instance).

@j4james , do you have a build you can publish on your fork that provides Sixel? I can think of a few things to use for testing/debugging such as the sample projects using notcurses. My biggest hope for this feature in fact is to use notcurses applications in WSL2 (you will find the library is available in many distro repos).

@WSLUser I'm afraid I don't have any plans to publish my build in the short term. It was really just an experimental framework for me to test different implementation strategies with, and to investigate whether it would be feasible for us to support both standard VT340 applications (which is my primary use case) as well as more modern sixel derivations (which require extended functionality).
Unfortunately what I've found so far is that modern apps often rely (usually unnecessarily) on broken behaviour in XTerm, and unless we replicate that behaviour (which then makes us unusable as a VT340 emulator) we won't be able to run those apps. So before releasing anything, I thought it might be best to try and get some of those apps fixed first, but I want to be absolutely certain of my facts when reporting bugs, which is one of the reasons I'm looking for a VT340 to test with.
I should also mention that I've been talking to some other terminal devs to see if they'd be willing to agree on a standard for apps to negotiate extended sixel functionality, again in the interests of supporting both VT340 and more modern apps. Unfortunately that discussion doesn't look like it's going anywhere, and I'm about ready to give up at this point.

In case it might benefit anyone who has subscribed to this thread but doesn't want to wait for sixel support, I have decided to release the version of sixel-tmux that had been demonstrated here last year.
Compared to the previous version that only respected sixel sequences and therefore required a compatible terminal, this new version provides an immediate way to display sixel content inside most terminals (and therefore Windows Terminal) as it features an integrated derasterize.
This provides a way of converting sixels into something that can be displayed even by terminals that can't handle sixels natively, as long as they have enough colors for the content (fortunately, WT already support truecolor)
The source code is on https://github.com/csdvrx/sixel-tmux and the binary for msys2 on https://github.com/csdvrx/sixel-tmux/blob/main/tmux.exe
It works nicely in Windows Terminal to offer sixel-like features without being much slower than a standalone derasterize: see for example how it display the usual test images, after which derasterize is being used to show the lemur example:

Here's another one with Windows Terminal next to mintty, then being fed the content currently displayed in mintty when it connects to the shared session:

To launch it inside WT, please use script due to https://cygwin.com/pipermail/cygwin/2020-May/244878.html for example /usr/bin/script -c '/usr/bin/tmux c' /dev/null to create a new session or /usr/bin/script -c '/usr/bin/tmux a' /dev/null to attach to an existing session started in another terminal.
@yatli let me know if you find this sixel-tmux interesting, I would enjoy working with you on practical improvements as explained before
@hpjansson feel free to add other supports besides sixels, or to integrate a chafa backend if you can make a license exemption for tmux (as it's BSD)

@csdvrx Amazing work! Since you already integrated Derasterize, you probably don't need Chafa. But if you want a unified API to simultaneously support Kitty, iTerm, sixels and full-Unicode pseudographics at some point, you should be able to link with it from BSD source as the LGPL permits that. If there are license or other issues holding you back, I'm happy to help.
Maybe we should consider continuing the general sixel/terminal graphics talk in a discussion thread... Feel free to create one in the Chafa repo; as a project it touches on more or less every aspect of terminal graphics. E.g. we just started a thread about improving font support for pseudographics.

@csdvrx Regarding moving: I'm just concerned that we're hijacking an issue where the poor MS Terminal developers are trying to track their work :) But if they don't mind, it's obviously a non-issue.

At the moment, I don't care 😋 We know that sixel is something we need to work on, we've got some steps towards getting it to work done already. I'm fairly confident that @j4james is continuing to experiment with it. When we need to wrest control of this thread for our own feature tracking, I'll come back through and mark it all as off topic. Till then, go for it.

Success! Tested on a real VT340 and it worked perfectly.
For evidence, here is the output from the VT340 after I told it to send a MediaCopy to the host (essentially a screenshot) of the VT340 screen in sixel format:
https://gist.github.com/hackerb9/fb5eb56391e51de23af6dd5cedb12464/raw/ac9032df3afd62ea6e5f6f0b6a5621923c1a1630/vt340mediacopy.six
And here is that MediaCopy file converted to a PNG:

Note the occasionally glitches where a byte has gotten its eight bit set high is an artifact of my mediacopy.sh script. The owl looks perfect on the VT340's screen.

@naikrovek cough cough  Scroll all the way to the bottom for some notes on mixing images and text in a single cell.
Has anyone requested SGR-Pixels (mouse mode 1016) yet?  I see it mentioned here as not supported.  Decode sixel, add SGR-Mouse, and you've got a viable new gaming medium.  Sure would be nice to have some roguelikes that could put real images in when they needed them.
Or if we need to be all business-y to justify it, how about a nice little MSPaint app that works over ssh?

I wasn't considering 1016, but I do have a POC of the DEC locator mode. There's no point in doing either of them until we have sixel, though, because the pixel coordinates will need to be tightly coupled to the sixel resolution.

Moving a discussion comment from #13024

In theory, we could work on DRCS and Sixel support without having to figure out how they transit ConPTY up front, which will let us parallelize that work

Could we now just flush the frame with TriggerFlush(false) to end any current buffered conpty content, and then pass that string through? I think we're getting pretty close here to getting the strings to the Terminal, albeit not rendered yet

Could we now just flush the frame with TriggerFlush(false) to end any current buffered conpty content, and then pass that string through?

I don't think so, no. I suppose if all you want to do is "cat" an image from the command line with something like img2sixel, that might suffice, but anything complicated will probably break. We either need the full passthrough mode working, or for the conpty renderer to be capable of regenerating the sixel on the fly so it can repaint areas of the screen that have been invalidated. Of the two, passthrough mode seems more feasible.

Has there been any update on this?

Happy new year without Sixel, fellow dreamers!
Speaking of subscribing and sitting tight… GitHub seems to have basically hid this subscription from my account even despite re-subscribing. Having to find it through search to check back on the lack of progress.

sixel support is rapidly becoming as mandatory as ANSI escape sequence support.  Please do not allow Terminal to become as obsolete as the old Windows console host by not implementing sixel.

As far as I know, terminal emulators are the only platform capable of running cross platform, statically linked, mouse driven apps that can even be run remotely over ssh.
The main feature missing from this platform is a universally implemented graphics API. I've spent a bit of time researching this, and I believe implementing at least one of sixel, kitty, or iTerm2 would be well worth the effort.

IS there something similar to sixel, but for audio? I think it would makes sense over SSH connections...

IS there something similar to sixel, but for audio? I think it would makes sense over SSH connections...

Similar, insofar as that representing anything even moderately complex will require extreme downsampling? Absolutely there is: DECPS.

@piranna : Off-topic here, but apart from what @DHowett mentioned, you probably have more success to stream audio via a pulseaudio TCP connection which you can tunnel via SSH.  :bulb:

@rgpublic I know it's offtopic, sorry for that, but hoped you could guide me. Good to know something exists, thank you to both of you :-)

I hope this i the central issue for image display, so I am following this. Unfortunately there seem to be many other issues which essentially request the same feature, so that its relevance might be underestimated ...

@j4james How stale is this: https://github.com/microsoft/terminal/issues/448#issuecomment-937751369 branch? With eyes on #1173 in the immediate term, I wanna see just how far away that would be from merging in a full passthrough world

@zadjii-msft I actually just got my sixel branch merged with main in the past week or so. And with the latest progress on #1173 I was going to ask whether it might be a good time to create a PR for it. Worst case I thought we could put it behind a feature flag until the Windows Terminal side is ready.
That said, I've still got a lot of cleaning up to do before the code is presentable, so it may be a while before I can create a PR. Also, I should note that it only has a GDI renderer - I was hoping somebody else would handle the Atlas implementation, because I know very little about how that works.

NOW IS THE TIME

Sorry I got a little excited but yes - I think this would be the perfect time to start merging that in. We can absolutely merge this all piece by piece. We can work on merging the conhost sixel implementation while passthrough is getting ready. Then we can get the Atlas sixel implementation in in post.

I was hoping somebody else would handle the Atlas implementation, because I know very little about how that works.

Once I have an idea of what you need, I'll give you an implementation the same day. :)
FYI ideally, I need a per-sixel-texture ID so that I can cache the CPU-side texture as a GPU-side one between frames. But it will also work sufficiently well without that, at least in the beginning.

This is my favorite GitHub notification in today's catch-up. So excited to see this make it in 😁

And this FFmpeg-SIXEL is crazy cool.  IDK why I am so excited for running videos in the terminal... or maybe because then we can get graphics output directly from python scripts in both REPL and from posh CLI.

I would rather recommend mpv. And you can already play movies in the terminal with mpv --vo=tct --really-quiet <file>,

obviously this is not what can be achieved with --vo=sixel or --vo=kitty, they are both supported in mpv and will just work ones they hit Terminal.

since #17510 is merged, maybe close this issue?

@mominshaikhdevs @lhecker Well, sixel support still isn't implemented in Windows Terminal.


Turned maestro networking management back on

Needed by this PR: https://github.com/armPelionEdge/snap-pelion-edge/pull/216

Since this repo contains configs for multiple devices, can you please specify "dell5000" in the commit summary?


add support for Task with context as the function argument

In current realisation in case if any of the tasks is a long running function, (s *Scheduler) Wait() might hang for a long period of time, before exiting.
If the job function hangs, it will consume the resources, staying as a running goroutine forever. It will not hang the worker, since it will return on timeout, but it will still be in scheduler and consume resources.
I suggest to add TaskWithContext instead of the regular Task and make it a standard. It will allow to add such parameters for scheduler as the real timeout. Give caller more flexible controls of the task execution and will allow to actually gracefully stop the task.

@emar-kar Thanks for your contribution! I will check it out soon.

You are right, when f hangs and the timeout signal arrived, the resource in f won't be release. We need a method to pass the cancel signal to the task object.
Maybe we should change the signature of func Do() error in the Task interface?
// Task is an interface that defines a task
type Task interface {
	// Do starts the task, returns error if failed
	// If an error is returned, the task will be retried until MaxRetries
	// You can set MaxRetries by calling SetMaxRetries on the scheduler
	Do(ctx context.Context) error
}

And the RunWithTimeout may becomes like this.
func RunWithTimeout(f func(ctx context.Context) error, timeout time.Duration) error {
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()

	done := make(chan error, 1)

	go func() {
		done <- f(ctx)
	}()

	select {
	case <-ctx.Done():
		return ctx.Err()
	case err := <-done:
		return err
	}
}

Or we might need to add additional context for RunWithTimeout? Or just embed the context into the concrete Task struct?
What's your option on this change?

Hello there, I'm currently looking into the code, to see what I can suggest here. I'll introduce my ideas for you a little bit later.


2.5.0

Signed-off-by: Michée Lengronne<EMAIL_ADDRESS>

Once merged, I publish the release draft pointing on this commit.

Forget that and go to https://github.com/dev-sec/ssh-baseline/issues/156

Thank you @micheelengronne


aws-sso-cli: 1.9.5 -> 1.9.6

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/synfinatic/aws-sso-cli/releases.
meta.description for aws-sso-cli is: "AWS SSO CLI is a secure replacement for using the aws configure sso wizard"
meta.homepage for aws-sso-cli is: "https://github.com/synfinatic/aws-sso-cli"
meta.changelog for aws-sso-cli is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.9.6 with grep in /nix/store/9ly929syx1kq4yr6y5yahwmjma7gzrca-aws-sso-cli-1.9.6
found 1.9.6 in filename of file in /nix/store/9ly929syx1kq4yr6y5yahwmjma7gzrca-aws-sso-cli-1.9.6

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
aws-sso-cli

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/9ly929syx1kq4yr6y5yahwmjma7gzrca-aws-sso-cli-1.9.6 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A aws-sso-cli https://github.com/r-ryantm/nixpkgs/archive/790134b280661d903a8fb2f3e3ef0074ebe0852f.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/9ly929syx1kq4yr6y5yahwmjma7gzrca-aws-sso-cli-1.9.6
ls -la /nix/store/9ly929syx1kq4yr6y5yahwmjma7gzrca-aws-sso-cli-1.9.6/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

aws-sso-cli

Maintainer pings
cc @devusb for testing.

Result of nixpkgs-review pr 204632 run on aarch64-darwin 1

1 package failed to build:

aws-sso-cli


chore(main): release 4.2.3

:robot: I have created a release beep boop
4.2.3 (2024-09-09)
Bug Fixes

response-stream: improve chunk identification (fixes #260) (2aa474e)

Documentation

response-stream: add comments and references explaining implementation (d39db53)

Tests

response-stream: test eagerly flushed headers (0f33c29)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/H4ad/serverless-adapter/releases/tag/v4.2.3 :sunflower:


Append file extension based on content-type

As the title says, if the asset file extension is not present, now it's being appended based on response header content-type for http and https protocols.

Thanks for the contribution!


error says keychain GitHub credentials may be invalid when they they are correct

[x] Confirmed this is a problem with running a brew command and not brew installing or the post-install behaviour of one or more formulae? If it's a formulae-specific problem please file this issue at the relevant tap e.g. for Homebrew/homebrew-core https://github.com/Homebrew/homebrew-core/issues/new
[x] Ran brew update and retried your prior step?
[x] Ran brew doctor, fixed all issues and retried your prior step?
[x] Ran brew config and brew doctor and included their output with your issue?

brew search gives error about invalid GitHub credentials in keychain, even though these credentials are correct. The only thing I can think of that could be causing this is symbols in my github password.
brew config output:
$ brew config
HOMEBREW_VERSION: 1.3.6
ORIGIN: https://github.com/Homebrew/brew
HEAD: 270b752f5d9d218bfbed6fe85b6974fa653fb25f
Last commit: 6 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 0a32f798a1c944bc52b53a3fb36fb04cca99e386
Core tap last commit: 2 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/Homebrew
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit broadwell
Homebrew Ruby: 2.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby
Clang: 9.0 build 900
Git: 2.13.5 => /Library/Developer/CommandLineTools/usr/bin/git
Perl: /usr/local/bin/perl => /usr/local/Cellar/perl/5.26.1/bin/perl
Python: /usr/local/opt/python/libexec/bin/python => /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/local/bin/ruby => /usr/local/Cellar/ruby/2.4.2/bin/ruby
Java: 1.8.0_112
macOS: 10.12.6-x86_64
Xcode: N/A
CLT: <IP_ADDRESS>.1.1506734476
X11: 2.7.11 => /opt/X11

brew doctor output:
$ brew doctor
Your system is ready to brew.

To help us debug your issue please explain:

What you were trying to do (and why)
Use brew search to search for a package
What happened (include command output)$ brew search nyancat
==> Searching local taps...
nyancat
==> Searching taps on GitHub...
Error: GitHub
The GitHub credentials in the macOS keychain may be invalid.
Clear them with:
  printf "protocol=https\nhost=github.com\n" | git credential-osxkeychain erase
Or create a personal access token:
  https://github.com/settings/tokens/new?scopes=gist,public_repo&description=Homebrew
and then set the token as: export HOMEBREW_GITHUB_API_TOKEN="your_new_token"

What you expected to happen
Not to get the error about GitHub credentials (the credentials in the keychain are correct, I have confirmed this by cloning and pushing to a repository)
Step-by-step reproduction instructions (by running brew commands)

Have github credentials in your keychain
brew search nyancat (Any other string instead of nyancat also works)

Sorry, this isn't a general bug but something specific to your setup. The output provides several alternatives that it's not clear if you've tried.

Sorry, I should have included that and is as follows:
Clearing the credentials does work, but then it is unauthenticated and has the lower rate limits that come with that, and when your credentials inevitably get put back in your keychain by git (storing credentials in keychain is a very useful feature of git IMO). Creating a personal access token also works.
However both of these are just workarounds. The fact that you can do something differently and not get the error, doesn't mean that getting the error in the first place is not a bug.

However both of these are just workarounds.

Yes but, for example, most users do not need either of these workarounds and as there are available workarounds we do not consider there to be any changes warranted here. We would welcome a pull request to address your specific bug but there's nothing here for us to be able to debug what it is or why it affects your machine (and not mine, for example).

OK, I have tracked down the bug, the only reason I can see for you not seeing it is that you are using a higher precedence authentication method than just the credentials in keychain (I have none of the environment variables set, for example). Anyway a very simple patch should be coming shortly by way of a PR.

@tomjonandy thanks for the PR!


fix: type issue when void or undefined is returned from query function

Resolves https://github.com/tannerlinsley/react-query/issues/3516
Problem with current implementation is that T | Promise<T> is passed to QueryFunctionData<T> in types.ts, but Promise<T> does not extend undefined, so in that case the conditional type fails its intent.
useQuery(key, () => undefined) // errors, as expected ✅
useQuery(key, async () => undefined) // errors, as expected ✅

useQuery(key, () => 'hello world') // no errors, as expected ✅
useQuery(key, async () => 'hello world') // no errors, as expected ✅

thank you for the contribution 🙌 . Could you maybe add some type-level tests for useQuery and useInfiniteQuery that assure that the types work as expected, so that we can't inadvertently break them. A good starting point would be around here for useQuery:
https://github.com/tannerlinsley/react-query/blob/5562cfaaa04bbcea141c553dbdb211fec8224357/src/reactjs/tests/useQuery.test.tsx#L30
and here for useQueries:
https://github.com/tannerlinsley/react-query/blob/5562cfaaa04bbcea141c553dbdb211fec8224357/src/reactjs/tests/useQueries.test.tsx#L601

also, something doesn't compile at the moment:
Error: src/persistQueryClient/tests/PersistQueryClientProvider.test.tsx([15](https://github.com/tannerlinsley/react-query/runs/6107368533?check_suite_focus=true#step:5:15)0,15): error TS23[22](https://github.com/tannerlinsley/react-query/runs/6107368533?check_suite_focus=true#step:5:22): Type 'string | Promise<string> | undefined' is not assignable to type 'ReactNode'.
  Type 'Promise<string>' is not assignable to type 'ReactNode'.
    Type 'Promise<string>' is missing the following properties from type 'ReactPortal': key, children, type, props
Error: src/reactjs/tests/useQueries.test.tsx(785,13): error TS[23](https://github.com/tannerlinsley/react-query/runs/6107368533?check_suite_focus=true#step:5:23)22: Type '(a: string) => void' is not assignable to type '(data: string | Promise<string>) => void'.
  Types of parameters 'a' and 'data' are incompatible.
    Type 'string | Promise<string>' is not assignable to type 'string'.
      Type 'Promise<string>' is not assignable to type 'string'.
Error: src/reactjs/tests/useQueries.test.tsx(794,56): error TS2345: Argument of type 'UseQueryResult<string | Promise<string>, unknown>' is not assignable to parameter of type 'QueryObserverResult<string, unknown>'.
  Type 'QueryObserverLoadingErrorResult<string | Promise<string>, unknown>' is not assignable to type 'QueryObserverResult<string, unknown>'.
    Type 'QueryObserverLoadingErrorResult<string | Promise<string>, unknown>' is not assignable to type 'QueryObserverLoadingErrorResult<string, unknown>'.
      Type 'string | Promise<string>' is not assignable to type 'string'.
        Type 'Promise<string>' is not assignable to type 'string'.

@TkDodo Yes I see. A lot of code and complex types has been build on top of the assumption that the QueryFunctionData is correct. Let's see if I can find a way to fix all the type paths. The useQueries types are looking pretty complex though 😂

@TkDodo In the meantime, some questions have been arisen:
1st:
A lot of test cases use a query function with return type Promise<never>, which results from constructs like
const queryFn = () => Promise.reject('<error>') // typeof queryFn = () => Promise<never>

Is this a valid use case for react-query usage in the wild which we need to consider or just a thing occurring in internal test cases? If last, to please the compiler I could add a explicit function return types to all affected test cases like so:
const queryFn = ():Promise<unknown> => Promise.reject('<error>') // pleases tsc

2nd:
In the end what the compiler tells the user is something like Type 'Promise<void>' is not assignable to type 'never', where I am not sure if this is good DX since the user might wonder where the never type comes from, since the specified return type of the query function is Promise<void>.
However, as long as typescript does not offer some kind of custom error message I guess this is the only way to solve it at the moment besides the concept of type branding, which is also not great.
3rd:
At some locations in the codebase any is used for generic parameters, e.g. QueryClient.setQueryDefault, where all four generics of QueryObserverOptions are set to any, even thought QueryObserverOptions has default values for its generics TQueryFnData and TError set to unknown. This causes some problems with the new restrictions on the desired query function return type, since conditional types and any aren't good friends at all.
Would it be ok to change those generics from any to unknown, since conditional types can handle unknown much better than any type.
E.g.
setQueryDefaults(
    queryKey: QueryKey,
    options: QueryObserverOptions<unknown, unknown, any, any> // change first two generics from any to unknown
  ): void {
    const result = this.queryDefaults.find(
      x => hashQueryKey(queryKey) === hashQueryKey(x.queryKey)
    )
    if (result) {
      result.defaultOptions = options
    } else {
      this.queryDefaults.push({ queryKey, defaultOptions: options })
    }
  }

Types for useQuery hook are working as expected now.
Regarding useQueries hook maybe @artysidorenko can elaborate if he sees any way how to accomplish it there?
Problem here is I guess that there is no direct constraint of the TQueryFnData generic from useQuery down to QueryFunction<T> since there can be n different return types which are inferred by complex recursive types.
By now, it's still possible to have a query function with return type void | undefined | Promise<void> | Promise<undefined>.

Hi @yss14 , thanks for jumping on this 💪
On the useQueries bit, I think we could defined the type we want to guard against and bake that into the type-checker. It means another conditional on that logic, which I was hoping to avoid, but ran into similar issues you mentioned when trying to check lower down the chain at QueryFunction
type InvalidQueryFn = QueryFunction<undefined | Promise<undefined> | Promise<undefined> | void | Promise<void>>

something like this - https://github.com/artysidorenko/react-query/commit/10f55855925c274835628c423927721bbe33d7f8. Let me know what you think.
sidenote: interestingly, I think TS will make this kinda thing easier to deal with in the next version by allowing us to specify constraints on infer clauses

A lot of test cases use a query function with return type Promise, which results from constructs like
const queryFn = () => Promise.reject('') // typeof queryFn = () => Promise

If possible, I think we should allow Promise<never>, as it's different to Promise<void>. It's not a real production case I think, but the easiest way to test a failing query / mutation in codesandbox for example, and the type error would be hindering.

@TkDodo Thanks for the reply. I was thinking a lot about it during the past days and I guess the "bad" news is that this would not be possible at the moment.
QueryFunction<T> is using the generic for the data type returned by query function, not the return type of the query function. This is important because typescript leverages this constraint to automatically infer the data type of a query, independent of returning the data sync (T) or async (Promise<T>). If we would change this on a query function return type level, this would break the constraint and typescript would always infer T | Promise<T> as query function return type, and thus also as TData of every query, which is obviously not desired.
Further, I guess for sake of consistency we should not allow both never and Promise<never>, otherwise it could be a little bit confusing to other users of the library why never is not allowed, but Promise<never> is.
Regarding your use case of failing query functions in code sandboxes, I guess there is an easy fix to make it work in the future:
- useQuery(['key'], () => Promise.reject(new Error('Some error')))
+ useQuery(['key'], () => Promise.reject<unknown>(new Error('Some error')))

Furthermore, I implemented the suggested solution for useQueries (many thanks to @artysidorenko for your help!) and added test cases for both useQuery and useQueries.
Would appreciate a first code review of both of you guys :)

@TkDodo I rebased my branch with latest upstream beta branch and fixed the typescript errors.
How can I run test cases locally? Executing yarn test results in a TypeError: Cannot set property CancelledError of [object Object] which has only a getter error. Sounds like an error related to es modules?

Oh you get those errors locally, too? So weird, I've been having these for months - so from before the esm bundling changes. I found no way to fix it except doing a clean checkout to a new directory - then it works again. Sadly I have no clue how to properly fix it - maybe you do?

@TkDodo I think https://github.com/tannerlinsley/react-query/pull/3521 fixed the issue. It seems like the issue was caused by separate ./es build directory. I had a stale version still on disk because I checked out the beta branch before https://github.com/tannerlinsley/react-query/pull/3521 was merged. Removing it solves the issue and I can run tests locally.
That would also explain why is has been working for you when working on a clean checkout :D

@yss14 thanks, that sounds like it could be the issue. I'm not getting it at the moment, but I also think it happened when switching between master / beta, which I have been doing a lot recently 😅 . Can't wait to finally release v4 🙌

:tada: This PR is included in version 4.0.0-beta.9 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

@markivancho
Best solution of course would be to resolve the any type. But I guess this is not possible for some reason?
Are you in control of the someFunc return type? If so, simply change it to Promise<unknown>.
If not, you can try to explicitly set the TData generic of useQuery to unknown like useQuery<unknown>.

@TkDodo I guess we should add some notes to the breaking change log of v4 migration guide?

@yss14 this looks like a regression imo. Is there any way we can allow any / Promise<any> ?

@TkDodo I don't think so, since undefined | void is a subset of any type. So we can't say undefined | void should be restricted, but any should pass.
If there are scenarios where the query function returns any and there is no way of changing the return type, explicitly specifying the TQueryFnData as unknown would still allow using such a query function.
const fetchAnyData = (): Promise<any> => Promise.resolve()

function SomePage(){
  useQuery<unknown>(['mydata'], () => fetchAnyData()) // no typescript error
}

So in the end this change is a regression, since we restricted the allowed type space of the query function return type by "removing" undefined and void from it. And since both are a subset of any, we also eliminated any from the allowed type space.

I see. Personally, I think it's still a good change. The problem is if you just use:
useQuery(key, () => axios.get(...)

it will still default to any, which will now type error. Some people do:
useQuery<MyType, Error>(key, () => axios.get(...)

which is fine and will still work.

I'm inclined to keep the change, because returning any from your queryFn is likely not what you want, as it's a leaking any that you might have missed. Maybe we can:

make the error message a bit better (is not assignable to type '"queryFn must not return undefined or void"') to maybe include that any` can also not be returned?
add this to the migration docs, potentially here: https://react-query-beta.tanstack.com/guides/migrating-to-react-query-4#undefined-is-an-illegal-cache-value-for-successful-queries
also add this to the typescript docs: https://react-query-beta.tanstack.com/typescript with the proposed workaround of at least using unknown over any.

what do you think?

thanks, after further digging, my actual problem was options type;
my custom hook goes like
const getEntity = (gid: string): Promise<{ entity: Entity }> => api.get(`/api/entity/${gid}`);
const useEntity = (gid?: string, options?: UseQueryOptions<any, any, any, any>) =>
  useQuery(['entity_by_gid', gid], () => getEntity(gid as string), {
    ...options,
    enabled: !!gid,
    select: data => data.entity
  });

With this code, typescript complains Type 'Promise<{ entity: Entity }>' is not assignable to type '"queryFn must not return undefined or void"'. which confused me the very first time
after changing options type to UseQueryOptions<{ entity: Entity }, any, any, any> everything is fine
by the way, it would be very nice if you could include in the react-query + typescript docs best practices regarding generics usage

by the way, it would be very nice if you could include in the react-query + typescript docs best practices regarding generics usage

I'll try to improve the docs. In the meantime, have a read here: https://tkdodo.eu/blog/react-query-and-type-script

@TkDodo thanks! your blog is a golden mine of react-query

@yss14 here is another reproduction that works fine on v3, but fails on v4:
https://codesandbox.io/s/serene-panini-6o70bk?file=/src/hooks/useApi.ts

@TkDodo Thanks, I will work on a follow-up PR in the coming days regarding migration docs and typescript docs, as well as tackling the reproduction example you provided.

@TkDodo A little update from my side. I'm still investigating why typescript is complaining here, since your abstract data fetcher example looks fine to me. I already posted some question in r/typescript and the FullStackTypeScript Slack Community in hope anyone can give me some hint.
I also boiled down the use case into a minimal typescript playground
Maybe @artysidorenko has some clue why typescript is complaining here? :)

Hi @yss14, yeah your first example passes because TS sees a string and can make a direct comparison with the QueryFunction constraints. However soon as you deal with a factory function it becomes more context-sensitive (without more information it assumes TQueryFnData could be anything). It's not clever enough to realise you want to enforce the “must not be undefined or void” constraint on the factory. All it sees is (1) the factory will accept a function that returns a promise, so for e.g. () => Promise<void> is a valid candidate, and (2) since there's at least this 1 candidate that gives never when applied to QueryFunction<T>, it won’t let you pass the "maybe-never" generic to useQuery at all.
Not ideal, but one workaround for now could be to re-use the QueryFunction type in the factory itself to apply the same constraints upstream (thanks to ReturnType<T>).
here's what it looks like forking the codesandbox
// fails
fetcher: (params: TQueryKey[1], token: string) => Promise<TQueryFnData>

// passes
fetcher: (params: TQueryKey[1], token: string) => ReturnType<QueryFunction<TQueryFnData>>,

@artysidorenko would it maybe bet helpful to extract ReturnType<QueryFunction<TQueryFnData>> into an extra type like TQueryFnResult<T> and separately export that, and document that this is the type that custom generic Query Functions should return?

@artysidorenko another thing that might be a bit weird is that I can't return Promise<any>, which is what you get by fetch or axios per default. See this codesandbox reproduction, where I've inlined the queryFn to just call axios directly.
Error is:
Type 'Promise<any>' is not assignable to type '"queryFn must not return undefined or void"'

While I'm all for avoiding leaking anys like that, I think it goes a bit against the mantra of "the queryFn just needs to return a Promise to work". I'm not sure this is doable with the current types though, but not even useQuery<any> is going to fix it.
With that in mind, do you think it would be better to just remove the type level check again?

Hi @TkDodo yeah, I think I agree with you.
While I'm able to fix that particular Promise<any> issue for useQuery (by adding the TS equivalent of an "early return" in case of Promise<any>, got a forked branch here where I'm running the tests)
... I keep finding more and more edge cases that throw inconsistent results - for example the above solution doesn't work for useQueries yet, and neither does the "wrapping customer fetcher factory with the TQueryFnResult<T> helper".
Am mindful that you'll probably keep getting more and more of these issues reported, generating a lot of noise.
I'll put up a PR to remove the type-check later this weekend if there's no other opposition to that.
*longer term will see if it's possible to think of all the edge cases and potentially bring it later if we feel confident, but don't have an ETA on that


Bluetooth: Mesh: Persistent storage of Virtual Addresses

The 16-bit format group addresses will be stored,
but we don't store (or restore) the virtual label UUIDs,
i.e. after a power cycle the 16-bit group addresses
would be meaningless.
Fixes #19342
Signed-off-by: Lingao Meng<EMAIL_ADDRESS>

How does this related to #19499? Why did you open a new PR instead of continuing with the existing one?

This PR should be more concise than before.

@jhedberg: Perhaps the settings module should be restructured to only be a multiplexer for the "bt/mesh" settings directory, and the handlers should go into their respective modules? In a separate task, of course.

@trond-snekvik yes, sounds like a good idea

@LingoMeng please stop pushing to this PR, I've fixed up some coding style issues and pushed them to it, and I've approved the PR, so let's just wait for CI to finish so we can merge it.