Research diacritics

'umlaut' is pretty popular in demand, figure out how to incorporate them into the keyboard.

All character with umlauts :
ä, ë, ï, ö, ü, ÿ, Ä, Ë, Ï, Ö, Ü, Ÿ
Other characters we should think about as well :
à, è, ì, ò, ù, À, È, Ì, Ò, Ù
á, é, í, ó, ú, ý, Á, É, Í, Ó, Ú, Ý; ð, Ð
â, ê, î, ô, û, Â, Ê, Î, Ô, Û
ã, ñ, õ, Ã, Ñ, Õ
å, Å
æ, Æ; œ, Œ; ß
ç, Ç
ø, Ø, ¢
¿, ¡, €
Seems like most are modifying vowels.

I don't think thr keyboard needs to contain all the umlauts, but only those relevant for the language. In German as example it would reduce the list to: äöüÄÖÜß

How hard would it be to add the umlaut by combining it with the gesture already there? I'm not sure if the German layout is different from English, but a possible solution would be, for example using "a" :
Swipe from circle to right quadrant then to the top quadrant and back to the circle (which would give "a" if you lift a finger) but then without lifting your finger, go back to the top quadrant one more time then bring it back to the circle to give a "ä" .
It seems like you could just choose a quadrant to always go back to without crossing a line to modify the letter. Maybe instead of going to the top, choose the left quadrant or bottom to double back on for "ã"
A possible issue is the current layout doesn't appear to let you keep typing if you move your finger in such a way, so you might have to lift a finger and delete the inserted space to keep typing?

@reacherjack422 That could be a great way to do it in my opinion; in Italian for example the two accents are needed to form all letters (àèìòùé), in this way I could have a couple movements (although not very natural) to add those accents.
The main problem remains having alphabets with more than 32 characters; right now crossing more than 4 lines produces capital letters, it could be possible to have the extra letters in a fifth line crossing.
This is not at all a good solution as movements regarding those letters will get longer and longer.

@davidossky I don't know the limitations or complications involved in implementing into the app, but to me it seems like a possible solution would be to add a letter at the very top of each line.
A rough example, if there was a ñ at the tip of the line starting with "a"  and "y", let the user swipe either from the top section to the right section, or from the right section to the top (both would do the same thing) and then swipe back over that same line before returning to the circle.
So: from circle through y and then to a and then back to y before returning to the circle. Or from a to y to a to circle.
I'm just brainstorming what might seem easier without adding an extra line...
Or just add a modifier key to shift the whole keyboard to special letters, which would remove the smooth flow...

I think you guys are on track with how the keyboard implements the input for Esperanto specific characters.
https://github.com/flide/8VIM/blob/f83bb5ca17f6dc230f6ffaa463351d8e2fe844db/8vim/src/main/res/raw/keyboard_actions.xml#L567
If it makes sense, I guess, we are already on track and simply need to agree on the gesture for umlauts.
Also, following gestures are already taken:
Top -> center : hides the keyboard
left -> center : paste from clipboard
right -> center : start selection backwards
Also, In principle I don't allow gestures to cross the center.
General idea is : "Start in the Center, End in the Center". Obviously, the makers of rules are the first ones to break them.

Sorry, I did not know about how the Esperanto characters worked.
To me it looks like a great way to introduce all diacritics: moving back one sector enters in diacritics modality, from where i can select any diacritic to add on the character moving clockwise or counterclockwise.
The system is in this way very flexible and allows introducing all the diacritics needed on all the characters.
Therefore if I compose an 'a' with right -> top I can go back before going center, therefore top -> right.
At this point I can move either clockwise or counterclockwise to introduce the various diacritics.
For example:
center = ä; top -> center = à; bottom -> center = â
Doing the same movements with another letter should introduce the same diacritics consistently: e ë è ê
This will prevent having to learn a specific pattern for every single character.
However I do not really have any suggestion on which gestures to assign to each diacritic, this behavior is very language dependent.

I totally agree with davidoskky, but i think that the only way to solve that is to slowly implement every lenguage.

@davidoskky I don't fully understand your proposal. Could you please describe the exact movements you would perform for a few letters?
(let's use ä, é, and, because it's important for German ß, a special form of s)

However I do not really have any suggestion on which gestures to assign to each diacritic, this behavior is very language dependent.
For now, just decide what gestures you feel like using.

@Glitchy-Tozier sorry for the bad explanation, I will try to clarify:
Once i go back one line I will select the letter in the same way I would have done going back to center, going back will tell the program I want to insert that letter with a diacritic or a modification.
I will therefore first select the letter a, then select the diacritic umlaut and then confirm going center to get ä.
At this point we can define 9 different types of diacritics or modification: if I immediately go center I will get the umlaut, if I cross 1 line back and then center (going the opposite direction of the original movement) I will insert a grave accent (à), if I cross 1 line forward and then center I will insert the Esperanto character as it is already implemented.
Of course you can have a different diacritic crossing 1-4 lines back and forward, therefore a total of 9 different diacritics or letter modifications.
à: right, top, right, top, center
è: bottom, left, bottom, right, center
ü: bottom, right, top, right, center
ë: bottom, left, bottom, center
I am not a German speaker, therefore I do not know about the best way to insert the ß character, and I do not know how common it is. I guess this system could be used as well, as far as I know grave accents are only used over vowels, it may be the same for the umlaut. We could substitute the diacritic combination for the umlaut or the grave accents on the consonants to form one of its modified forms.
Since you say  ß is a special form of s, we could obtain it by applying a modification on the letter s.
ß: top, right, bottom, right, center
I'm not sure I can suggest good gestures, I am mainly an Italian speaker and I would therefore place the acute and grave accents in the easiest positions.
If different keyboard layouts will be implemented this will not be a problem at all, but creating a Latin alphabet with all the diacritics is just too difficult, in this case I think a little bit of research would have to be done.
If this is just needed for a quick implementation before adding the different layouts i would say:
umlaut: immediately center
grave accent: 1 back -> center
acute accent: 2 back -> center
Esperanto: 1 forward -> center (just how it is now)

Ok, I understand now.
Long post ahead.
I want to mention a few ideas for alternatives that present slightly fewer options to the user, but may be WAY quicker:
Option 1: Adding diacritics AFTER writing the characters:
####The idea:
This is somewhat similar to what has been proposed before, but sounds a little faster:

The user writes a letter which he wants to modify
a = center -> right -> top -> center
Then, the user does an "add diacritic"-movement. For example, for umlauts, that movement could be
.. = center -> top -> center
This "add diacritic"-movement would then cause the "normal" letter to be replaced by a modified one. This means, that
ä = center -> right -> top -> center -> top -> center.

This is all that's needed for German. (s should be turned into ß in every sector.) For languages that can use multiple diacritics, we can use all four sectors.
For example:
a-Z = do no extra movements
à, è, ì, ò, ù, À, È, Ì, Ò, Ù = add (center ->) left -> center
á, é, í, ó, ú, ý, Á, É, Í, Ó, Ú, Ý; ð, Ð = add (center ->) right -> center
â, ê, î, ô, û, Â, Ê, Î, Ô, Û = add (center ->) top -> center
ã, ñ, õ, Ã, Ñ, Õ = add (center ->) bottom -> center
####Why use it?
I like this way of doing it, because there is even more diacritic consistency. umlauts are always in the top-sector, no matter where you come from. Compare this to the current ĉ-method, where this diacritic-orientation rotates depending on where your letter is.
Also, it kind-of makes sense positioning-wise. â has a kind of top-heavy addon. é has a line going to the right. (hence the center -> RIGHT -> center movement.
Of course, users should be able to configure on their own, which sectors to assign which diacritics to.
(oh, and s should always turn into ß, on every sector I think. @davidoskky, it's somewhat common, definitely more common than something like q or y in German.)
####Problems with this idea:
It can only support 4 (user-chosen) diacritics. In our previous input methods, we could spin around the circle infinitely to access all diacritics on our 6th line crossing.
(though honestly, at that point switching to a different keyboard would probably be almost as fast)
Option 2: Breaking some rules  :)
This is an option that sounds REALLY NICE for languages, that don't have multiple modified versions of the same letter.
For example, this could work in German, where there's only
a -> ä
However, in Italian, where there is (i think)...
e -> è, é
...the following proposal is not productive. I'd still think we should consider giving users of non-diacritic-heavy languages the choice.
####The idea:
This removes the extra-motion-component entirely. Basically, you would type a diacritic letter by lifting your finger off the screen in the sector of the letter, and then finish the movement.
This is amazing for languages such as German, as it gets rid of all extra movements. Just quickly raising your finger. This should be way quicker than all the previous options and it should be more comfortable, as there's no "high energy" -movements (= big direction changes as described here: https://8pen.blogspot.com/.
Compare that to those other options, where you have to

Travel extra distance with your finger
Do pretty hefty direction-changes.

####Problems with this idea:
There are two big problems:

Italian speakers
(Sorry, i just thought it was funny. What I meant was languages that need multiple  â, á, à, â and ä cannot use this.)
It clashes with one of the preexisting rules:

Also, following gestures are already taken:
Top -> center : hides the keyboard
left -> center : paste from clipboard
right -> center : start selection backwards

This means, we probably should either specify or change those rules (examples for top->center):
Specify:
"Only when the trail hasn't previously stopped in the top-sector, does top -> center hide the keyboard."
OR
"Only when the trail hasn't previously stopped in the top-sector or the stop lasted longer than 1 second, does top -> center hide the keyboard."
Change:
Instead of
Top -> center : hides the keyboard
, we could kind of...
Top -> bottom : hides the keyboard.

Also, In principle I don't allow gestures to cross the center.

Yes, I read this, but I don't really understand why. There already are center -> right and  center -> left shortcuts which break the center -> ... ->  center principle.
.
This was more than half an hour of work, I hope it's somewhat comprehensible.
I think both of those options should be good ideas, and both could simultaneously be implemented (and even be used at the same time)
Users should be able to switch them on & off though, if they for some reason prefer not having one of them exist.

@Glitchy-Tozier the first idea you propose is the same coming from @reacherjack422 as far as I can tell.
I do really like the methodology you proposed in fact, I have tried doing the various movements and to me it seems faster to add the diacritic with a subsequent movement.
Therefore I prefer the usability of your proposal to that of mine.
My only concern was the limit of 4 diacritics; i checked wikipedia and it looks like no language with Latin alphabet uses more than 4 different diacritics on the same letter.
Therefore this method looks completely motivated to me, and right now it looks like the best proposal.
Regarding the second method proposed there obviously is the "Italian speaker" problem... however I guess it could be a nice optional feature if it can be implemented.
Of course I would not prioritize developing this particular methodology over the other.
Both these methods rely on language dependent keyboard schemes. We could start implementing the four movements you suggested, but in order to have this keyboard usable by everyone an option to change keyboard scheme is absolutely needed.
If keyboard schemes are not implemented a way to choose the diacritics that can be inserted is needed, this could be a good Idea even if keyboard schemes are implemented in fact: I could start speaking German and therefore I could use èéë who wouldn't all be included in the Italian scheme nor in the German one. Changing a setting I could insert the umlaut and be able to write in two different languages.

@davidoskky I agree with pretty much everything.

It's nice to know that no language uses more than 4 diacritics.
The "add diacritic"-movement seems like a good idea.
I agree that the first approach from my post should be prioritized over the second. I just hope the second, too, will eventually be implemented, for the love of fast German typing. (and to the benefit of other diacritic-limited languages)
I agree that the user should be able to configure which sector adds which diacritic. This is especially helpful if the user is used to rapidly switching between languages that use different diacritics. (for example when alternating between Italian and German)

Regarding @reacherjack422's idea:
It seems I didn't quite understand some of the first messages. However, now that I have re-read them, there actually seem two ideas were proposed by reacherjack:

The right is the same thing I proposed. I never noticed that he suggested the same thing.
The left idea is an interesting one as well. Here is where he describes it:

@davidoskky I don't know the limitations or complications involved in implementing into the app, but to me it seems like a possible solution would be to add a letter at the very top of each line.
A rough example, if there was a ñ at the tip of the line starting with "a" and "y", let the user swipe either from the top section to the right section, or from the right section to the top (both would do the same thing) and then swipe back over that same line before returning to the circle.
So: from circle through y and then to a and then back to y before returning to the circle. Or from a to y to a to circle.

Both of those ideas have pros and cons. His second (the left) idea has the benefit of

being able to produce TONS of diacritics
or
being able to choose a direction from which to add the diacritic. You can choose whether to do a movement that starts off to the right or to the top. This may or may not result in smoother writing.

However, i prefer his first idea (the right one). Mainly because

It's simpler.
The user doesn't have to go the extra step of learning to use path 1 or path 2 depending on where they're coming from and where they're going.
There's less distance being traveled by your finger and it's a "lower-energy-movement". This is the main reason why I think his first idea (the right one) is the better one. It should, ultimately result in quicker typing.

ß
@flide

I just wanted to mention: Regardless of what layout gets implemented, please remember that there not only is ß, a special form of s, but there's also the capitalized version of ß (ẞ). It's really uncommon actually but for the sake of completeness, it should be added, just as is the case with ä & Ä for example. https://en.wikipedia.org/wiki/Capital_ẞ
Also, you might want to add "ẞ" to your list of

Other characters we should think about as well :

Thoughts on the proposed ideas so far?

Very good ideas here. I don't want to crash this, but I absolutely prefer the initial method brought up by @davidoskky instead of the method suggested by @Glitchy-Tozier for several reasons.
Warning, very long post approaching ... :/
First I would like to stress the most important aspects any implementation should satisfy, at least imho:

Being fluent, i.e. don't interrupt movements. I would consider even the quick lift to enter diacritics described by @Glitchy-Tozier as disturbing the movement (It is a great idea to expand pattern space, though.)
Don't fall into the pit to (heavily) optimise for specific languages, i.e. you should be able to write words for any language with the very same keyboard. (Don't go the same unfortunate route PC keyboard layouts did :) Shortcuts for the most used characters for the preferred language are still great and I like that idea of @Glitchy-Tozier. Let's make these available within a defined range of user defined patterns.
Be careful with patterns which blocks (perhaps not yet for-seen) additions or tons of others gestures. Using the very first grade patterns x -> circle -> top/left/right/bottom -> circle just to attach a diacritics to the current character seems like overkill for me.

On a a little bit more abstract view, I would like to see the following concepts supported:

Be able to insert any letter, i.e. also capital letters, digits, punctuation, special characters etc. fluently.
Be able to make corrections (e.g. backspace or even undo/redo) fluently.
Both points also implies that switching layouts has to be a fluent pattern.
(Just to clarify: pressing top to toggle shift is not fluent as far as I concern. The existing method of adding a complete circular round (though long) is.)

List of possible patterns:

circle -> monotonous movement -> circle: character/elements of a "thing"  separated by spaces (a.k.a word).
sector: Pole position shortcuts at word boundaries.
circle: There is only a single semantic which is just right: space :)
circle -> non-monotonous movement -> circle: Ideal for modifying current character.
sector -> x: A complete orthogonal dimension of patterns, which are not able to be used fluently within a word, but could start one.
x -> sector: Already in great use to cancel word without space and be near the to-be-typed-next pole position shortcut. But there are additional patterns possible.
x -> circle -> sector -> circle: Some (currently unsupported) case which would be unambiguous, though. (That was also brought up by @reacherjack422 as well as @Glitchy-Tozier.)
Fancy patterns like proposed by @Glitchy-Tozier (short lift etc.)

My proposal as far as I thought through it:

Keep usual characters, space, backspace, toggle shift, return, cursor, start selection, paste, hide keyboard.
Add consistent support for diacritics like (already available for esperanto) and proposed by @davidoskky. This is great to get any one even if you usually don't need it on your language but just want to write a foreign name for instance.
Specify range(s) of user defined patterns. See below.
Support fluent single backspace by _ x -> circle -> right -> circle_.
Start fluent edit mode with x -> circle -> right -> y where ... (in fact the end of this mode is non-fluent except for single backspace)

... circle inserts a single backspace.
... ccw movements inserts a backspace for ever passed sector.
... cw movements deletes a character on the right for every passed sector.
... circle -> left -> ccw is undo.
... circle -> left -> cw is redo.
... circle -> top -> ccw is cursor left.
... circle -> top -> cw is cursor right.
... circle -> bottom -> ccw is selection left.
... circle -> bottom -> cw is selection right.

Switch layouts temporarily (i.e. for one character) with circle -> top -> cw/ccw and to locked layout circle -> bottom -> cw/ccw.

List of possible layouts:

Usual letters + important punctuation/special characters (status quo)
Capital letters + additional punctuation
Digits, special characters, toggle of modifiers (ctrl, alt, etc.)
More characters (even special layouts for editing would be possible)
Any number of user defined layouts

The special shortcuts _circle -> top/bottom -> circle could be used switch to two specific layouts fluently (either locked or temporary) and should be user configurable (supporting any mapping). The user could for example choose to assign the capital layout to one these shortcuts if desired, which gives a fluent way to capital characters within a word (but shorter than the existing circulation variant).

Additionally, it would be very well designed if some (extension of) patterns are defined to to be user mappable explicitly. That means all user can rely on mappings, regardless which device and which version they are currently using, but also know that a specific space is reserved for user configuration.
My idea preopose these patterns to be user configurable:

circle -> top/bottom -> circle: Could be used e.g. as modifier for shift, diactritics, layout switching or anything else.
_sector -> x: Any pattern which starts in a sector (including existing pattern for paste, start selection etc.)
Perhaps even the short lift patterns (proposed by @Glitchy-Tozier) could be used as user defined.

Best regards,
doak

You guys know what... I haven't replied to anything for last 2 days because I felt overwhelmed by the size and passion of comments you guys have been posting. Took some courage to get back to github to answer other threads. Still reading your responses... Will comment and present my current ideas in sometime.

@flide Sorry for shocking you xD I imagine it must also feel good starting a Project and seeing people join in on the discussion!
.
Ok, back to the topic at hand!
@doak, thank you for providing an alternate view on the topic!
I would like to respond to a few very valid points you made.

First I would like to stress the most important aspects any implementation should satisfy, at least imho:
~  Being fluent, i.e. don't interrupt movements. I would consider even the quick lift to enter diacritics described by @Glitchy-Tozier as disturbing the movement

Fluidity
I agree: Fluidity is important. But as far as I understand the purpose of this app, it's not the end goal. It's a means to the end, and a very elegant to accomplish this end goal: Speed.
Let me give you an example: We can tap the top section to capitalize the next letter. Alternatively, we can add an extra circle of movement, crossing at least 5 section-borders. I would argue that the tapping mechanism is better, but thats debateable I guess. But let me ask you this:
If you had to add TWO full circles to get your capitalized letter, would you still prefer the circle-method? What if it was THREE turns?
Imagine this movement and compare it to tapping once. Yes, it is more fluent. But it is slower and thus less desireable.
I hope we can agree on this... while using a fun and stylish keyboard sounds like a treat, I think we should place our main focus on efficiency.
Speed
There are a few conributing factors to speed, and while fluency my be one of them, we my want to examine all of them closer:

Keeping a steady trail, what we may call fluency or continuity.
Keeping all movements as "low energy" as possible (avoiding crazy direction changes)
Keeping the distance travelled by your Finger as short as possible.

To me, this is the issue whith... let's just call it @davidoskky's 1st idea.
The fastest movement he proposed (the movement for é I believe) is of the same speed as @reacherjack422's 1st idea.
They both (if i understood them correctly)

have the same fluency/continuity (you don't lift your finger off of the screen)
Have the same "Energy" (They both actually are a mid-high-energy manuver, which isn't optimal but the best we've currently got to add multiple diacritics.)
In both cases, you travel exactly the same diatance.

However, things get worse for @davidoskky's 1st idea when trying to type a "non-optimized" diacritic. (something like è or ë instead of é)
There,

Fluency/continuity still is given
Energy gets slightly worse higher (worse) for @davidoskky's 1st idea while @reacherjack422's 1st idea stays where it is regardless of diacritic. The change is not huge though, so let's ignore it.
Distance is the breaking point. @davidoskky's 1st idea adds more and more distance travelled while @reacherjack422's 1st idea stays the same.

Which makes inputting way more efficient and fast.
.

First I would like to stress the most important aspects any implementation should satisfy, at least imho:
~  Don't fall into the pit to (heavily) optimise for specific languages, i.e. you should be able to write words for any language with the very same keyboard. (Don't go the same unfortunate route PC keyboard layouts did :) Shortcuts for the most used characters for the preferred language are still great and I like that idea of @Glitchy-Tozier. Let's make these available within a defined range of user defined patterns.

I agree that we shouldn' introduce huge differences where none are required. But we will have to optimize for languages.
For example, in German, the letter y is way more uncommon than in English. Which means, for German at least, one important space is clogged by a rare letter. Which feels horribe
We will need to change this. So why not also make it easier for others to write their prefferred diacritics? We will also need to do this for languages that use ore than one diacritic (French, Italian, ect.)
.

First I would like to stress the most important aspects any implementation should satisfy, at least imho:
Be careful with patterns which blocks (perhaps not yet for-seen) additions or tons of others gestures. Using the very first grade patterns x -> circle -> top/left/right/bottom -> circle just to attach a diacritics to the current character seems like overkill for me.

Am I correct in assuming that your main issue with @reacherjack422's first idea is that it interfshouldith your other ideas for the keyboard?
(The main problem being that it uses up potential shortcuts)
I must say, I really like some of your ideas. I don't fully understand what you meant by some of them, but many of the ones I understood i liked. I would like to suggest two things though.

We shouldn't prioritize other shortcuts over diacritics (except, of course, if 8vim is supposed to be an english-speakers-only-keayboard)
Diacritics are a VERY importand featrure or other languages and if we assign them to uncomfortable movements, we simply annihilate important letters of those languages' alphabets.
As already mentioned, @doak, i wouldn't mind discussing those layout-ideas. But i think we should open another issue on GitHub, instead of mixing it with a discuasion on diacritics.
So please create a new issue, copy your ideas there and elaborate on them (i didn't understand all of them), and I'll happily reply.  :)

@Glitchy-Tozier, thanks for your response.
Yes, you are right speed is the correct metric -- as long it does not break other main goals (like being able to use it blindly and preferable one handed.) We respectively @flide really need to define these to streamline the ideas.The mentioned are only my understanding which may not be correct.
Regarding your examples about speed for this specific use case (diacritics) you are absolutely right.

Am I correct in assuming that your main issue with @reacherjack422's first idea is that it interfshouldith your other ideas for the keyboard?

Yes. that's true. I also realised after posting that it indeed got quite OT ;)
I will have look to setup some conceptual issues. Perhaps these already exists.
Finetuning the default layout will be very (let's say) interesting. Of course, diacritics are quite important.
I saw some issue planning to research th optimised layouts for different languages bases on articles, books, etc.
(OT again, sorry: my hope would be to gain a quite good compromiss as the default layout, support configuration for some pole position shortcuts for language specific improvements and also support whole owner defined layouts (which can be bundled or easily added) for all these guys who want to go even further.)
Regards,
doak

I do like the first idea from @reacherjack422 but I do realize it is getting in the way of other usages.
For example those movements are already been proposed in #1 to select the predicted words. @doak proposes to use those movements for capitalization and deletion.
This is not a simple decision to take, these seem to be the easiest movements and we have to decide which use will be most beneficial to users: diacritics, editing or predictive text.
In my opinion diacritics are the most important generally: I do not mind having to tap in order to capitalize or to delete, moreover if I am using predictive text I do not mind tapping in order to save some time. On the other hand I want to insert diacritics as fast as possible and as smoothly as possible in order to be able to write everything I need to write.
The insertion of diacritics is in my opinion a priority as the keyboard is useless without it, while I can use it without the other features.
I have seen @foolsbazar comment in #10, he maintained a version of 8pen in which accents are added in a way similar to my proposal and the movements proposed by @reacherjack422 are used to insert spaces and make deletions.

Inserting letters with accent by first typing the origin letter and then the accent (e.g. a+´ = á). The accents layout would be triggered by doing again back and forth into another sector.

@foolsbazar could you please let us know why you chose to have this behavior in your app and if you had positive or negative user feedback?

I have finally read the long posts!!!! Thank you guys!! I love it. I did not think in a million years that this keyboard will be successful enough to gather heated debates!!
Okay now, rather then replying to each idea, which I promise I have read, I am going to present my ideas from the perspective of implementation and my vision for the keyboard and usability.

First thing first, this keyboard is indeed a English first keyboard - As of now. This is because the language was easy to implement and fulfilled my immediate need for the keyboard. I want to change that. The logical step would be to move to a 'Latin-script' based keyboard.
So what I am proposing is to somehow incorporate the following characters :
ä, ë, ï, ö, ü, ÿ, Ä, Ë, Ï, Ö, Ü, Ÿ, à, è, ì, ò, ù, À, È, Ì, Ò, Ù, á, é, í, ó, ú, ý, Á, É, Í, Ó, Ú, Ý; ð, Ð, â, ê, î, ô, û, Â, Ê, Î, Ô, Û, ã, ñ, õ, Ã, Ñ, Õ, å, Å, æ, Æ; œ, Œ; ß, ẞ, ç, Ç
Following characters I am excluding, and in my opinion should be added to the symbols keyboard -> open for discussion
ø, Ø, ¢, ¿, ¡, €

1. Language Specific Keyboard Action Add-on gestures
So currently, I have one big xml file under resources/raw that holds all the gestures. What I can offer is to add language specific files with language specific characters. Only one language would be allowed to be turned on at a time and the diacritic for that particular language will be inputted when the gesture loops over the last sector demarcating line, as is currently done for Esperanto characters.

Pros

This effectively launches the keyboard specific layouts and functionality
Easy to implement
Flow of input is not broken, provided we are able to find suitable gesture for each diacritic letter within the language where there exists more than one variations for a single character.

Cons

The gestures for the diacritics still need to be sorted out, and now on per language basis
A verification needs to be in place to not override any existing gesture, going to be a maintenance nightmare
Only one language can be turned on at a time and will infuriate multi-linguals.

2. Utilizing Automated Text Replacement (ATR) feature
If we are able to develop ATR, then the user can put in custom command to be overridden to the character of choice.
For example, In Esperanto the character 'x' is useless and can be appended to other characters so that the duo can be replaced automatically by the system in the text field:
ax -> â
Pros

ATR is implemented and it achieves more than just diacritics

Cons

Each user will either have to put in their own ATR's or accept the defaults that might not be acceptable
We will be stuck in a ditch for the languages that don't have any letter to spare for such pairing

3. Utilizing sidebar for letter variation
This idea was mailed to me by "Tijara", I am adding some improvements
Once you start typing a letter the sidebar can transform and show the letter variations, if available, for that character

Pros

Easy to understand

Cons

I see technical challenges implementing such a change
you will have to now type 2 handed
It might not be as user friendly since the sidebar can be on either side and your typing hand can be obstructing it.

4. Single Gesture to cycle through diacritics
This one is my favorite.
Once a letter is entered, the following gesture can be repeated to cycle through the diacritics till the user has found what he/she is looking for.
Gesture
Circle -> bottom -> Circle

It is possible that the diacritic you are looking for happens to be last in line usually, so the system can statistically move the more used character variation up the chain automatically as the user uses the keyboard more and more.
Pros

Easy to understand
Easy to remember
As the time passes, it gets easier to reach the more frequently used characters
It uses a gesture which I have not found any use of till now
blends with the typing flow

Cons

Cycling through the characters might be laborious in the beginning
New user will have no indication that this can be done
Will have to specify which diacritics are variations for which character, although I am assuming there must exit some kind of de-facto standard for the same.

I hope my ideas make sense as most of them have been derived from the ongoing discussion. If any idea seems particularly stupid, don't hold back and say so. If any idea is not clear, please point to it's serial number and ask the specific question haunting your thoughts.

 @foolsbazar could you please let us know why you chose to have this behavior in your app and if you had positive or negative user feedback?

Sure I can!
First of all I shell say the the user base of Sailfish OS was too small to have this many people interested in the keyboard. Probably daily users could be counted just with the fingers of one hand, so received feedbacks were mainly from beginners.
Main design choices came from personal experience, and what I found is that this keyboard needs to allow a lot of personalization. But i will discuss this maybe elsewhere.
Basically the option i chose was the same as @flide N. 4.  By any practical mean that is the faster and more intuitive. The original developer chose to perform the "choose and hold" method, in order to activate a different layout with the accented letters.
I'm sorry I will sound impolite, but anything that stopped the flow of the swiping was extremely uncomfortable to me.  The same as raising a finger in order insert a space.. very uncomfortable if you swipe with your thumb (acceptable when swiping with the index finger or a stylus) . Personally I firmly believe that it should be possible to type an entire message without lifting the thumb from the keyboard, hence solution 4 is the best.
The reason why I chose to use the method "select character"-"select modifier"-"print modified character" is that I find it more intuitive than "select character"-"select modified character"-"print modified character". Moreover it feels more similar to an actual handwriting experience (you first write the letter and then you add a modifier to it). I should mention that this method was actually introduced by the original author of the keyboard.
Anyway, I also felt less disorienting to deal with number of layouts: you have the main layout - the modifier layout - and special chars layouts. With the method proposed by @flide he will need to include a layout for any letter (although they will all be organized in the same way), BUT i can understand that for a beginner this would be much easier because he doesn't have to learn how to modify a character. Moreover certain letters how the heck are described as letter+modifier? d+/ = ð, s+?? = ß, o+/=Ø, a+e? =æ, o+e? =œ,l+/= ł .. (I didn't implement those ones, because the way the keyboard was working would have forbidden you to type a forward slash after an "o" without typing ø.. my bad)

Regarding language specific gestures (solution N.1) I don't know what to say. If @flide wants to provide multi lingual support, at some point he will need to provide different base layouts, there is no escape for that. You see, letters can be placed in a certain order in the layout not just according to they occurrence frequency, but also the frequency of occurrence of the following letter.. And these are strictly bound and peculiar to each language.
But for gestures? I don't know.. It feels wrong to me to include language specific gestures. I'd rather opt for user specific gestures.

We can discuss "Entering space and backspace without lifting a finger" on a different thread, let this one run diacritics.
Please.

We can discuss "Entering space and backspace without lifting a finger" on a different thread, let this one run diacritics.
Please.

Not sure how hard we are on this, but here's a thought that concerns both editing and diacritics: maybe we should not try to cram all the editing-shortcuts into this limited space.
I mean there is this thing after all ...

... and while I have no idea what half of it does, it seems like a good attempt to place (somewhat) common editing features in a place that doesn't interfere with the main keyboard.
Diacritics need to be as easy to write as normal letters, in many languages there are some diacritics that are more common than "normal" letters.
A multi-step process may not sound that bad for people that do not need diacritics on a frequent basis, but something like ...

Write letter
Do the start-adding-diacritics - motion
actually add a diacritic (maybe by doing another letter-like motion)

... sounds like a BIG hassle.
If there's no really easy and fast way of inserting them, non-English-speakers will be forced away and adding diacritics will only be a niche feature, in case someone needs to write "déjà vu" the correct way.

Which makes inputting faster and way more efficient when using @reacherjack422's 1st idea.

Actually, in languages that use less than four non-standard letters (anything that's not a-z), the motion above could straight up insert the letter, no modifying needed.

Just a small interesting brainwave: what about trying to solve all these special character issues independent of the chosen language in a compose key fashion, i. e. one compose gesture and then a couple of recorded standard gestures until there is a match with a predefined combination. So ä could be a" and ga could be α, for example. One could even allow the user to extend the list of combinations. I haven't really thought this through to the end, though, and very common characters should probably be accessible faster, depending on language.

What about trying to solve all these special character issues independent of the chosen language in a compose key fashion, i. e. one compose gesture and then a couple of recorded standard gestures until there is a match with a predefined combination. So a" could be ä and ga could be α, for example.

@qoheniac That sounds like a dream come true, and i wish this was possible, but I'm not sure it is. I can see two problems:

All those combinations fall short when we want to, well, actually type out the keys instead of combining them.
By that I mean: When we want to write
"USA", said the gambler to the flight attendant.
it would turn into
"USÄ, said the αmbler to the flight attendant.
Which is annoying. We could add some command to "literally use what we are typing next", but that would be an extra command whenever we write a certain combination of letters. Or when typing diacritics. Both options are undesirable.
That would make diacritics really complicated to write. Just think about it. writing ä would require you to

write an a
turn on uppercase
make sure with some weird trick that there's no space after the a (remember, you had to remove your finger from the keyboard to turn on uppercase)
write ", which requires crossing 4 borders.

That means you would need to cross 5 borders (a + "), and click at least once (uppercase) to input a letter that, in German for example, is used more frequently than j, x, q, and y combined.
Which means this is a problem for the many languages that use diacritic-letters regularly, more than some "regular" letters even.

I haven't really thought this through to the end, though, and very common characters should probably be accessible faster, depending on language.

I agree that a universal system sounds nice to have (and it would be ok to make this system a little more complicated as you would rarely use it), but I think what most people of diacritic-using languages are in dire need of is a way to input the diacritics that are important to THEM
A universal layout may be a nice add-on but it's not the most important thing for many people.

(Also, deciding on a way for people to add 4 letters/diacritics might be easier and faster right now than trying to find the ultimate layout that allows us to choose between and input all 50 special letters.)

Github swallowed the <compose> in my post and I had to do an edit, so you probably did not read it. Sorry for that! What I meant was that you make a compose gesture (<compose>) bringing you to "compose mode" and everything after that is interpreted as a combination to create a specific symbol, i. e. the code looks if this combination exists or a combination that is starting this way.

case: combination exists -> insert corresponding symbol and switch back to "normal mode"
case: multiple combinations starting this way exist -> wait for further input
case: combination does not exist -> ignore input and switch back to "normal mode"

This is exactly how XCompose works. So for example we could define <compose>o/ to be ø and <compose>o" to be ö. To write Brød you would use these gestures <shift>Br<compose>o/d. (By the way I did exactly that to write Brød on my laptop using XCompose and I guess this approach would feel very familiar to many Linux users.)

This was regarding your "USÄ example. You are right that a" would take too many gestures. I was just using that combination, because that's the default in XCompose. Of course, common letters with or without diacritics should be easily accessible and therefore there cannot be one layout fitting all languages. Stiil, a compose feature could be useful for loanwords, math symbols, emojis, greek letters and so on.

@qoheniac your proposal works very well when the user has to type those characters once in a while. It is not comfortable if one has to use it even once a day, but it is great to insert a letter that one time you have to.
In my opinion the problem that could arise is that using such a feature so rarely the user might not remember the right combination and might therefore have to look it up, which wouldn't be good.
At this point for such a feature it might be better to have a general popup to select the letter from...
However this is in my opinion quite a secondary function to add: most keyboards do not allow entering any character on smartphones, you are often limited to the characters used in your language and another few common ones.
It is in my opinion important to find a way to insert the common diacritics for a language now, after having decided such mechanism that should be implemented for the single languages.
I find issues such as #90 and #96 quite useless until a mechanism for entering diacritics is decided.
It could even be counterproductive as if a German layout is implemented following a certain logic then other languages would have to be implemented following the same logic. However we have seen that other Latin scripts have issues that the German language doesn't have. Therefore if the German language is implemented first in a rush, it is not obvious that it will then be possible to implement Spanish in the same way.

A verification needs to be in place to not override any existing gesture, going to be a maintenance nightmare

If a mechanism for inserting diacritics is clearly decided and used only for diacritics there will be no overriding and therefore no problem will arise.
Many ideas in these comments are good solutions to all the problems, though it is necessary to decide which one to use to implement this mechanism.
I find @reacherjack422 first idea the easiest and fastest way to include diacritics for a single language.
I like @flide idea number 4, it is slightly slower and it may allow more than 4 diacritics, although circling more than 5 times around the same sector can become quite numbing and it is therefore not a possible mechanism to insert all the diacritics.
I won't go into detail but I consider all other methods proposed inferior to these two, my opinion is solely based on speed: one must realize that a letter with a diacritic is just a letter like any other one, typing it must therefore be as easy and as fast as possible or otherwise the usability of the keyboard is compromised.

Only one language can be turned on at a time and will infuriate multi-linguals.

Not really; right now all keyboards provide only one language at a time and this is therefore normal. Moreover when writing one generally uses only one language and therefore has just to select the language once, before he starts writing. I guess translators have sometimes to mix two languages, but that is quite a small use case.
Moreover both methods I cited could allow customization and support diacritics from more than a language, although in a limited amount.
Regardless of which mechanism to insert diacritics is chosen I believe that layouts for each language still have to be made. The language is not only composed of the different diacritics. As already pointed out the frequency of each character varies greatly between languages, therefore having the same layout for different languages makes no sense. At this point it would be best to also customize the insertion of diacritics for each language.

I agree with pretty much everything @davidoskky said.
However, I think while @flide's idea number 4 is a worthy consideration of how to add many diacritics, it seems too slow to be the main method of adding diacritics. (For languages like German it would be fine, but for French or Italian, it slows the user down)
Also, I just wanted to re-iterate that in languages that only use 4 diacritics (which are most of them), we might want to consider those three options:

Use @reacherjack422's 1st idea to add diacritics to a letter (this gives us a lot of optioins as it makes us able to add one of 4 diacritics of our choosing to any letter. Also, it is almost the fastest movement we have.)
Use @reacherjack422's 1st idea to insert letter WITH diacritic in just one step. (This is best if we only ever need to use 4 modified letters. (for example, German only has ä, ö, ü, and ß. Also, this movement (when inserting a letter directly) seems to be tied for the fastest movement)
Use the "Breaking some rules :)"- movement (proposed here: https://github.com/flide/8VIM/issues/65#issuecomment-719968360) This is great and fast, the only downside being that you can only add one diacritic to each letter. For example (if that's what your settings are set to), you could only add é, but not è. You might want to insert â, but that means you can't insert ä. And for some languages that might be a problem. (French or Italian for example)

While all those movements seem fast and usable, I think the third option unfortunately is too limited for most languages.
So, in my opinion, we should implement @reacherjack422's 1st idea, either going for the middle or the top option. (Of course, ideally, users could choose which of these two implementations of @reacherjack422's 1st idea they use)

Hi!
I've just "discovered" this keyboard - what attracted my attention was its name. Looks very promising for me (I have to train my skills in using it, though). But one thing worries me. I can't see polish diacritics in this topic, which are: ą, ć, ę, ł, ń, ó, ś, ź, ż, and respectively: Ą, Ć, Ę, Ł, Ń, Ó, Ś, Ź, Ż. So I have kind request to take them into account, when diacritics will be added to 8VIM.
PS.Currently I'm using 8pen wordcup game to learn 8VIM. The problem is, it doesn't work with my phone and I have to use the old one (with older Androd version). Are there any good alternatives?

@Kamul-PL no alternatives yet my friend.

So, let's finally get this issue solved.
Thoughts, anyone? I stll love @reacherjack422's idea.

My personal vote would be for foolsbazar's method, mainly because you still keep the original feel of typing the letter, and then add to it, and can see how to add.
May I add a suggestion to instead of showing all umlauts, if I've chosen an A, show all the ones with an A. That way when I input a question mark, go into umlauts, there will only be ¿.
For the s that becomes a ß, just do an S character, go to umlauts, then in the B letter's place, put that character, so I type s+umlaut_mode+b.
æ = a+umlaut_mode+e.

Thank you for sharing your view on the issue! :)
It's a nice idea, but if at all, it's a keypad for inputting ALL diacritics at a moderately slow speed. Which IS NEEDED.
However, what all the diacritics-languages need right now is a FAST method to input mostly THEIR OWN diacritics.
There's stuff like

hundrets of 1-2% diacritics
ä in Finnish (3.577%)
ı in Turkish (5.114%)

We

Basically, we need two things:

A fast way of inputting a few diacritics
A (doesn't matter how fast) way of inputting all diacritics.

And the first thing is more important right now.

https://github.com/flide/8VIM/discussions/133


⚠ Bofrost.be has degraded performance

In e84fd5c, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 814 ms

Resolved: Bofrost.be performance has improved in 20db1f3 after 6 minutes.


[HIVEMALL-71] Handle null values and add a unit Tests to RescaleUDF.

What changes were proposed in this pull request?
Handle null values and add a unit Tests to RescaleUDF.
What type of PR is it?
Improvement
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-71
How was this patch tested?
unit tests

Coverage increased (+0.1%) to 35.914% when pulling 464e7dba8c0f5305db31075cc964bd4c5eaad972 on wangyum:HIVEMALL-71 into 6b462ae45fcf44987e0a0ec49dfa5e4b4a8855e1 on apache:master.

Thanks, I will take this PR on.

Coverage increased (+0.4%) to 36.222% when pulling 661571da0aa6a67470e4e703801ad0efeb65327b on wangyum:HIVEMALL-71 into 6b462ae45fcf44987e0a0ec49dfa5e4b4a8855e1 on apache:master.

Coverage increased (+0.4%) to 36.222% when pulling 661571da0aa6a67470e4e703801ad0efeb65327b on wangyum:HIVEMALL-71 into 6b462ae45fcf44987e0a0ec49dfa5e4b4a8855e1 on apache:master.

@wangyum Thanks. Merged with some modifications.
select rescale(v,min,max) from (
    select cast(1.2 as float) as v, 1.0 as min, 10.0 as max
    union all
    select cast(null as float) as v, 1.0 as min, 10.0 as max
    union all
    select cast(1.2 as double) as v, 1.0 as min, 10.0 as max
    union all
    select cast(null as double) as v, 1.0 as min, 10.0 as max
    union all
    select cast(1.2 as double) as v, cast(5.0 as double) as min, cast(10.0 as float) as max
) t;

0.022222228
NULL
0.022222228
NULL
-0.76


[NEW ALGORITHM] stack implementation using LinkedList

Issue will be closed if:

You mention more than one algorithm. You can create a separate issue for each algorithm once the current one is completed.
You propose an algorithm that is already present or has been mentioned in a previous issue.
You create a new issue without completing your previous issue.

Note: These actions will be taken seriously. Failure to follow the guidelines may result in the immediate closure of your issue.

Name:
[NEW ALGORITHM]:stack implementation using LinkedList
About:
Propose a new algorithm to be added to the repository

Labels:
new algorithm, gssoc-ext, hacktoberfest, level1

Assignees:

[ ] Contributor in GSSoC-ext
[ ] Want to work on it

I would like to work on this issue

I would definitely like to work on this


feat: add Tenant Invitations and Tenant Members operations

Description

Adds new methods to manage Tenant Members

bt.tenants.listMembers
bt.tenants.deleteMember

Adds new methods to manage Tenant Invitations

bt.tenants.createInvitation
bt.tenants.resendInvitation
bt.tenants.listInvitations
bt.tenants.retrieveInvitation
bt.tenants.deleteInvitation

Testing required outside of automated testing?

[X] Not Applicable

Screenshots (if appropriate):

[X] Not Applicable

Rollback / Rollforward Procedure

[X] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 1.24.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


server_generic option stopped working

It seems that in the latest release the server_generic option no longer generates any output. The reason for that is line where the options are generated:
https://github.com/timostamm/protobuf-ts/blob/4304289f4ed83f60d9ba5d6b1b55abc48db84d8d/packages/plugin/src/our-options.ts#L307-L309

The fix is released in v2.2.1


Scrutinizer Auto-Fixes

@SysCodeSoft requested this pull request.
It consists of patches automatically generated for this project on Scrutinizer:
https://scrutinizer-ci.com/g/SysCodeSoft/Sofi-Router/

Fixed


setup npm install socket-tunnel

hello @ericbarch.
I just created the PR that help us install this repo as a npm library.
Thanks
#7

@tuanngocptn appreciate the contribution!


swift compatability didCompletePasswordAuthenticationStepWithError should pass an Optional NSError

For swift compatability didCompletePasswordAuthenticationStepWithError should pass an Optional NSError to the delegate.  Swift works better if anything that can be nil is declared as Optional, and in the case of a successful login (authentication) didCompletePasswordAuthenticationStepWithError is passed a nil error.
This is really a problem with the declares in the API and not the sample.

fixed with 2.4.6

Thanks!


Set default remote for gh commands

gh commands require a default remote to be set before they can be run inside working copies. It would be useful to add a step in our clone process that sets each remote default correctly using gh repo set-default.

In light of https://github.com/cli/cli/pull/7768, I doubt this is still worth doing on our side.

Closing as this is now fixed in github cli.


Switch the Prometheis to m5.large instances

We recently discovered that the Prometheis for PaaS (the only ones we
run via this Terraform) are m4.large. We can save ourselves some cash by
switching them to m5.large, the next generation.
We had more variables than we needed for this, which I think were left
over from when we had ECS EC2 Prometheis, dev stacks, and the "enclave".
This gets rid of those extra variable settings in
prom-ec2/paas- projects and sets the default module
instance_size variable to m5.large.

Deployed to staging and discovered we needed ffed8d8 because M5 instances have different device names. It works now we've applied that change too.


Ascendex: Fail to cancel an order due to order status not found

Describe the bug
The client keeps tracking an invalid order  (below the exchange allowed size) resulting in failing to cancel an order and continuously receiving an order not found status.
2022-01-10 03:47:30,004 - 17391 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ONE-USDT) Creating 1 bid orders at (Size, Price): ['2E+1 ONE, 0.183457 USDT']
2022-01-10 03:47:30,004 - 17391 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ONE-USDT) Creating 1 ask orders at (Size, Price): ['2E+1 ONE, 0.504508 USDT']
2022-01-10 03:47:30,115 - 17391 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - The request to create the order HMBot-BONEUSD1641786450004733 failed
Traceback (most recent call last):
  File "/home/ubuntu/ascendex/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 541, in _create_order
    resp = await self._api_request(
  File "/home/ubuntu/ascendex/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 426, in _api_request
    raise IOError(f"{url} API call failed, response: {parsed_response}")
OSError: https://ascendex.com/6/api/pro/v1/cash/order API call failed, response: {'code': 300004, 'message': 'The order notional cannot be less than 5.0', 'reason': 'INVALID_NOTIONAL', 'accountId': 'cshrDwJt4MmbOPZl3pt5StX2zg2B7uSe', 'ac': 'CASH', 'action': 'place-order', 'status': 'Err', 'info': {'id': 'HMBot-ltktRJ9hurCzYO4bKb4GSI7UpI', 'symbol': 'ONE/USDT'}}

2022-01-10 03:48:00,020 - 17391 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ONE-USDT) Cancelling the limit order HMBot-BONEUSD1641786450004733. [clock=2022-01-10 03:48:00+00:00]
2022-01-10 03:48:00,021 - 17391 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ONE-USDT) Cancelling the limit order HMBot-SONEUSD1641786450004982. [clock=2022-01-10 03:48:00+00:00]
2022-01-10 03:48:00,021 - 17391 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - INFO - The order HMBot-BONEUSD1641786450004733 was finished before being cancelled
2022-01-10 03:48:01,054 - 17391 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>945, "order_id": "HMBot-SONEUSD1641786450004982", "exchange_order_id": "HMBot17e42192056U9496830767iMzMQ", "event_name": "OrderCancelledEvent", "event_source": "ascend_ex"}
2022-01-10 03:48:01,054 - 17391 - hummingbot.connector.client_order_tracker - INFO - Successfully cancelled order HMBot-SONEUSD1641786450004982.
2022-01-10 03:48:04,945 - 17391 - hummingbot.connector.exchange.ascend_ex.ascend_ex_api_order_book_data_source - INFO - Subscribed to ['ONE-USDT'] orderbook trading and delta streams...
2022-01-10 03:49:01,220 - 17391 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ONE-USDT) Cancelling the limit order HMBot-BONEUSD1641786450004733. [clock=2022-01-10 03:49:01+00:00]
2022-01-10 03:49:01,221 - 17391 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - Failed to cancel order HMBot-BONEUSD1641786450004733: Failed to cancel order - HMBot-BONEUSD1641786450004733. Order not found.
Traceback (most recent call last):
  File "/home/ubuntu/ascendex/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 640, in _execute_cancel
    raise ValueError(f"Failed to cancel order - {order_id}. Order not found.")
ValueError: Failed to cancel order - HMBot-BONEUSD1641786450004733. Order not found.

Steps To Reproduce
Here are the steps to reproduce the issue (see attachments in the section below):

Create a pure market-making strategy or an Avellaneda market-making strategy
Set a very widespread or adjust the risk factor for Avellaneda strategy so that once orders are created they will be very far from the mid-price.
Start the strategy
Run status --live and monitor the log pane

Screenshots

Release version
dev 1.0.0
Attachments
logs_pmm-order-not-found-test.log
logs_order-not-found-ave.log
WARNING: Do NOT publish any exchange API keys or your wallet's private key. Whoever has access to them may steal your assets!
Optional: your discord username:

To expand on the issue, it not only happens when the order is smaller than the minumum for the exange. If the request to create fails because of HTTP status 429 error, the same behaviour is observed.
2022-02-13 21:36:59,321 - 1 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - The request to create the order HMBot-BMAHUSD1644788219189592 failed
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 541, in _create_order
    resp = await self._api_request(
  File "/home/hummingbot/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 420, in _api_request
    raise IOError(f"Error calling {url}. HTTP status is {response.status}. " f"Message: {resp_text}")
OSError: Error calling https://ascendex.com/3/api/pro/v1/websocket-for-hummingbot-liq-mining/cash/order. HTTP status is 429. Message: <!DOCTYPE html>

<title>Access denied | ascendex.com used Cloudflare to restrict access</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" type="text/css" media="screen,projection" />

Please enable cookies.

           Error
           1015

         Ray ID: 6dd13202bb7e8a92 •
         2022-02-13 21:36:59 UTC

You are being rate limited

What happened?

The owner of this website (ascendex.com) has banned you temporarily from accessing this website.

            Was this page helpful?
            <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button>
            <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button>

            Thank you for your feedback!

    Cloudflare Ray ID: 6dd13202bb7e8a92
    •
    Your IP: <IP_ADDRESS>
    •
    Performance & security by Cloudflare

2022-02-13 21:37:51,219 - 1 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - Failed to cancel order HMBot-BMAHUSD1644788219189592: Failed to cancel order - HMBot-BMAHUSD1644788219189592. Order not found.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 640, in _execute_cancel
    raise ValueError(f"Failed to cancel order - {order_id}. Order not found.")
ValueError: Failed to cancel order - HMBot-BMAHUSD1644788219189592. Order not found.

The main issue is that the logging gets filled up. I've had now 5 robots will 100% full file systems filled with gigabytes of this log message. Once the file system is full the robot can no longer write to the sqlite database nor the csv file to keep track of the trades.
2022-02-13 21:37:51,219 - 1 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - Failed to cancel order HMBot-BMAHUSD1644788219189592: Failed to cancel order - HMBot-BMAHUSD1644788219189592. Order not found.
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 640, in _execute_cancel
    raise ValueError(f"Failed to cancel order - {order_id}. Order not found.")
ValueError: Failed to cancel order - HMBot-BMAHUSD1644788219189592. Order not found.

This is los happening to some of my bots running Avellaneda (mostly on Ascendex)

reproduced on staging 1.4.0 -
ValueError: Failed to cancel order - HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75. Order not found.
logs_conf_pure_mm_1.log
confdatalogs-files-staging 1.4.0.zip
2022-05-06 02:28:18,047 - 32423 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ETH-USDT) Creating 1 bid orders at (Size, Price): ['0.004 ETH, 2726.57 USDT']
2022-05-06 02:28:18,047 - 32423 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ETH-USDT) Creating 1 ask orders at (Size, Price): ['0.004 ETH, 2781.66 USDT']
2022-05-06 02:28:18,161 - 32423 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - The request to create the order HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75 failed
Traceback (most recent call last):
  File "/home/ubuntu/sta2/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 937, in _create_order
    resp = await self._api_request(
  File "/home/ubuntu/sta2/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 1137, in _api_request
    raise IOError(f"{url} API call failed, response: {parsed_response}")
OSError: https://ascendex.com/7/api/pro/v1/websocket-for-hummingbot-liq-mining/cash/order API call failed, response: {'code': 300011, 'message': 'Not Enough Account Balance', 'reason': 'INVALID_BALANCE', 'accountId': 'cshI5CreZOaXyYJeNwZ42iKqrB9VGJ2K', 'ac': 'CASH', 'action': 'place-order', 'status': 'Err', 'info': {'id': 'HMBot-XuRLXMzy83TNVezek8C2WgiJLQ', 'symbol': 'ETH/USDT'}}
2022-05-06 02:28:18,238 - 32423 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT_MAKER SELL order HMBotSEHUT18097324e00c56b7b1e9eef984229d17ec44528be75 for 0.004 ETH-USDT.
2022-05-06 02:28:18,247 - 32423 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "ETH-USDT", "amount": "0.004", "price": "2781.66", "order_id": "HMBotSEHUT18097324e00c56b7b1e9eef984229d17ec44528be75", "creation_timestamp":<PHONE_NUMBER>.0, "exchange_order_id": "HMBot18097324e02U0290229957TyNZX", "leverage": 1, "position": "NIL", "event_name": "SellOrderCreatedEvent", "event_source": "ascend_ex"}
2022-05-06 02:28:38,001 - 32423 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ETH-USDT) Canceling the limit order HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75. [clock=2022-05-06 02:28:38+00:00]
2022-05-06 02:28:38,001 - 32423 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ETH-USDT) Canceling the limit order HMBotSEHUT18097324e00c56b7b1e9eef984229d17ec44528be75. [clock=2022-05-06 02:28:38+00:00]
2022-05-06 02:28:38,002 - 32423 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - INFO - The order HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75 was finished before being canceled
2022-05-06 02:28:38,119 - 32423 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "HMBotSEHUT18097324e00c56b7b1e9eef984229d17ec44528be75", "exchange_order_id": "HMBot18097324e02U0290229957TyNZX", "event_name": "OrderCancelledEvent", "event_source": "ascend_ex"}
2022-05-06 02:28:38,119 - 32423 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order HMBotSEHUT18097324e00c56b7b1e9eef984229d17ec44528be75.
2022-05-06 02:29:39,028 - 32423 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (ETH-USDT) Canceling the limit order HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75. [clock=2022-05-06 02:29:39+00:00]
2022-05-06 02:29:39,029 - 32423 - hummingbot.connector.exchange.ascend_ex.ascend_ex_exchange - ERROR - Failed to cancel order HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75: Failed to cancel order - HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75. Order not found.
Traceback (most recent call last):
  File "/home/ubuntu/sta2/hummingbot/connector/exchange/ascend_ex/ascend_ex_exchange.py", line 718, in _execute_cancel
    raise ValueError(f"Failed to cancel order - {order_id}. Order not found.")
ValueError: Failed to cancel order - HMBotBEHUT18097324dffc56b7b1e9eef984229d17ec44528be75. Order not found.

That issue showed on latest staging (dev-1.5.0) when tested with multiple order levels and insufficient balance for place every order for one side.


Media repo memory usage should be predictable/calculable

So people don't have to build spreadsheets, graphs, and guess as to how much memory they're going to use. In order to achieve this it might make sense to have #15 done, as well as ensuring that the memory usage of the repo is consistent through the entire request process (ie: no spikes during upload, no weirdly high usage because of the cache, etc)

This is now done. See https://github.com/turt2live/matrix-media-repo/issues/415


Part 1

drawing the @ symbol to the screen and moving it around

the final addition of unit tests means part 1 is finished! we can merge and close up this branch :)


Create tk swamp helper

Helps you when you get the swamp room in the temple trekking, tells you which way to go without losing time for nothing since the pathing is terrible

Is there something wrong that it's not being checked on by the mods?

Is there something wrong that it's not being checked on by the mods?

I think there's just a big backlog since the mods that review hub PRs have been busy

FYI we originally closed access to the developer tools plugin from the launcher because players were using it to solve the temple trekking swamp puzzle. That said, that was a change made years ago, so this is probably allowable under current client guidelines. I'm prioritizing plugin updates in the queue at the moment, but will make sure to get back to this in time.

We've decided this is probably fine now under current client guidelines.


gha(python): fix W391 blank line at end of file

flake8 --select=W291,W292,W293,W391 .
./test/integration/smoke/test_register_userdata.py:766:1: W391 blank line at end of file
./test/integration/component/test_network_vpc_custom_dns.py:732:1: W391 blank line at end of file
Description
This PR fixes the flake8 test

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[X] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[X] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[X] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@jbampton are you running these tests locally? they are not run in any of the travis jobs.

Yes I run flake8 locally.  But it runs with the GitHub Actions SuperLinter Check on modified files only on pull requests:
https://github.com/apache/cloudstack/actions/runs/3454836056/jobs/5766433087

In future we could add flake8 to pre-commit to allow everyone to run the tests locally on git commit.
There is another pull request where I was adding the first check with pre-commit.
refs #6273

daring to go against principle on this one, only empty lines on end of files


HttpTrigger function has empty body when content-length is empty or zero

When client-side code sends a request to the isolated azure function HTTP trigger and content-length can not be calculated, then HttpRequestsData.Body == Stream.Null;
As a result - all HTTP clients that use System.Net.Http.Json - not working, and there is no way to send a stream without buffering it.
Is there any workaround except buffering?
Examples:
using System.Net.Http.Json;
using System.Text;
using System.IO;
using System.Net.Http;
using System;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

public class ClientSideCodeExample
    {
        private readonly HttpClient _client;

        public ClientSideCodeExample(HttpClient client)
        {
            _client = client;
        }

        public class Request2
        {
            public string? Name { get; set; }
        }

        public async Task Example1Async(Stream stream, CancellationToken cancellationToken = default)
        {
            using var content = new MultipartFormDataContent
            {
                { new StreamContent(stream), "attachment", "fileName.txt" },

            };

            using var response = await _client.PostAsync("api/AzureFunction", content, cancellationToken);

        }

        public async Task Example2Async(Request2 request, CancellationToken cancellationToken = default)
        {
            using var response = await _client.PostAsJsonAsync("api/AzureFunction", request, cancellationToken);
        }
    }

public static class AzureFunctions
{
    [Function("AzureFunction")]
    public static async Task Run([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestData request,
        FunctionContext executionContext)
    {
            if( request.Body == System.IO.Stream.Null)
            {
                //will throw the error for all requests in the example
                throw new InvalidOperationException();
            }
    }
}

@ramonesz297 this is not supported by the out-of-process models in Functions today, but we have an issue tracking work that is currently in progress to address this limitation.
You can find more information about the work here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1008 and new issues with the feature: HTTP Types label will start being assigned to sprints soon.
Closing this in favor of the linked, issue, but please let us know if you have any additional questions.


There is no ./build/iceoryx_examples/icedelivery/ice_sender and ./build/iceoryx_examples/icedelivery/ice_receiver

Required information
Operating system:
Ubuntu 18.04
Compiler version:
E.g. gcc 7.5.0
Observed result or behaviour:
the console output bash: ./build/iceoryx_examples/icedelivery/ice_sender: No such file or directory when I run docker exec -it roudi /bin/bash,before that, I installed iceoryx using the command ./tools/docker/build_and_run.sh  and I replace the source image ubuntu:bionic to ubuntu:18.04 in the dockerfile, and add the options build-test clean release after the command ./tools/iceoryx_build_test.sh .
Additionally, when I install iceoryx with command ./tools/docker/build_and_run.sh with ./tools/iceoryx_build_test.sh options test clean release, it shows many errors during run tests.
Expected result or behaviour:
What do you expect to happen?
As the README.md shows:
Sending: 0
Sending: 1
Sending: 2
Sending: 3

Conditions where it occurred / Performed steps:
Describe how one can reproduce the bug.
Each time

@zzzzzzli Thank you for your report! It seems that we did not keep track of our docker documentation when we updated the file names of our icedelivery example. We updated the documentation and when the pull request https://github.com/eclipse/iceoryx/pull/159 is merged the documentation should be correct again.
In the meantime you can replace the path to the sender with: ./build/iceoryx_examples/icedelivery/ice-publisher-simple and the receiver is located at ./build/iceoryx_examples/icedelivery/ice-subscriber-simple.

Thank you very much.


Give attention to and triage in our pull request list

We have many PRs that have gone stale, to a large degree because we've had a low capacity to review them. This is a bit problematic in itself as its not a fun experience for anyone involved.
Currently, resolving this would unblock #1381 which is deemed to cause too much disruption on currently open PRs to be worth merging at this point. But I'm most interested in getting to a state where we have a healthy state and concrete action plan for all PRs rather than unblocking #1381 which can wait.
How do we address this?

I think the action to take to reduce the PR backlog is simple and hard at the same time: start reviewing PRs and making decisions on merge vs close. It is simple because "start reviewing" is only two words, it is hard because reviewing PRs is hard work.
AFAIK the disruption from #1381 would be merge conflicts. So another way to deal with that (for example for PRs that we want to land but will take a while or need discussion before a decision can be made) is to provide an instruction on how to run the tools on the branch of a PR. I think it should be possible to resolve conflicts by accepting the PR branch's changes when there is a conflict, running the reformatting tools, making a new commit with the formatting changes. My thinking is that the places where there are conflicts are because on master there are formatting changes and in the PR's branch the logic changed. So accepting the PR branch's change makes sense, then reformatting to get it consistent.
More experienced users might be able to do clever things with rebasing their PRs branch onto the latest master. But I am not sure I'd want to provide support to PR authors who don't go down this route themselves. This means unless we have a bullet proof recipe, I'd not suggest this.


io_uring Benchmark

Could a proper(file, socket, features) benchmark be written so others can compare io_uring speed with their software and so on... Maybe this can double as test to see if new/comparing feature yields benefit or not.
Everyone writes their own and I feel like they always miss something or do it wrong!
Thanks

Good idea. One standard bench is fio/t/io_uring, but there are other interesting cases like sockets. The only thing left is to find a volunteer :)

Agree it's a good idea, the problem is usually finding either the time or someone to do it... Disk io is simpler and already has some options, but networking is a pretty broad topic.

I have a repo which compares networking performance between io_uring and epoll with various operations
https://github.com/CarterLi/io_uring-echo-server
I'd like to contribute if someone is interested in it.

@CarterLi, would be awesome! Does it require a 3rd-party client?

It does require a 3rd party client. Would be nice if we had a client as well for this kind of purpose.

It should be Ok, if that's a well known protocol / tools, e.g. netperf. So curious if that's the case or it's a yet another custom thing. From a quick look it appears to be the second.

It's a well known client, most people end up using it.

It does require rust, though...

It's a well known client, most people end up using it.

I can't find a package in AUR, though. Not as popular as I'd wish. Or do I miss it?

I was planning to implement client with io_uring & epoll too, but I found that async programming is rather complex.
I'd like to use some coroutine library like what I did in the example, but not ucontext because it's slow and not available on some libc implementation.
Is it ok to use some 3rd party library such as https://github.com/hnes/libaco?

It's a well known client, most people end up using it.

I can't find an AUR package, though. Not as popular as I'd wish. Or do I miss it?

I don't think It's a popular tool.
My io_uring-echo-server is just an echo server, the easy way to test it is to use telnet. But for benchmarking, you need some special tools.

Thanks everyone and @CarterLi for taking on this task. To give some more contest

This benchmark should be simple anyone can use/run using their favorite tools (could recommend what to use)
Code must be reproducible, so others can write something equal in their language/software to compare. Like if you write something only you know what's going on, its too complex.
Benchmark should only use functions implemented in liburing (if available), for example if liburing has io_uring_prep_openat should use it and not open()

Lets starts with few very basic file/socket benchmarks then go from there?
The goal is to provide standardize benchmark! Jens posts IO benchmark and there is no way for anyone else to test it, run it on a different system to see if it produces similar results. By comparing a standard benchmark, on different system/setup you can improve your code, find bugs, ...

Jens posts IO benchmark and there is no way for anyone else to test it, run it on a different system to see if it produces similar results

That's not true, what I run is publicly available, and I also tell people exactly what I run and on exactly what hardware. It's the nature of benchmarks that they will produce different results depending on your setup, I don't think that is that important. What is important is showing people what they can expect from their system, and best practices in getting the best performance out of it.

Here's the one I use as a basic echo server. It's based on the one that freevib posted a while back. Not claiming it's the best thing out there, but it's a starting point. Then I use the same client that was posted above, and run it ala:
cargo run --release -- --address "<IP_ADDRESS>:1234" --number 20 --duration 10 --length 64

which yields ~430K/sec on my test box.
#include <liburing.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <strings.h>
#include <errno.h>

#include <stdlib.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/poll.h>

#define MAX_CONNECTIONS		1024
#define BACKLOG			128
#define MAX_MESSAGE_LEN		2048

enum {
	ACCEPT,
	POLL_LISTEN,
	POLL_NEW_CONNECTION,
	READ,
	WRITE,
};

struct conn_info {
	unsigned fd;
	unsigned type;
};

static struct conn_info conns[MAX_CONNECTIONS];
static char bufs[MAX_CONNECTIONS][MAX_MESSAGE_LEN];
static struct sockaddr_in client_addr;
static socklen_t client_len = sizeof(client_addr);
static int sock_listen_fd;

static void add_accept(struct io_uring *ring, int fd, unsigned flags)
{
	struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
	struct conn_info *conn_i = &conns[fd];

	io_uring_prep_accept(sqe, fd, (struct sockaddr *) &client_addr,
				&client_len, 0);
	io_uring_sqe_set_flags(sqe, flags);

	conn_i->fd = fd;
	conn_i->type = ACCEPT;

	io_uring_sqe_set_data(sqe, conn_i);
}

static void add_socket_read(struct io_uring *ring, int fd, size_t size,
			    unsigned flags)
{
	struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
	struct conn_info *conn_i = &conns[fd];

	io_uring_prep_recv(sqe, fd, &bufs[fd], size, 0);
	io_uring_sqe_set_flags(sqe, flags);

	conn_i->fd = fd;
	conn_i->type = READ;

	io_uring_sqe_set_data(sqe, conn_i);
}

static void add_socket_write(struct io_uring *ring, int fd, size_t size,
			     unsigned flags)
{
	struct io_uring_sqe *sqe = io_uring_get_sqe(ring);
	struct conn_info *conn_i = &conns[fd];

	io_uring_prep_send(sqe, fd, &bufs[fd], size, 0);
	io_uring_sqe_set_flags(sqe, flags);

	conn_i->fd = fd;
	conn_i->type = WRITE;

	io_uring_sqe_set_data(sqe, conn_i);
}

static void handle_cqes(struct io_uring *ring, struct io_uring_cqe **cqes,
			int cqe_count)
{
	int i;

	for (i = 0; i < cqe_count; ++i) {
		struct io_uring_cqe *cqe = cqes[i];
		struct conn_info *user_data;

		user_data = (struct conn_info *) io_uring_cqe_get_data(cqe);
		switch (user_data->type) {
		case ACCEPT: {
			int sock_conn_fd = cqe->res;
			io_uring_cqe_seen(ring, cqe);

			add_socket_read(ring, sock_conn_fd, MAX_MESSAGE_LEN, 0);
			add_accept(ring, sock_listen_fd, 0);
			break;
			}
		case READ: {
			int bytes_read = cqe->res;
			if (bytes_read <= 0) {
				/*
				 * no bytes available on socket, client must be
				 * disconnected
				 */
				io_uring_cqe_seen(ring, cqe);
				shutdown(user_data->fd, SHUT_RDWR);
			} else {
				/*
				 * bytes have been read into bufs, now add
				 * write to socket sqe
				 */
				io_uring_cqe_seen(ring, cqe);
				add_socket_write(ring, user_data->fd,
						 bytes_read, 0);
			}
			break;
			}
		case WRITE:
			// write to socket completed, re-add socket read
			io_uring_cqe_seen(ring, cqe);
			add_socket_read(ring, user_data->fd,
						MAX_MESSAGE_LEN, 0);
			break;
		}
	}
}

int main(int argc, char *argv[])
{
	struct sockaddr_in serv_addr;
	struct io_uring_params params = { };
	struct io_uring ring;
	const int val = 1;
	int portno;

	if (argc < 2) {
		fprintf(stderr, "Please give a port number: %s [port]\n", argv[0]);
		exit(0);
	}
	portno = strtol(argv[1], NULL, 10);

	sock_listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
	if (sock_listen_fd < 0) {
		perror("socket");
		return 1;
	}

	setsockopt(sock_listen_fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));

	memset(&serv_addr, 0, sizeof(serv_addr));
	serv_addr.sin_family = AF_INET;
	serv_addr.sin_port = htons(portno);
	serv_addr.sin_addr.s_addr = INADDR_ANY;

	if (bind(sock_listen_fd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {
		perror("Error binding socket..\n");
		exit(1);
	}
	if (listen(sock_listen_fd, BACKLOG) < 0) {
		perror("Error listening..\n");
		exit(1);
	}
	printf("%d: Listening for connections on port: %d\n", getpid(), portno);

	if (io_uring_queue_init_params(MAX_CONNECTIONS, &ring, &params) < 0) {
		perror("io_uring_init_failed...\n");
		exit(1);
	}
	if (!(params.features & IORING_FEAT_FAST_POLL)) {
		fprintf(stderr, "IORING_FEAT_FAST_POLL not available\n");
		exit(0);
	}

	/* add first accept sqe, to monitor for new incoming connections */
	add_accept(&ring, sock_listen_fd, 0);

	while (1) {
		struct io_uring_cqe *cqes[BACKLOG];
		struct io_uring_cqe *cqe;
		int ret, cqe_count;

		io_uring_submit(&ring);

		ret = io_uring_wait_cqe(&ring, &cqe);
		if (ret != 0) {
			perror("Error io_uring_wait_cqe\n");
			exit(1);
		}
		/*
		 * check how many cqe's are on the cqe ring, and put these
		 * cqe's in an array
		 */
		cqe_count = io_uring_peek_batch_cqe(&ring, cqes,
					    sizeof(cqes) / sizeof(cqes[0]));

		handle_cqes(&ring, cqes, cqe_count);
	}

	return 0;
}

Here's an example using 500 clients, I'm running this:
cargo run --release -- --address "<IP_ADDRESS>:1234" --number 500 --duration 10 --length 64

and using the above included echo server:
Benchmarking: <IP_ADDRESS>:1234
500 clients, running 64 bytes, 10 sec.

Speed: 199191 request/sec, 199190 response/sec
Requests: 1991911
Responses: 1991909

and for an equivalent epoll based server:
Benchmarking: <IP_ADDRESS>:1234
500 clients, running 64 bytes, 10 sec.

Speed: 184895 request/sec, 184895 response/sec
Requests: 1848956
Responses: 1848953

This is a kernel with none of the mitigations enabled.

That's not true, what I run is publicly available,

O, ok, sorry Jens, I wasn't trying to blame you, I didn't know they were available or where they were. I meant it as if you had such tools in liburing/benchmark people could easily access and use it.

Here's the one I use as a basic echo server. It's based on the one that freevib posted a while back. Not claiming it's the best thing out there, but it's a starting point. Then I use the same client that was posted above, and run it ala:

Thanks Jens, This is what I meant by everyone has their own benchmark! If something like this could be added in liburing/benchmark now everyone can use it vs creating their own and improve your code as well. Its a win win thing.

O, ok, sorry Jens, I wasn't trying to blame you, I didn't know they were available or where they were. I meant it as if you had such tools in liburing/benchmark people could easily access and use it.

It's just important to note that it's not the case that I just post random things and nobody knows where it's from. For the more "official" posts, I reference it directly. And for a lot of others, I always mention that it's the usual test. More times than not people will then ask, and I reply. The benchmark may not be included with liburing, but it is included with fio. It doesn't use liburing as it so happens, I wrote that tool before liburing was really a thing. Hence it uses the kernel interface directly, and you can find it here:
https://git.kernel.dk/cgit/fio/tree/t/io_uring.c
Might be interesting to turn it into a liburing API instead. Would potentially also help with making sure that liburing is as efficient as it can be. Only reason I haven't done that is that I don't want to maintain two different copies of it. If it was converted and we're happy with it, then the raw API one could be dropped.

The code used by Axboe can be optimised more:

io_uring_submit & io_uring_wait_cqe can be combined into io_uring_submit_and_wait to avoid an extra syscall
for_each_cqe can be used instead of io_uring_peek_batch_cqe to avoid extra data copy
multiple io_uring_cqe_seen can be combined into one io_uring_cqe_advance

I believe that the author has updated the code in the original repo. You may use it for better results.

I personally am interested in splice for zero-copy technology. But for my old quick test with epoll it performed even worse than recv/send. Lets see will splice with io_uring will be better.

It'd be great if someone would take on that project of providing a reference implementation, not saying what I posted is it, it's just what I've run in the past. I just don't have time to take on more on that front. And I do question the validity of a simple echo server, but at the same time it's a nice and simple demo. I'm going to leave that to the folks that would potentially do this work.
samba uses io_uring with splice and has posted some impressive numbers, so perhaps that'd be interesting.

Got some numbers: https://github.com/CarterLi/liburing4cpp/tree/master#benchmark
io_uring has overall better performance than epoll and linux aio thanks to syscall batching, except:

io_uring's IORING_IO_POLL is slightly slower than linux aio's IO_CMD_POLL on high load ( 500 conns ).
IORING_IO_SPLICE is very slow, much slower than splice(2)

There's no major performance benefits using SPLICE over RECV-SEND. I think it's because SPLICE_F_MOVE is not implemented in kernel.
I will bench IORING_SETUP_SQPOLL, fixed file, fixed buffer later.

Good work, I see though later.

poll, it has never been of much importance but looks there is enough of parties now interesting in making it performant, so hopefully we'll get it.
splice is slow most probably because it's punted to io-wq, there are internal reasons for that. We can try to revisit if interesting.

splice is slow most probably because it's punted to io-wq, there are internal reasons for that. We can try to revisit if interesting.

It seems that splice will be punted into io-wq even if it can be completed inline ( i.e. prefix splice with poll wont help ). It definitely needs some revisits.

i wrote a performance testbed for QUIC servers several months ago, to compare the performance of all the open source QUIC implementations but also using io_uring vs syscalls. i haven't touched it in a few months, at the moment the only test it implements is a maximum throughput test, where recvmmsg and sendmmsg always outperform io_uring sending and recving single messages at a time (i forget the differential).
but this traffic pattern is totally alien to that of my application server, where it's... timeout fires and one PING is sent... or the server finishes processing a client message and sends back a single GSO packet. so in certain that on the send path io_uring will be significantly more efficient. the recv path i'm unsure of, and would have to test on recorded production traffic, because it depends on how many packets are waiting on socket to be recved in between application work being done. the entire server is based on C++20 coroutines, so we yield on every database or 3rd party api wait... so likely io_uring will still be the better choice.
if we had the ability to send and recv multiple messages at once with io_uring then it would surely always be faster than syscalls.
i share this anecdote to illuminate how often traffic and application patterns totally dictate best interface choices.
anyone here's the link: https://github.com/victorstewart/quicperf
if you try to build it and it breaks just let me know and i'll fix it.

I found that SETUP_SQPOLL has negative impact on performance. Does anyone have some benchmark results of SETUP_SQPOLL too?
I got similar results with/without FIXED_FILE, while the code used FIXED_FILE was much more complex than the code without FIXED_FILE. I think FIXED_FILE should only be used with SETUP_IOPOLL. I didn't bench SETUP_IOPOLL though.

I found that SETUP_SQPOLL has negative impact on performance. Does anyone have some benchmark results of SETUP_SQPOLL too?

Easily can be, it not always guarantees a performance win

I got similar results with/without FIXED_FILE, while the code used FIXED_FILE was much more complex than the code without FIXED_FILE. I think FIXED_FILE should only be used with SETUP_IOPOLL.

It's definitely not IOPOLL-only, but indeed depends on your workload. If it spends lot of time anywhere but in io_uring, e.g. overhead of network

i wrote a performance testbed for QUIC servers several months ago, to compare the performance of all the open source QUIC implementations but also using io_uring vs syscalls. i haven't touched it in a few months. at the moment the only test it implements is a maximum throughput test, where recvmmsg and sendmmsg always outperform io_uring sending and recving single messages at a time (i forget the differential).

@victorstewart, I'm curious, is it "only 1 request in-flight at a time" or just submitting by 1?
If the former, it won't beat the sync version (unless there are some features that can't be used w/o io_uring). The latter is expensive as well.

@isilence submitting hundreds at once. i forget how many but i did rough experiments to isolate the optimal batch size for throughput.

@isilence submitting hundreds at once. i forget how many but i did rough experiments to isolate the optimal batch size for throughput.

Ok, we need to try it out then and see where all the overhead is

@isilence would be great to improve the performance of the UDP path. though as i stated i think this is quite an artificial workload, i'm sure the rising tide would lift all boats.

@CarterLi Are you still planning on adding a basic file and socket benchmark into Liburing?

@CarterLi Are you still planning on adding a basic file and socket benchmark into Liburing?

As I mentioned before, I used C++ and 3rd party coroutines library to write the benchmark code. I don't think liburing will accept my code.
https://github.com/CarterLi/liburing4cpp/tree/master
The good part of this repo is that

it contains both client and server code ( ping-pong test ). You don't need other tools like rust_echo_bench
It compares various techniques. epoll, linux-aio, io_uring poll only, io_uring with/without IO_LINK, io_uring recv/send, io_uring splice, io_uring fixed_file, etc.

It's worth mentioning that the boost::asio folks have done a little bit of this too, since they can now swap io_conext's backend between epoll and io_uring:
https://github.com/chriskohlhoff/asio/issues/401#issuecomment-902613423

I'd like to use some coroutine library like what I did in the example, but not ucontext because it's slow and not available on some libc implementations.

A little late to the game, but I just pushed https://github.com/pallas/ioucontext which uses ucontext to create coroutines but then sigsetjmp/siglongjmp to move between them. I'm relatively new to io_uring, so I have no idea if ioucontext is tuned correctly or if I'm doing something egregious.
The library currently comes with two example programs: cat & a tcp/udp echo server.

@axboe would you say the benchmark you added is still up to date? Also could it be added into example/bench, so its maintained over time.

It's not uptodate, we've made many improvements and added performance related features since 3 years ago. You'd have to write a new one using multishot, ring provided buffers, and DEFER_TASKRUN (just to name a few). Going to close this issue as it's pretty out of date at this point.

@axboe I am running bunch of benchmark atm and I want to compare it with Liburing as well see how much of a different there is. Would you consider writing one? It would be really helpful, thanks.

Sorry no, I don't have time to do that. The last related thing I wrote was the proxy in examples/, which is io_uring/liburing only.

Wow, looks like you put a lots of work into proxy.*


Auto restart of compute instances utilizing local storage

ISSUE TYPE

Feature Idea

COMPONENT NAME

New feature (UI, API, etc)

CLOUDSTACK VERSION

N/A

CONFIGURATION

N/A
OS / ENVIRONMENT

N/A
SUMMARY

There should be an ability to set auto boot (restart) for VMs with local storage, i.e. similar functionality as ´virsh autostart VM´ in KVM.
When a server reboot occurs, for instance if heartbeat on the host can't write to NFS share, the server will reboot. In current versions of cloudstack, all instances using local storage on that host will need to be manually started after the reboot. There should be a functionality to mark VMs to be auto started when the host is live again.
This is particularly useful when a compute instance is using local storage (SSDs) for system disk, but NFS etc. for data storage, or when not using networked attached storage but local storage only configurations.
This is functionality similar to what is requested in #8687
STEPS TO REPRODUCE

Example of possible implementation:
- UI and API option for compute instance to auto boot on host startup (enable/disable feature per compute instance).
- Mgmt server to issue startup of compute instances when Host is ready.

EXPECTED RESULTS

Compute instances with local storage should be able able to auto start (if configured) when a host is live again after reboot.

ACTUAL RESULTS

Currently all local storage compute instances needs to be manually started after host reboot.

Would like to have this option as well in local storage :)


Add RS-485 RTS line driver control to UART driver

This PR adds an initial implementation for RS-485 half-duplex line driver control using the RTS control line.
It uses the UART[uart_num]->rs485_conf.en bit to track that this mode is enabled.

Hi @hwmaier ,
Thanks for sending this. Sorry noone got back to you earlier.
I've asked the maintainer of the IDF peripheral drivers to take a look, then we can look to merging this.
Angus

One thing wrong with this implementation - according to TM, UART2 does not have fifo reset capability, also resetting UART1 fifo might affect UART2 (and it does), this is also ignored in uart_reset_fifo function & rx_ovf interrupt handler. SW workaround would be to manually ofload RX buffer before enabling reception, something like:
while(uart_reg->status.rxfifo_cnt) 					dummy = uart_reg->fifo.rw_byte;

Sorry for the long delay. Espressif has implemented RS-485 support based on the implementation provided here, it will be merged soon.


upload file with dropzone traps focus

when using octuple file upload with dropzone, when focus gets to the select file button it gets stuck there.
The Bug is reproduced in the following sandbox instance: https://k2t32g.csb.app/

Fix is local to the button in Upload, but there is a bug that needs to be addressed in Dropzone as the button prevents default behavior internally. Will need to add a compare for tab keycode to fix the tab loop.
<Upload {...uploadProps}>
  <PrimaryButton
    htmlType={"button"}
    onKeyDown={(event) => {
      if (event.keyCode !== 9) {
        event.preventDefault();
      }
    }}
    text={"Select file"}
  />
</Upload>

Will be released in vNext


ISPN-8063 HealthCheckAPI with secure caches

https://issues.jboss.org/browse/ISPN-8063

Merged to master, thanks