WIP: OSSL_STORE 'file' loader: Try several pass phrase variants for PKCS#12

Pass phrases for PKCS#12 objects are problematic, as our conversion to
BMPString was very naïve in OpenSSL <1.1.0, and while being less so in
OpenSSL >=1.1.0, it still doesn't account for the application locale.
OSSL_STORE, on the other hand, is supposed to be fairly agnostic to
the types of objects it loads, and thereby all internal intricaties
concerning pass phrases.  Therefore, we need to create a few variants
of the given pass phrase when trying to open a PKCS#12 object, with
the hope that this will work on PKCS#12 objects generated by OpenSSL
<1.1.0 as well as OpenSSL >=1.1.0 as well as other software.
While doing this, we need to make sure the locale is set correctly.
Fortunately, that's a simple setlocale(LC_ALL, "").
Ideas largely inspired by @dwmw2

@dwmw2, if you wouldn't mind having a look at this.

Note to everyone: this is an attempt to "do the right thing", and I'm sorry this came in so late.

Looks sane to me; thanks. Wants torture test cases, of course. I have some for OpenConnect (passwords of U+0102 U+017B which I try to open ∀ LC_CTYPE, etc.) which you're welcome to lift.

We probably want to do something similar for PKCS#8 and encrypted PKCS#1 files, right? At least try the locale charset and the UTF-8 conversion, although some of the PKCS#12 complexity is not there.

I'm not going to do anything about PKCS#5 (which is what you mean when you mention PKCS#1 and PKCS#8), because that standard sees the pass phrase as a byte sequence with no inherent meaning, i.e. it can literally be any array of bytes. Limiting it to any given charset is wrong, and would potentially cause more problems than it solves.

Actually, let me take some time to think again about PKCS#5... My brain is still ruminating on that one....

IIRC PKCS#5 suggests that ASCII or UTF-8 are good character sets to use, although it lamentably falls short of even saying SHOULD. You've seen that we're working on a new I-D which addresses that.
If the native charset doesn't work then converting to UTF-8 would certainly be a useful thing to try.

@dwmw2, I'll quote PKCS#5 again:

Throughout this document, a password is considered to be an octet
string of arbitrary length whose interpretation as a text string is
unspecified. In the interest of interoperability, however, it is
recommended that applications follow some common text encoding rules.
ASCII and UTF-8 are two possibilities. (ASCII is a subset of
UTF-8.)

The question that goes through my head is if OSSL_STORE should be regarded as an application or not.  Regarding it as an application seems weird to me, considering it's very much a library API, but I'm not gonna pretend that my opinion is the absolute truth.

Let's assume for the sake of argument that OSSL_STORE is not an application — or more to the point, should not behave by default in the ways which make sense for an application.
Now, let's realise that you've just doomed all the actual applications to tweaking and working around its "non-application" behaviour in order to Do The Right Thing. And further realise that most of them won't, and will continue to do the wrong thing.
Then let's revisit the assumption at the top, and suggest that actually, OSSL_STORE should make it really easy for applications to Do The Right Thing.
Back to the question at hand.... we have a password which, pedantry aside, is fairly much known to be a string and it's probably in the local charset. We have established that it doesn't unlock the file. Why would we not want to attempt conversion to UTF-8 and try that to see if it works? Or put another way, why would we want to force applications to have to do that additional step?

I'd like argue that [as far as compatibility with 1.0.x goes] second variant is effectively redundant. Allow me to illustrate with examples.
We are on Linux. Imagine UTF-8 input method that was used with OpenSSL 1.0.x to enter Hungarian passphrase. Now you want to access the object with OpenSSL 1.1.x. If you use UTF-8 input method and call setlocale, second variant is essentially a no-op, as it would yield same output as input, which has already been exercised by 1st method(*). If you use UTF-8 input method and don't call setlocale, mbstowcs fails, variant is not exercised. If you use non-UTF-8 Hungarian input method and call setlocale, 2nd method will yield same password you would have entered with UTF-8 input method, and it's not right one, not for object created with 1.0.x. If you use non-UTF-8 Hungarian input method and don't call setlocale, mbstowcs fails, variant is not exercised.
We are still on Linux. Imagine non-UTF-8 Hungarian input method that was used with OpenSSL 1.0.x to enter Hungarian passphrase, one that can be misinterpreted if attempted as UTF-8. Now you want to access the object with OpenSSL 1.1.x. If you use UTF-8 input method, none of the three variants would actually do the job. You have no option, but to use non-UTF-8 input method. If you use non-UTF-8 Hungarian input method and call setlocale, 2nd method will yield same password you would have entered with UTF-8 input method, and it's not right one, not for object created with 1.0.x. If you use non-UTF-8 Hungarian input method and don't call setlocale, mbstowcs fails, variant is not exercised.
In other words in no combination second method does what hasn't been tried already or is right answer.
(*) To be completely accurate 1st call to PKCS12_verify_mac would actually succeed, because it does fall back to equivalent of 3rd method internally.

Now let's take Windows. In 1.0.x for Windows UTF-8 input method was not an option, so that as long as we're talking about backward compatibility for Hungarian passphrases, we should only consider non-UTF-8 Hungarian input method used to type password to 1.0.x. Now we want to access it with 1.1.x. If user sets OPENSSL_WIN32_UTF8, no password is right, irregardless whether or not application called setlocale. If user does not set OPENSSL_WIN32_UTF8 and application calls setlocale, 2nd method will yield same password you would have entered with UTF-8 input method, and it's not right one. If application doesn't call setlocale, 2nd method will yield same password as 3rd method. Even here 2nd method doesn't offer any advantage.

Do note that message prior last one had "as far as compatibility with 1.0.x goes" in opening sentence. This is because there is case when 2nd method is kind of meaningful. But[!] it has nothing to do with backward compatibility. Imagine standard-compliant PKCS#12 object protected with Hungarian passphrase. What does it take to read it with 1.1.x? UTF-8 input method would always do. The only question is what happens if user uses non-UTF-8 [Hungarian] input method. That's where 2nd method can become handy. It will translate the passphrase to UTF-8, and it will be the right passphrase. Now question is if this convenience feature is justified. Thing is that it will yield correct UTF-8 passphrase only if application called setlocale. So it all boils down to following. Do we simply recommend users to use UTF-8 input method with standard-compliant PKCS#12 objects? Or do we tell application developers call setlocale? I for one would say that former is sufficient. Overwhelming majority of Linux/Unix users are using UTF-8 input method, so they wouldn't even notice the problem. Now one can still argue that Windows user would tend to forget to set OPENSSL_WIN32_UTF8 variable. All right, it might be appropriate to make life a bit easier for them. But would I use mbstowcs? No, I'd use MultiByteToWideChar with CP_ACP. Why? Because it would be independent on whether or not application called setlocale.

We are still on Linux. Imagine non-UTF-8 Hungarian input method that was used with OpenSSL 1.0.x to enter Hungarian passphrase, one that can be misinterpreted if attempted as UTF-8. Now you want to access the object with OpenSSL 1.1.x. If you use UTF-8 input method, none of the three variants would actually do the job.

That is an uninteresting case. For backward compatibility we care about someone operating in a legacy 8-bit locale who is still operating in that same legacy 8-bit locale with an updated version of OpenSSL. But when we weren't making any attempt to deal with correct encodings, moving from one input encoding to another (e.g. taking my file created under ISO8859-1 and using it on a ISO8859-2 system, where my passphrase happens to have characters which are in both but not at the same codepoint) was never expected to work anyway. It's a red herring.

To summarize three messages above. I'd omit 2nd method on Unix and replace mcstowcs with MultiByteToWideChar with CP_ACP on Windows. This way library remains neutral to whether or not application called setlocale. You may say "but doesn't it limit user's ability?" No. User is in control of input method, so that amount of tried combinations would be sufficient. It's just that some combinations are performed by library and one by user.

Now, for your suggestion that this exercise (variant 2) is necessary or not, I'll let @dwmw2 and you fight it out.

It is clearly necessary. Sure, we can all come up with examples (like plain ASCII password) where the first attempt will work. But for a PKCS#12 file with that pathological password example in an ISO8859-2 locale, there are incontrovertibly three different ways it could have been created: OpenSSL 1.0, OpenSSL 1.1, and correct. And we need to attempt all three.

Now let's take Windows. In 1.0.x for Windows UTF-8 input method was not an option, so that as long as we're talking about backward compatibility for Hungarian passphrases, we should only consider non-UTF-8 Hungarian input method used to type password to 1.0.x. Now we want to access it with 1.1.x. If user sets OPENSSL_WIN32_UTF8, no password is right, [regardless] whether or not application called setlocale.

Yeah. Perhaps with OPENSSL_WIN32_UTF8 we should actually be calling wcstombs() instead, because we expect the input bytes to be in UTF8? Except we don't... that's only true if the default PEM  callback was used. If it came from somewhere else it probably is in the ANSI code page?

If user does not set OPENSSL_WIN32_UTF8 and application calls setlocale, 2nd method will yield sad same password you would have entered with UTF-8 input method, and it's not right one. If application doesn't call setlocale, 2nd method will yield same password as 3rd method. Even here 2nd method doesn't offer any advantage.

If the user doesn't set OPENSSL_WIN32_UTF8 then, as ever, they need to be operating in the same locale as before and then the first attempt with the actual byte sequence will work. In this case, the second and third attempts aren't needed. That doesn't mean they're never needed.

I think using MultiByteToWideChar() with CP_ACP on Windows probably does make sense.

Perhaps with OPENSSL_WIN32_UTF8 we should actually be calling wcstombs() instead

That's effectively what happens in crypto/ui/ui_openssl.c (which is also called down the line by the default PEM callback, incidently)

Perhaps with OPENSSL_WIN32_UTF8 we should actually be calling wcstombs() instead

That's effectively what happens in crypto/ui/ui_openssl.c (which is also called down the line by the default PEM callback, incidently)

Sorry, I was being sloppy. In crypto/ui/ui_openssl.c it's converting to UTF-8. I meant converting to CP_ACP. I'll try restating that...
Yeah. With OPENSSL_WIN32_UTF8 we actually get UTF-8 from the PEM callback. Which means one of our attempts should actually be converting that to CP_ACP (i.e. manually to UTF16 and then MultiByteToWideChar(CP_ACP) for compatibility with older / non-OPENSSL_WIN32_UTF8 OpenSSL? Of course, that's true if the default PEM callback is used but the input bytes might still be in the ANSI codepage.
And note that without OPENSSL_WIN32_UTF8 you have extra fun on Windows because the password might be in CP_ACP and it might be in CP_OEMCP depending on where it came from. Arguably if it came from the console without OPENSSL_WIN32_UTF8 it would have been in CP_OEMCP? So that's what we would try converting to?

Yeah. With OPENSSL_WIN32_UTF8 we actually get UTF-8 from the PEM callback. Which means one of our attempts should actually be converting that to CP_ACP (i.e. manually to UTF16 and then MultiByteToWideChar(CP_ACP) for compatibility with older / non-OPENSSL_WIN32_UTF8 OpenSSL? Of course, that's true if the default PEM callback is used but the input bytes might still be in the ANSI codepage if they came from a file.

I'm sorry, you lost me...  are we still talking about the three variants tried with a PKCS#12 object, or are you on a different subject (you did mention doing something similar for PKCS#5 protected files...)?

It [2nd variant] is clearly necessary.

In the messages above it was discussed that 2nd variant [as it stands!] doesn't give any advantage [in backward compatibility cases]. For any given combination it yields either wrong phrase or one that was already tried or is going to be tried next. Is any of the presented outcomes wrong? [There might be, jsut tell which one is.] Is any combination missing? However, I make difference between 2nd variant as suggested implementation, and 2nd variant as option to try. What I'm talking about is that 2nd variant as suggested implementation is redundant. 2nd variant as option to try is something user effectively does by choosing between UTF-8 and non-UTF-8 input methods. And point here is that if we were to implement 2nd variant for real, we would have to do things we don't want to do in libcrypto. Like calling iconv, figuring out if application called setlocale or not, and if it was with "" or "some-non-empty-string", and in such case which one...

That [non-UTF-8 Hungarian input method that was used with OpenSSL 1.0.x to enter Hungarian passphrase attempted with UTF-8 locale] is an uninteresting case.

It was considered for completeness.

I don't think that, as implemented, the second variant is redundant.
Let's look at the case of ISO8859-2 locale, password of "ĂŻ", U+0102 U+017B, which in ISO8859-2 is C3 AF
First variant tries C3 AF exactly as it was presented. OpenSSL 1.1 treats that as UTF-8 and interprets it as U+00EF "ï", and a corresponding BMPString of 00EF 0000 for key derivation. This works for files created with OpenSSL 1.1 in this locale.
Second option converts to wchar_t 0102 017B and to the correct UTF-8 representation C4 82 C5 BB. The resulting BMPString 0102 017B 0000 matches correctly generated PKCS#12 files.
Third option "casts" the input bytes to ISO8859-1, where C3 AF would mean "Ã¯" (U+00C3 U+00AF), and then converts to UTF-8 giving the bytes C3 83 C2 AF. OpenSSL creates a BMPString 00C3 00AF 0000 for deriving the key, which matches files created with OpenSSL 1.0.
Each of those might match a given PKCS#12 file, depending on how that file was created.

I don't think that, as implemented, the second variant is redundant.

Do note that I argue that it's redundant from backward compatibility viewpoint. I also wrote that it does make some sense in cases when you use non-UTF-8 input method and want to access standard-compliant PKCS#12 object. However! In this last case it boils down to question whether or not application[!] called setlocale(LC_ALL,""). And so we are getting to already posed question. Do we tell users to stick with UTF-8 input method when they want to access standard-compliant PKCS#12 object? Or do we tell application developers to call setlocale(LC_ALL,"")? Are we even in position to do the latter, and be that specific, i.e. insists on ""? We are not. So the only reliable option for us is to say "use UTF-8 input method with standard-compliant PKCS#12 objects, and original method with objects created by 1.0.x." Once again, 2nd method is effectively in user's hands. And as long as we don't want to get involved with iconv[-like] activities, it has to be user's hands.

What if we try to phrase the requirement in a way that doesn't make normative reference to setlocale() at all. For example:
"On POSIX systems, the passphrase provide to OSSL_STORE routines shall be in the character encoding corresponding to the LC_CTYPE category of the current locale."
You can then add a reminder that they might want to call setlocale().
But that's fine. Without setlocale() the legacy compatibility should still work. You need to use setlocale() to interact with correctly generated files. But variants 1 and 3 which interoperate with OpenSSL 1.0 and 1.1 respectively don't need LC_CTYPE.

I don't think that, as implemented, the second variant is redundant.

I'd like to add following. I'm not categorically against suggested implementation of the 2nd method. I'm against the dependency on application making the setlocale call. Richard says "it doesn't harm [if it's attempted]." Well, on utilitarian level, no, it doesn't harm. But it does create an undesired illusion and potential confusion. What illusion/confusion? That non-UTF-8 method sometimes works to access standard-compliant PKCS#12 objects and sometimes doesn't. It's so much more appropriate to give just one message. In sense that when user says "it doesn't work" question should be "do you use UTF-8 method with standard-compliant object, yes/no? Or "do you use original method with legacy objects, yes/no?" It shouldn't be "it all depends, you know, does your application call setlocale(LC_ALL,"")?"

It shouldn't be "it all depends, you know, does your application call setlocale(LC_ALL,"")?"

It isn't. It's _"does your application provide the passphrase in the character encoding corresponding to LC_CTYPE as it should?".

You need to use setlocale() to interact with correctly generated files.

Not if you use UTF-8 input method.

There are two sane options for any API which takes a string, in C. You either take UTF-8, or you use LC_CTYPE.
Using LC_CTYPE is the simplest thing to do for applications. It means that naïve application authors (i.e. most of them) will accidentally Do The Right Thing.
Using UTF-8 is also a viable option, but I think it's suboptimal in this case. Sure, you can define the OSSL_STORE API as UTF-8-only. And you can make it error out if it's ever given a byte sequence which isn't valid UTF-8, to help people remember.
 And you also have the problem of converting back from UTF-8 to LC_CTYPE for the backward compatibility, so it isn't even any easier.

So hey, you got me, this PR is very much a convenience for applications. Think of the cycle, where the application does OSSL_STORE_open, followed by a loop around OSSL_STORE_load and then a OSSL_STORE_close to finish up.
The pass phrase UI is passed to OSSL_STORE_open, so for the application to try several pass phrase variants requires starting the cycle over for each of them, and the application needs to know what kind of object is getting opened... it gets no such information beforehand, which makes it more difficult. Note that this is by design, the loaders are supposed to to know what they deal with and handle it appropriately, while the API is supposed to be fairly agnostic to what happens below its surface.
@dot-asm, you're talking about the user trying this and that, but then you seem to think in terms of small command line programs. What about larger systems with a long running time? Leaving it to the user would effectively mean leaving it to the application to do the guesswork, and that is much harder than asking for a simple setlocale(LC_ALL, "") (or whatever else they seem appropriate, 'cause yes, I do think that we should be able trust the application to get the input method and locale right, if they care)

@dot-asm, you're talking about the user trying this and that, but then you seem to think in terms of small command line programs. What about larger systems with a long running time?

If it doesn't work, then user would have to figure out what does prior letting it run for long time. If there is irreconcilable mismatch between two objects, then user should be advised to re-encrypt non-compliant object to standard form. It's right thing to do in long run, and sooner it's done is better... That last thing by the way is another danger with creating the said illusion/confusion. It doesn't encourage users to do the right thing.

Leaving it to the user would effectively mean leaving it to the application to do the guesswork,

As already said, as long as we don't want to engage in iconv[-like] activities (and we really don't), ultimately it will always be up to application in either case.

and that is much harder than asking for a simple setlocale(LC_ALL, "")

I'd like to hear more opinions on this. Namely, what's more appropriate? To insist that all applications call setlocale(LC_ALL,"")? Or to give consistent message "use UTF-8 input method with standard-compliant PKCS#12 objects [and original one with legacy]". Well, it should be noted that they are not actually mutually exclusive. I mean one can tell "use UTF-8 method" and "call setlocale(LC_ALL,"")". So it's rather do we actually want to be dependent on application code calling setlocale [when we don't actually have to]?

So here's another (philosophical) thing to think about; we're all talking about "users", but who are we talking about? I would say that re libcrypto, our users are the application authors first and foremost. It seems, though, that we're skipping over those and talking about their users...

The way I see it, is that we should provide a library which makes it hard for applications to do the wrong thing for their users.
Which might explain why we seem to be "skipping over" the applications themselves.
Seriously, I have hundreds if not thousands of lines of code in OpenConnect just to attempt to make it Do The Right Thing with a client certificate, wherever that cert came from and whatever form it is in. I object to all but one line of that, where I pass the strings I got from the command line (be they filenames, PKCS#11 URIs, or passphrases) into the library.

I agree that as long as we stay in modern enough Unix, telling people to go UTF-8 is the right move, because most modern shells with editing capabilities support it.  However, we still have active shells where UTF-8 is problematic.  Windows' CMD is one of them (and I wouldn't be surprised if that's a reason you can't choose code page 65001, or that choosing it leads to unexpected problems).  OpenVMS' DCL is another.  So there are classes of people that won't be entirely happy with the message "move to UTF-8", 'cause of the problems that will lead to.
Now granted, for Windows, we have the OPENSSL_WIN32_UTF8 environment variable hack.  I've consciously chosen not to introduce the same hack on VMS, 'cause 1) there is a UTF-8 locale (which only affects programs that call setlocale(LC_CTYPE, "") or setlocale(LC_CTYPE, "UTF8-20")), and 2) 'cause it would cause a mbstowcs call anyway, which would make the exercise a moot point.
Are there other platforms/shells with the same kind of problem?  Do we know?

we should provide a library which makes it hard for applications to do the wrong thing for their users.

This would have to be translated to a) requirement for application to call setlocale; b) skipping the 1st variant. Indeed, library that doesn't permit to do wrong thing would have to assume that input is in correct LC_CTYPE encoding that has to be converted to UTF-8. And conversion should succeed. Everything else should be an error. I'm naturally talking about primary case, handling standard-compliant PKCS#12 objects. Then of course one would still have to do something about handling legacy objects. This formally calls for following question. With this in mind shouldn't 2nd variant be 1st? It actually should. And right there you'll see my point about undesired dependency on application calling setlocale. I mean you'll see the contradiction that kind of illuminates my point. Contradiction between order that was considered right so far, 1st then 2nd, and order that you would have to choose if you take it to the logical end, 2nd then 1st (if not straight to 3rd).
[BTW, just in case, attempt to follow this would effectively fall apart on Windows. Because of impossibility to use CP_UTF8 as system code page/locale, and because setlocale is "skewed" toward ANSI. Well, it's "skewed" toward system code page that is ANSI. Again, presumably modulo Asian Windows.]

This would have to be translated to a) requirement for application to call setlocale; b) skipping the 1st variant.

But that wouldn't open legacy files. That's not the right thing to do for users. I didn't say that we should force applications to be pedantically correct and refuse to use "invalid" files such as those produced by current OpenSSL. I said that applications should do the "Right Thing", whatever that is... but mostly that was in response to @levitte's philosophical question about whether we're thinking of application authors, or their users. Let's not get too distracted by it.

With this in mind shouldn't 2nd variant be 1st? It actually should.

This is purely an optimisation question, surely? I genuinely don't care what order you try them in as long as you try them all. The common case is that the password will be ASCII anyway, so all variants do the same thing and whichever you try first will succeed. If that wasn't true, we wouldn't have survived so many years without actually trying to get this Right™.

This would have to be translated to a) requirement for application to call setlocale; b) skipping the 1st variant.

But that wouldn't open legacy files.

Come on! I did write "Then of course one would still have to do something about handling legacy objects." Point is that following this line of reasoning ought to mean that one should try what is supposed to be "the right thing" fist, and that this contradicts the suggested order. [Keep in mind that I'm trying to argue against dependency on setlocale.]

With this in mind shouldn't 2nd variant be 1st? It actually should.

This is purely an optimisation question, surely?

No, not at all. This is question of what is the Right order, not optimal.

modern enough Unix, telling people to go UTF-8 is the right move, because most modern shells with editing capabilities support it.

Is it actually about shell? I mean when you type passphrase at ui_openssl prompt, shell is not involved in any way, it's really between application and terminal [emulator]. So it's not about sh, but xterm, not about Windows CMD, but Console Window, and it shouldn't be about VMS DCL either... Also note that if you login remotely, it's about terminal emulator on computer you logged in from...

You can pass pass phrases on the command line.  Not a good thing on Unix, quite innocuous on VMS.
Re order, I didn't think too much about it...  I have zero problems reordering things if that's desirable for signaling The Right Thing™.  So if we all agree that variant 1 and 2 should trade places with that in mind, I'll do it.

Frankly, though, I don't quite understand why there's a need to be so darn categorical about The Right Thing™.  My aim is to support UTF-8 input (which is variant 1) as well as input according to non-UTF-8 LC_CTYPE (which is variant 2), as well as non-compliant OpenSSL PKCS#12 objects (variant 1 and 3), as far as reasonably possible.  But the priority between supporting UTF-8 input and input according to non-UTF-8 LC_CTYPE is utterly unimportant to me...  I seriously couldn't care less

Frankly, though, I don't quite understand why there's a need to be so darn categorical about The Right Thing™.

??? Because that's what differs us from animals?

Frankly, though, I don't quite understand why there's a need to be so darn categorical about The Right Thing™.

But on serious note, programming is also about formal logic and it's only appropriate to do what makes most sense given the circumstances. I'd like to remind that I argue for omission of mbstowcs call(*) [but not necessarily MultiByteToWideChar(**)]. As for order, I'm only saying that if we formulate the circumstances the way David suggests, then it would be only logical to exercise 2nd variant 1st.
(*) Because it's dependency on whether or not application called setlocale.
(**) Because there is no dependency on what application did, and as convenience feature in cases when user forgot to set OPENSSL_WIN32_UTF8.

As for order, I'm only saying that if we formulate the circumstances the way David suggests, then it would be only logical to exercise 2nd variant 1st.

If we want to be picky, I'd suggest in parallell rather than ordered...  but we program in C, so that's an exercise in futility, but perhaps that view helps explain why I don't see the order between the two as terribly important...  although if you want to signal a preference for UTF-8 over LC_CTYPE, then the current order is the more logical one.
What I hear, though, is that the battle here is first of all about calling mbstowcs or not.  None of us have moved an inch on our opinions, and considering no one else has said a peep, we seem to be stuck.  How do we move forward?

What I hear, though, is that the battle here is first of all about calling mbstowcs or not. None of us have moved an inch on our opinions, and considering no one else has said a peep, we seem to be stuck. How do we move forward?

Do we actually want to fix this or not? Do we want to make a best effort to open files, whether they are correctly generated or whether they are "real world" examples from older (or current) OpenSSL?
We don't actually have a difference of opinion on that, I hope?
Fundamentally we cannot fix it without doing a conversion either to or from the current locale.
Either we take strings in the locale encoding and we convert to UTF-8 for the "correct" case, or we take strings in UTF-8 and we convert to the locale encoding for the "legacy" (OpenSSL <= 1.0) case.
Do we want to care about backward compatibility for naïve applications which never attempt to get anything right w.r.t. locales? I would suggest that we do. In which case we have to take the locale encoding. Sure, if the app doesn't set LC_CTYPE then it'll never be able to open "correct" files. That's OK; it never could before either.

Richard Levitte wrote:

Pass phrases for PKCS#12 objects are problematic, as our conversion to
BMPString was very naïve in OpenSSL <1.1.0, and while being less so in
OpenSSL >=1.1.0, it still doesn't account for the application locale.
[SNIP]

[1] I'm not convinced that a library should use "application" locale.
For POSIX like this mean library to depend if setlocale() is called or
not. More over application may set locate different than environment
(please do not comment correctness of those apps ;) ) .
[2] library could use locale aware functions without to depend from [1].
Unfortunately functions like mbstowcs_l() are not widely implemented.
Regards,
Roumen

Either we take strings in the locale encoding and we convert to UTF-8 for the "correct" case, or we take strings in UTF-8 and we convert to the locale encoding for the "legacy" (OpenSSL <= 1.0) case.

Correction: We need the locale encoding to reliably read files from all versions of OpenSSL, not just <=1.0.

Interesting...  it took me a few moments to wrap my head around that other option, but yeah, I see what you mean.

I added a bit of documentation that should clarify things, including the optional (at the option of the application writer) support of PKCS#12 pass phrases encoded with something other than UTF-8 or ISO-8859-1.

It is very subtle that there is not a comma after "as well as objects produced by other programs". (There's only a stray "with")
Can we make it clearer that the "ISO8859-1 or UTF-8" condition only applies to object created by non-OpenSSL?
I know the situation and had to read it three times in order to find an interpretation that was true.

It is very subtle that there is not a comma after "as well as objects produced by other programs". (There's only a stray "with")

:-)  (oh, will fix)

Can we make it clearer that the "ISO8859-1 or UTF-8" condition only applies to object created by non-OpenSSL?

Please suggest something, whatever I've tried becomes clumsy...

Turns out that stray 'with' didn't want removing, only moving a couple of words earlier. Now it says "objects produced other programs".... Sorry :)

Please suggest something, whatever I've tried becomes clumsier...

Yeah, it gets clumsy really quickly. Let me have a quick try...

Where pass phrases for PKCS#12 objects contain non-ASCII characters, careful consideration is required. The file scheme loader will attempt to decrypt the object by interpreting the provided passphrase in three encodings. Each of these three encodings may be required in order to decrypt files generated with different versions of OpenSSL or other software:

UTF-8.
ISO8859-1.
The character encoding corresponding to LC_CTYPE on a POSIX system, or CP_ACP on Windows.

On POSIX systems, setting LC_CTYPE correctly requires explicit action from the application. The most usual method is to call setlocale() as follows as part of program startup:
      setlocale(LC_CTYPE, "");

When operating in a locale which uses neither UTF-8 not ISO8859-1 (each of which is attempted anyway), setting LC_CTYPE correctly may be necessary in order to decrypt PKCS#12 objects created by compliant software.

Ok, I took that text and adapted it.

We should probably make it clear that we do expect the UI_METHOD to provide a string encoded in the native (LC_CTYPE/CP_ACP) encoding. Yes, we interpret it in three ways to open three kinds of file... but those only work if it's in the native encoding to start with. If the application author reads this and says "aha, they try UTF-8 so I'll always pass in UTF-8 and then I'm good", then that's not going to work.

Perhaps something along the lines of...

The above conversions are designed to achieve compatibility with current and older versions of OpenSSL when the passphrase provided by the UI_METHOD is in the native (i.e. LC_CTYPE/CP_ACP) encoding. If the UI_METHOD returns a passphrase encoded in UTF-8 where the native encoding is not UTF-8 (including when OpenSSL is built with OPENSSL_WIN32_UTF8 on Windows), that will suffice to decrypt correctly-generated PKCS#11 objects but may not work for objects created by OpenSSL.

And we should still try converting to UTF-8 for PKCS#5.

including when OpenSSL is built with OPENSSL_WIN32_UTF8 on Windows

OPENSSL_WIN32_UTF8 is not a build parameter, but environment variable name you set at run time. In other words "Windows openssl.exe invoked with OPENSSL_WIN32_UTF8 environment variable set."

OPENSSL_WIN32_UTF8 is not a build parameter, but environment variable name you set at run time. In other words "Windows openssl.exe invoked with OPENSSL_WIN32_UTF8 environment variable set."

Ah yes, thanks. Although in this case not openssl.exe necessarily, but the application using the STORE API. I suppose one might argue that if OPENSSL_WIN32_UTF8 is set, the file loader should actually assume that its input is UTF-8 and convert to CP_ACP in its attempt to be compatible with older OpenSSL.

At some point it was suggested that goal is to make it hard for application to do wrong. But it's equivalent to making it do right. Which means refuse to do things that are not right. While suggestion effectively is to guess what application "meant." This kind of goes against proclaimed goal. It's not making hard for application to do wrong, as it tolerates application doing wrong. Unfortunately (as already said) actually making application to do right is equivalent to demanding that it calls setlocale. And (as already said) this opens for question if we are in position to demand that it does (and expect that it does it right). I argue that answer is (unfortunately) no.
Note that above is not argument for doing nothing, only for not making assumption about setlocale.
How about following. Modify apps/pkcs12.c to try variants in reverse order and issue diagnostic to user suggesting what to do to "normalize" the object so that it's guaranteed to work with libcrypto. So that if user finds PKCS#12 failing to load, problem can be at least diagnosed and hopefully resolved on the spot [without debugging custom application].

At some point it was suggested that goal is to make it hard for application to do wrong. But it's equivalent to making it do right. Which means refuse to do things that are not right. While suggestion effectively is to guess what application "meant." This kind of goes against proclaimed goal. It's not making hard for application to do wrong, as it tolerates application doing wrong.

The repeated attempts to open the file here are not attempting to guess what the application has done. These are attempting to guess what the tool making the PKCS#12 file has done, which is different — and outside the control of the application.

Unfortunately (as already said) actually making application to do right is equivalent to demanding that it calls setlocale. And (as already said) this opens for question if we are in position to demand that it does (and expect that it does it right). I argue that answer is (unfortunately) no.

That's OK. It'll still open PKCS#12 files generated with OpenSSL, which doesn't get it right either. Documenting "if you want to open correct files then you have to call setlocale()" should be fine.

Note that above is not argument for doing nothing, only for not making assumption about setlocale.

We're not assuming anything. We are documenting it as a requirement if you actually want your application to behave correctly.

How about following. Modify apps/pkcs12.c to try variants in reverse order and issue diagnostic to user suggesting what to do to "normalize" the object so that it's guaranteed to work with libcrypto. So that if user finds PKCS#12 failing to load, problem can be at least diagnosed and hopefully resolved on the spot [without debugging custom application].

Hm, you mean to try the LC_CTYPE→UTF-8 conversion first and bitch "recreate your broken PKCS#12 file" if that doesn't work and one of the other two (treating it as UTF-8 or ISO8859-1) does work? OK, I can live with that if you can. You understand you'll be instructing them to create the file with something other than OpenSSL (even 1.1), yes?

you'll be instructing them to create the file with something other than OpenSSL (even 1.1), yes?

??? No!

you'll be instructing them to create the file with something other than OpenSSL (even 1.1), yes?

??? No!

Um... how do you not realise that? :)

Again, ISO8859-2 locale, password of U+0102 U+017B "ĂŻ". You can't use openssl on the command line to create a valid PKCS#12 file.

You understand you'll be instructing them to create the file with something other than OpenSSL (even 1.1), yes?

Actually, I think @dot-asm wants to say "please create with an input method that produces UTF-8".  In that case, OpenSSL >=1.1.0 does get it right.  This has been @dot-asm's message from the start.

Actually, I think @dot-asm wants to say "please create with an input method that produces UTF-8". In that case, OpenSSL >=1.1.0 does get it right. This has been @dot-asm's message from the start.

Sure, but it needs to be explained in ways that a user sitting at an ISO8859-2 terminal can implement, preferably without requiring a C compiler on the system. Probably something along the lines of echo $PASSWORD | iconv -t UTF-8 | openssl pkcs12 -passin stdin …

Or, you know, you could actually fix the bug :)

The other option for the user is to switch to program versions that support UTF-8 natively.  On the Unix end of things, that's not excessively hard (hell, even good old xterm seems to support it these days).  So at some point, the locale dependent thing will not actually be needed any more...  I don't believe we're quite there yet, though, which is the reason I include locale dependent support here.

So yeah, if openssl pkcs12 is going to do something to detect non-compliant pass phrase encodings, I think it should suggest to the user to switch to a UTF-8 input method.

Taking this out of WIP.

Looking good. I think we just need to indicate that the passphrase input should be in the locale charset (to offset confusion about the ways in which we interpret it).

To clarify. I don't  view it unreasonable to say "pass UTF-8 to libcrypto". It's not unreasonable to put burden of conversion to UTF-8 on application (or burden of choice of UTF-8 input method on user). It's really not bigger deal on contemporary Unices, in overwhelming majority of cases it's actually "do nothing special." As for "ĂŻ" with ISO8859-2 input method. It's correct that as it stands now, 1.1.0 will fail to produce compliant PKCS#12 with non-UTF-8 input method. It's just that so far(*) assumption was that user will use UTF-8 input method. And majority de-facto does and has been doing it for a while by now. So that chances are that UTF-8 method was used to enter "ĂŻ" already to 1.0.x(**). But if we're to aim to support even non-UTF-8 input method, that is not impossible to resolve in openssl command line tool. Note that I'm arguing against dependency on setlocale in librypto, but it doesn't mean that it would be inappropriate to perform conversion in apps/pkcs12.c. One can probably even argue that it would be reasonable to better support non-UTF-8 input methods, at least to facilitate "normalization" of legacy PKCS#12 object. I mean for far in order to "normalize" object created with 1.1.0 with non-UTF-8 input method user would have to a) choose original method and change password to ASCII-only pass phrase, then b) switch to UTF-8 method and change back to original pass phase. One can argue that it should be possible to do this in one step.
(*) Or rather upon 1.1.0 release.
(**) Yes, it means that it was reckoned that this would be more common backward compatibility problem than password in question entered with non-UTF-8 input method. I can't recall any problem reports about accessing legacy PKCS#12 files, so... Well, this can mean two things: it was actually adequate call, or people suffer in silence. But it's probably unreasonable to assume that literally everybody would suffer in silence...

So let's see if I can summarize, if an application wants to support standard compliant PKCS#12 objects and non-UTF-8 input method when loading through OSSL_STORE, there's the choice between:

this PR and have the application call setlocale(LC_CTYPE, "")
creating a UI_METHOD in the application that converts all its input to UTF-8 encoding (which, incidently, requires setlocale(LC_CTYPE, "")), and have the user convert any object that has been created using a non-compliant pass phrase...

In terms of burden on application writers and on users that have to handle non-compliant (in terms of pass phrase conversion into BMPString) PKCS#12 objects, I don't agree with you about what's reasonable.
Regarding (**), I would believe that the vast majority type ASCII only in their pass phrases...  which might be their way, incidently, of "suffering in silence".

Side note: you're right to point out apps/pkcs12.c, 'cause it should ensure that the pass phrase that's fed to PKCS12_create and PKCS12_set_mac is UTF-8 (or well, I personally think that PKCS12_set_mac should do that work, but since @dot-asm obviously won't approve anything in libcrypto that relies on locale being set, there ya go...)

Regarding (**), I would believe that the vast majority type ASCII only in their pass phrases...

I would too. My remark was about those who actually typed non-ASCII characters. I mean "everybody" was not "every OpenSSL user", but "everybody who used non-ASCII characters in passphrase." Whole request is about them, and naturally the rest of this remark.

there's the choice between:

this PR and have the application call setlocale(LC_CTYPE, "")
creating a UI_METHOD in the application that converts all its input to UTF-8 encoding (which, incidently, requires setlocale(LC_CTYPE, "")), and have the user convert any object that has been created using a non-compliant pass phrase...

As for 2, part about having the user convert. It is desired direction, users should be encouraged to do that. As for rest of 2. It would be required only if application insists on using non-UTF-8 input method with standard-compliant PKCS#12 object. Question is do we have to meet that demand? Application insists -> application fixes. Note that UTF-8 input methods are common place and customarily is matter of end user's choice. And in unlike event UTF-8 input method is not an option, well, such system probably doesn't have mbstowcs either, and application would have to come up with some custom solution anyway. Or simply adhere to non-standard PKCS#12... Yes, you can sense contradiction. On one hand I'm saying that users should "normalize" and on the other hand suggest that there might be situations when "non-normalized" could be an option. Well, life is not fair... But note that not making any assumptions about locale in libcrypto aligns better with situation when you're stuck with non-UTF-8 input method. In sense that it would be more appropriate to interfere as little as possible with what application would have to do to pull it off.

@dot-asm obviously won't approve anything in libcrypto that relies on locale being set, there ya go...)

But it doesn't mean that somebody else couldn't. I merely express my opinion. It's just that me not approving it is an [unfortunate] expression of the opinion, but what can I do :-( It's not like I don't appreciate the effort (as I do!) or sentiment...

And in unlike event UTF-8 input method is not an option, well, such system probably doesn't have mbstowcs either, and application would have to come up with some custom solution anyway.

In this, you are wrong.  VMS terminal drivers do not generate UTF-8, but there is locale support in DCL (you can set a locale, and you can compile a locale, but DCL itself isn't wide char aware, so editing command lines that happens to have UTF-8 characters is quirky), and the C RTL is fully C95 compliant in this regard, and is perfectly possible to use mbstowcs.
Basically, what locale and wide char support there is on VMS, it currently only applies to programs written in C.  You can imagine that the lack of support in the native tools that run in a terminal session (DCL itself, various native editors such as EVE, .....) makes a UTF-8 input method not entirely desirable (as soon as there is one multibyte character on a line, the position of the cursor isn't trustable).

I've taken this to<EMAIL_ADDRESS>(@dwmw2, are you listening to it?)

And in unlike event UTF-8 input method is not an option, well, such system probably doesn't have mbstowcs either,

In this, you are wrong.

All right. I'm utterly surprised that they actually don't support UTF-8 as input option while providing UTF-8 locale[s]. By this they kind of say "application is on its own when it comes to UTF-8, we just let it make something out if it, it's not our busyness." And so question would be why should our message be different? One can also put it as following. If target platform doesn't support UTF-8 as input option, would it be unreasonable for us to disclaim support for UTF-8 in general and standard-compliant PKCS#12 in particular? Once again, all this is about users who actually type non-ASCII characters in pass phrases.

At one time, there were plenty of programs on Unix that didn't provide a UTF-8 input method, while UTF-8 locales did exist.  xterm was one such program (this has now changed, but it took a while before that happened).  VMS' native programs haven't received the same love and care for quite a few years, and VSI are currently working their collective asses off to catch up on quite a number of topics (I do hope this is one of them).

VSI are currently working their collective asses off to catch up on quite a number of topics (I do hope this [UTF-8 input method] is one of them).

So it just will have to start working when they catch up... Once again, is it actually unreasonable for us to disclaim support if vendor doesn't provide UTF-8 input method? Note that it doesn't prevent application from solving the problem...

@dot-asm I'm getting a bit lost. Can you be specific about what you're suggesting?
The STORE API should be defined to take filenames etc. in the locale-encoding like all normal APIs do except that the passphrase (and the passphrase alone?) shall be in UTF-8?
What does the default PEM callback do, in order to meet that requirement? Or do we stop providing a default? Or do we provide a default which silently does the wrong thing?

Errr, the STORE API takes URIs, and the exact syntax of a URI is well defined (percent encoding for anything non-ASCII), so I don't quite see how your argument holds there, @dwmw2 (but I should look over loader_file.c with regards to percent encoding).
Otherwise, @dot-asm's message seems to be clear, the application should be made to provide UTF-8 encoded pass phrases, period, and libcrypto shouldn't care how that happens, and the user should convert any object that has used a differently encoded pass phrase.
(note that in a way, this complies with your desire for UTF-8 uniformity, just not quite the way Postel stipulated)

Errr, the STORE API takes URIs, and the exact syntax of a URI is well defined (percent encoding for anything non-ASCII), so I don't quite see how your argument holds there, @dwmw2 (but I should look over loader_file.c with regards to percent encoding).

Yeah, let's make test cases for that. Create a PKCS#12 object in a file called ♥.p12 and access it via a URI on Windows and other platforms.

Individuals can post; it’s moderated.

@dot-asm I'm getting a bit lost. Can you be specific about what you're suggesting?

the application should be made to provide UTF-8 encoded pass phrases, period

Right. Modulo fact that libcrypto would still have to tolerate and application occasionally pass non-UTF-8 input for backward compatibility with 1.0.x PKCS#12. Unfortunately :-( But IMHO libcrypto shouldn't "help" application by performing operations that are dependent on call to setlocale, i.e. calling mbstowcs, for accessing standard-compliant objects. One can put it like this. Demand UTF-8 input, but tolerate non-UTF-8 for backward compatibility, not forward.

The trouble is that the application cannot be expected to know what kind of object the UI_METHOD is asked to provide a pass phrase for.
So either we tell application writers that they should loop around the whole loading cycle (open, load, close) with one pass phrase variant for each iteration, or we must accept one well specified variant.
I have also fantasized around making a UI retriable, and thereby have the UI_METHOD serve more than one pass phrase variant. Haven't quite figured out the mechanics for that yet. That would be a fairly practical way that still puts the problem of variants (a problem that we have created, I might add) into the application's lap. I don't think that's very fair (because we created the problem in the first place), but if our app can constitute a good source to copy from, it's still a somewhat viable option.
Yet another way to deal with this is to add a wchar_t only UI API and use and only that from OSSL_STORE. Of course, we can then forget about non-compliant objects from openssl 1.0.x then.

We cannot lock ourselves in an ivory tower and pretend not to be affected by the locale character set. Every filename we pass to open(), every string we receive on the command line, every I/O from the console and most text files, will be in that character set. And most applications will use that in every char * API unless explicitly directed otherwise (and in fact even then, unless we enforce otherwise).
We have three choices:
• Continue to bury our heads in the sand and get things wrong (perhaps with the slight improvement of documenting that we get it wrong in non-UTF8 locales including Windows).
• Document that applications ought to call setlocale() if they want things to work right, and do appropriate conversions when we need a specific encoding (as we do for PKCS#12).
• Require UTF-8.
The problem with the last option is that we still have to do conversions back to the locale encoding, if we want to attempt to load PKCS#12 files created with OpenSSL <=1.0 (or even 1.1 in some cases). And choosing that because we "don't want to require setlocale()" is kind of nonsensical because how in $DEITY's name do we think they're going to manage to do that conversion if they haven't set the locale appropriately? And we'd need the locale set anyway if we want to load PKCS#12 files created with OpenSSL.
I understand the desire to keep life simple for ourselves and not properly address this rather complicated issue. I think it's entirely the wrong choice though, and is very detrimental to our users.

The project consensus seems to go toward requiring UTF-8 pass phrase encoding for the STORE API, and that any file that can't be loaded with that encoding has to be converted first.  For PKCS#12, this has the benefit to push for standard compliant objects.  It also means that the job of making sure there is a correctly encoded pass phrase depends entirely on the application, and only takes one attempt.
What's needed in this case is a simple conversion tool, designed to handle three object types:

legacy encrypted PEM
PKCS#8
PKCS#12

Did I forget anything?

There seems to be an assumption here that wchar_t represents UTF-32 on most UNIX systems, like it (generally) does on Linux.  It does not.  On AIX, HP-UX, and Solaris, wchar_t is a locale-specific encoding.  I just don't want someone to come along and see this thread, and think that using mbstowcs will convert to UTF-32 on other UNIX systems, because it won't.  I really don't think the code in this PR does what the author intended on most UNIX systems.  And really, the majority of the incorrect encodings for passphrases are coming from Windows, which would be expecting UTF-16 encoded passphrases, so even on Linux, the problem is not addressed appropriately.
This first link is a little out of date, as several of the systems have more locales than listed, and have added more API support.  The second link includes clear information regarding wchar_t on Solaris.
https://www.tldp.org/HOWTO/Unicode-HOWTO-6.html
https://docs.oracle.com/cd/E36784_01/html/E39536/gmwkm.html

Richard Levitte wrote:

The trouble is that the application cannot be expected to know what kind of object the UI_METHOD is asked to provide a pass phrase for.

This is true. More over application could launch another application
that prompt for 'password'.

So either we tell application writers that they should loop around the whole loading cycle (open, load, close) with one pass phrase variant for each iteration, or we must accept one well specified variant.

As application writer I don't expect to perform some conversions.
PKIX-SSH supports OpenSSL "Store-API" to load PKCS#12 files. It uses own
UI method that launch ASKPASS program.
I don't know how many such programs exists in the world.
The common if that ASKPASS program runs in X-Window locale not in locale
of ssh command.
Remark: usually both match but in some cases are different (connection
to old systems).

I have also fantasized around making a UI retriable, and thereby have the UI_METHOD serve more than one pass phrase variant. Haven't quite figured out the mechanics for that yet. That would be a fairly practical way that still puts the problem of variants (a problem that we have created, I might add) into the application's lap. I don't think that's very fair (because we created the problem in the first place), but if our app can constitute a good source to copy from, it's still a somewhat viable option.

I'm not sure that issue is with openssl implementation. This is normal
evolution : first 7-bit standard,  then 8-bit locales, then UNICODE locales.

Yet another way to deal with this is to add a wchar_t only UI API and use and only that from OSSL_STORE. Of course, we can then forget about non-compliant objects from openssl 1.0.x then.

I thin that current documentation about pass management is enough - I
mean PR for password callback.
On modern OS-es (POSIX 2k8 compatible) OpenSSL library always could
check user locale (code snippet):
locale_t loc;
loc = newlocale(LC_CTYPE_MASK, "", (locale_t) 0);
if (loc == (locale_t) 0) {
perror("newlocale fail");
return 1;
}
printf("CODESET=%s\n", nl_langinfo_l(CODESET, loc));
freelocale(loc);
Above code bay-pass application settings (setlocale) and returns value
according application environment.
In X-terminals password (or user input) is more closed to environment of
X-window  then application locale.
Perhaps based on CODESET library could reject some inputs?
What about if codeset is not ANSI_X3.4-1968,  UTF-8 or ISO-8859-1 to
raise error when UI is called to get password for PKCS#12 or etc files ?
Remark: UI method could be used for other prompts,some  confirmations
for instance.
Regards,
Roumen

So after some discussion on openssl-project, it's been concluded that the path forward is to reinforce that the PKCS12 functions really should be given UTF-8 encoded strings, period, and that the application is responsible for ensuring this.  Anything else cannot be expected to be standard compliant.  This could be phrased "undefined".
Furthermore, the OSSL_STORE API is new, and we are therefore free to define exactly what can be used through that API.  In the same discussion, the conclusion was that we can simply require that the pass phrase input be UTF-8 encoded, and that any object that's made accessible through a URI to OSSL_STORE must be converted in such a way that the UTF-8 encoded passphrase gets you the expected result.  That's simply a matter of unpacking the object with the old encoding and repack it with the UTF-8 encoded pass phrase.
With all that, this PR has become moot, as far as I can see.  However, since we make demands on the application, I thought it wise to try and avoid creating non-compliant PKCS#12 objects (thanks @dot-asm for the inspiration), see #6392.


Support for static text in formatting string

It would be very useful (and actually addresses a handful of other issues, like #281) to allow static text in the formatting string, like moment.js does by escaping the test with [ ]. This would allow unit suffixes, non-standard prefixes, etc. I realize numeral.js already makes use of [ ] for certain formatting, so I'm not sure of the best way to add this while maintaining backward compatibility. Use of [ ] would be great (in that it keeps with moment.js syntax), but that might also break existing uses of [ ]. Of course a different escape character is the easy choice, but I think there is value in maintaining the similarity to moment.js.

I have seen this requested a couple of times, so I will add it to the roadmap.  I'll probably use { }.

Glad to hear it. But since there already are configurable settings in numeral.js, why not make the token delimiters a configurable option (along with the existing [ ])? That way, static text is { } by default and the existing [ ] stays in place, but can be redefined if needed.
By the way, I'm not entirely sure what to call the current use of [ ]. How would you define it?

Hey @adamwdraper,
I know this story is  probably much more complicated than I seems, but do you have any update on it ?
Is is this part of the roadmap for  now ?

is there any update on this? it's a much needed feature for me. thanks

Hello,
any updates ?
thanks

after 3 years still no updates for this issue?

After 4 years and I still can't prefix my kibana visualizations with R$ because of this.

+1
I want to format 90539600 as 90.54 Million AUD in Kibana. Simple requirement!
PS: I know about adding a pattern for abbreviation.

+1
A "simple" approach like this custom format should theoretically work
// handle prefix or suffix enclosed in {}
numeral.register("format", "suffixprefix", {
  regexps: {
    format: /^(({.+})?([^{}]+)({.+})|({.+})([^{}]+)({.+})?|({.+})([^{}]+)({.+}))$/
  },
  format: function (value, format, roundingFunction) {
    // extract prefixes, suffixes and inner format
    let match = /^({.+})?([^{}]+)({.+})?$/.exec(format);
    let prefix = (match[1] ?? "").replace(/{|}/g,'');
    let innerFormat = match[2] ?? "";
    let suffix = (match[3] ?? "").replace(/{|}/g,'');
    output = numeral(value).format(innerFormat);

    return prefix + output + suffix;
  }
});

(see also this codepen: https://codepen.io/digemall/pen/jOpzbXv)
But the problem is that it cannot work when suffix or prefix contain another format specifier (e.g. the ordinal "o"): in that case that format will be used instead of the custom one.
The problem is that there isn't a way to "prioritize" formats in case of conflicts, but when more than one format regex matches, simply the first one is taken from the dictionary.
As a suggestion you could define a 4th parameter "priority", then the loop over the dictionary should be done sorting by priority.


New WROVER modules are missing from HW documentation

Environment
Documentation https://docs.espressif.com/projects/esp-idf/en/stable/hw-reference/modules-and-boards.html#wroom-and-wrover-modules
Problem Description
WROVER modules with new external RAM and Flash sizes are missing from table.
Expected Behavior
8MB RAM and 8/16GB Flash models added.
Actual Behavior
n/a
Steps to reproduce
n/a
Code to reproduce this issue
n/a
Debug Logs
n/a

Thanks for raising this issue. I'm going to close it as the issue is not related to ESP-IDF project. Please use the "Contact us" section of the website to report such issues, or post them on the esp32.com forum in Documentation requests/feedback topic: https://esp32.com/viewtopic.php?f=5&t=905.

Resolved in 752c7c32d460a92903adb50a60b33db729bfc28a.
Thank for reporting @gmos!


Removed os:linux agent attribute.

High-level description
This is a part of the windows cleanup that reverts adding 'os:linux` attribute to agents.
Corresponding DC/OS tickets (required)

D2IQ-68446 Remove the os:linux attribute from mesos agents.

@mesosphere-mergebot changelog-not-required no user-facing changes

@mesosphere-mergebot request-review DominikDary

@mesosphere-mergebot request-review greggomann


Specifying "-debug -verbose" in the dita command line seems to favor "verbose"

I think that it should favor "-debug" because it outputs more console output.

When multiple arguments that define the same thing are uses, the last of wins. I'd prefer to not change this, because then we'd need to start defining multiple rules for resolving multiple same arguments. A simple fix would throw an error when duplicates are defined, but that would be backwards incompatible.

Makes sense


x86 issue

I can't play in x86  processor. do you have any idea about this ?

Ya, i have problem with Intel processor based phone like Asus Zenfone

I solved this issue.. You should change libarm.so

@mos1907 could please tell me how i change libarm.so , i'm newbie in this

Now I am outside. When I came back I will explain to you..

K, Thanks

@mos1907 i'm waiting for your reply

I am at holiday now. I need a computer for explain.. Because I dont remember which file where.. When I came back to home I will say

Hello, I came back. First download this folder https://github.com/yixia/VitamioBundle/tree/master/dist/arm_x86
after change with into your project file ...projectname/platform/android/res/raw/ Change this--->( libarm.so)
then .projectname/platform/android/libs/ change all folders in this folder.
now you can play in x86 ;)
good luck ;)


feat: Upload User Profile Picture

Fixes #1182
Not ready for review yet
Checklist:

[x] I have checked for PMD and check-style issues
[x] I have read the Contribution & Best practices Guide and my PR follows them.
[ ] My branch is up-to-date with the Upstream development branch.

Changes:

User gets option to select images from File Explorer or local Gallery app.
The Image is converted to Base64 and sent to server.
The url returned by server is used to update user.avatarUrl in database and further used by glide to load image.

Screenshots for the change:

@iamareebjamal There is a bug in server. The url which is returned by server sows error on opening in browser and therefore Glide shows Failed to load resource when it's loaded through it.
Example url,
https://open-event-api-dev.herokuapp.com/static/media/temp/images/9afcf54b-0592-47d9-a1d8-1b6349b04176/emdHRkNCOU/bfcc5ef6-5489-414f-aba1-73e26a9c7100.jpg
I'll create an issue in server for the incorrect urls. Other than that, please review.

@Masquerade0097 Status?

@Masquerade0097 Status?

@iamareebjamal Apologies for the delay, i was facing some errors earlier mentioned above so kept it on hold. I will try my best to finalize it in 1 or 2 days.

The goal of GSoC is to get permanent addition to teams in Open Source projects. It's sad to see everyone leave after the party when there is time when we need people. Although GSoC is over and it is completely up to you guys if you want to continue contributing to the project, but I believe that it would be beneficial for both you and the project if you continue to add support to it at your own pace.
This PR is open for days. Anyone up to finalize it @Masquerade0097 @sridharjajoo @mishuvs

@iamareebjamal  Apologies, I'm very much occupied and won't be able to fix it for now. If anyone else want to work on it then they can.


The browse() method in Opera does not work.

The browse() method in "Opera" does not work.
<file-pond
            :name="filepondInput"
            @init="handleFilePondInit"
            :ref="pond"
            ...
            />

// calling open browse
this.$refs['pond'].browse();

Environment
Version

OS
Windows

Browser
Opera 56.0.3051.107

How are you calling the browse method?

Works for every browser. With the exception of the Opera.

It seems to work here. Not sure why it doesn't work in FilePond.
https://codepen.io/rikschennink/pen/vPRdqB

now I see that we have not understood ;) I'm using vue-filepond.
concept (handwritten):
<template>

        <file-pond
            :name="filepondInput"
            @init="handleFilePondInit"
            :ref="pond"
            />

test

</template>

It does not work for the Opera.

I’ll see if I can test this next week

I am sorry for my mistake. The above example works.
I had it differently, now I see what the mistake was.

I ran the trigger in the parent.
Filepond watched it.
When it changed, and tried to launch the browser.
But this was already beyond the onclick event.

How I changed:

adding to your component filepond, addEventListener ('click, this.openBrowsePond)
and in it I narrowed down listening for a specific object
It now works on any browser.

Glad to hear that, and thanks for sharing the solution for future reference.


NO_ISSUE: Fix CVE-2022-48345 (@braintree/sanitize-url)  & CVE-2022-25883(semver)

CVE-2022-48345: @braintree/sanitize-url Cross-site Scripting vulnerability
CVE-2022-25883: semver vulnerable to Regular Expression Denial of Service
Many thanks for submitting your Pull Request :heart:!
Please make sure that your PR meets the following requirements:

[ ] You have read the contributors guide
[ ] Pull Request title is properly formatted: KOGITO-XYZ Subject
[ ] Pull Request title contains the target branch if not targeting main: [0.9.x] KOGITO-XYZ Subject
[ ] Pull Request contains link to the JIRA issue
[ ] Pull Request contains link to any dependent or related Pull Request
[ ] Pull Request contains description of the issue
[ ] Pull Request does not include fixes for issues other than the main ticket

How to replicate CI configuration locally?

Build Chain tool does "simple" maven build(s), the builds are just Maven commands, but because the repositories relates and depends on each other and any change in API or class method could affect several of those repositories there is a need to use build-chain tool to handle cross repository builds and be sure that we always use latest version of the code for each repository.
build-chain tool is a build tool which can be used on command line locally or in Github Actions workflow(s), in case you need to change multiple repositories and send multiple dependent pull requests related with a change you can easily reproduce the same build by executing it on Github hosted environment or locally in your development environment. See local execution details to get more information about it.

How to retest this PR or trigger a specific build:

for pull request checks
Please add comment: Jenkins retest this

for a specific pull request check
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] tests

for quarkus branch checks
Run checks against Quarkus current used branch
Please add comment: Jenkins run quarkus-branch

for a quarkus branch specific check
Run checks against Quarkus current used branch
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-branch

for quarkus main checks
Run checks against Quarkus main branch
Please add comment: Jenkins run quarkus-main

for a specific quarkus main check
Run checks against Quarkus main branch
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-main

for quarkus lts checks
Run checks against Quarkus lts branch
Please add comment: Jenkins run quarkus-lts

for a specific quarkus lts check
Run checks against Quarkus lts branch
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] quarkus-lts

for native checks
Run native checks
Please add comment: Jenkins run native

for a specific native check
Run native checks
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] native

for native lts checks
Run native checks against quarkus lts branch
Please add comment: Jenkins run native-lts

for a specific native lts check
Run native checks against quarkus lts branch
Please add comment: Jenkins (re)run [kogito-apps|kogito-examples] native-lts

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.
If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

Quarkus-3 PR check is failing ... what to do ?

The Quarkus 3 check is applying patches from the `.ci/environments/quarkus-3/patches`.
The first patch, called 0001_before_sh.patch, is generated from Openrewrite .ci/environments/quarkus-3/quarkus3.yml recipe. The patch is created to speed up the check. But it may be that some changes in the PR broke this patch.
No panic, there is an easy way to regenerate it. You just need to comment on the PR:
jenkins rewrite quarkus-3

and it should, after some minutes (~20/30min) apply a commit on the PR with the patch regenerated.
Other patches were generated manually. If any of it fails, you will need to manually update it... and push your changes.

PR job #2 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer
build-chain build full_downstream  -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-apps -u https://github.com/apache/incubator-kie-kogito-apps/pull/1907 --skipParallelCheckout
NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/temporary/job/kogito-apps-1.40.x-pr/job/PR-1907/2/display/redirect
See console log:

Console Logs
[INFO] Execution summary for apache/incubator-kie-kogito-examples# [BEFORE] [apache/incubator-kie-kogito-examples] export INTEGRATION_BRANCH=[INFO] OK [Executed in 0.181219 ms] # [BEFORE] [apache/incubator-kie-kogito-examples] bash -c "if [ ! -z '' ] && [ -f .ci/environments/update.sh ]; then .ci/environments/update.sh  ; fi"[INFO] OK [Executed in 15.475233 ms] # [COMMANDS] [apache/incubator-kie-kogito-examples] mvn dependency:tree clean install -DskipTests -DskipITs  -s /home/jenkins/jenkins-agent/workspace/ry_kogito-apps-1.40.x-pr_PR-1907/kogito-pipelines@tmp/config10731027822991881872tmp -Dmaven.wagon.http.ssl.insecure=true -Dmaven.test.failure.ignore=true -nsu -ntp -fae -e -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 -Dmaven.wagon.http.retryHandler.count=3   -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B[INFO] OK [Executed in 1788801.176936 ms] [INFO] [AFTER] No commands were found for apache/incubator-kie-kogito-examples # Uploading artifacts[INFO] Will not upload any artifacts in CLI environment [Pipeline] sh+ find . -type d -name node_modules -exec rm -rf '{}' ';'find: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/node/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/runtime-tools-dev-ui-webapp/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/management-console-webapp/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/patternfly-base/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/common/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/trusty/node_modulesÃ¢ÂÂ: No such file or directoryfind: Ã¢ÂÂ./bc/apache_incubator-kie-kogito-apps/ui-packages/packages/components-common/node_modulesÃ¢ÂÂ: No such file or directory+ true[Pipeline] junitRecording test results[Checks API] No suitable checks publisher found.[Pipeline] archiveArtifactsArchiving artifacts[Pipeline] }[Pipeline] // withCredentials[Pipeline] }Deleting 1 temporary files[Pipeline] // configFileProvider[Pipeline] }[Pipeline] // stage[Pipeline] stage[Pipeline] { (Sonar analysis)[Pipeline] }[Pipeline] // stage[Pipeline] echoGot build result UNSTABLE[Pipeline] sh+ wget --no-check-certificate -qO - 'https://ci-builds.apache.org/job/KIE/job/temporary/job/kogito-apps-1.40.x-pr/job/PR-1907/2/api/json?depth=0'[Pipeline] readJSON[Pipeline] sh+ wget --no-check-certificate -qO - https://ci-builds.apache.org/job/KIE/job/temporary/job/kogito-apps-1.40.x-pr/job/PR-1907/2/consoleText+ tail -n 50

PR job #3 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer
build-chain build full_downstream  -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-apps -u https://github.com/apache/incubator-kie-kogito-apps/pull/1907 --skipParallelCheckout
NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/temporary/job/kogito-apps-1.40.x-pr/job/PR-1907/3/display/redirect
Test results:

PASSED: 3099
FAILED: 1

Those are the test failures:

org.kie.kogito.index.infinispan.ProcessDataIndexInfinispanKafkaIT.testProcessInstanceEvents
1 expectation failed.JSON path errors doesn't match.Expected: null  Actual: <[{message=Exception while fetching data (/UserTaskInstanceCommentUpdate) : Index 0 out of bounds for length 0, locations=[{line=1, column=12}], path=[UserTaskInstanceCommentUpdate], extensions={classification=DataFetchingException}}]>

@paulovmr would you mind taking a look?


MQTT as Endpoint

Would it be possible to have an MQTT endpoint be an option, instead of Home Assistant or REST?

It's not impossible. Can you provide an idea of what you would be looking for?
I assume it would be something like publishing the STT transcript to an MQTT topic or similar?

Exactly, basically I want to pull it into Node Red and then be able to do what I want.  Assist takes alot of work to set up simple functionality (I.e. timers or the like) that could be done quickly and simple with json/filter nodes. I know HA plans on making a lot of this stuff boiler plate in the future, but that's just one example.  Once I'm able to get the stt into an engine of my choosing (mqtt is pretty universal) I can move alot quicker.

If you don't mind a bit of python fiddling, then if you configure the exiting REST endpoint to point at a flask server something like this will pick out the STT results so you can post them to your MQTT broker:
from flask import Flask, request
import asyncio, websockets, ssl, json, os
app = Flask(__name__)

@app.route('/mqtt4willow', methods=['POST', 'GET'])
def rest4willow():
    print(f"request: {request}")
    jdata = request.json
    text = jdata.get('text')
    # post to MQTT here

If you don't mind a bit of python fiddling, then if you configure the exiting REST endpoint to point at a flask server something like this will pick out the STT results so you can post them to your MQTT broker:
from flask import Flask, request
import asyncio, websockets, ssl, json, os
app = Flask(__name__)

@app.route('/mqtt4willow', methods=['POST', 'GET'])
def rest4willow():
    print(f"request: {request}")
    jdata = request.json
    text = jdata.get('text')
    # post to MQTT here

I tried the above code and I see the post come in on flask, but I'm getting an error message from willow

Not sure what your error is but it might be that the method isn't finished and needs to return a value, eg the TTS string. You also need to fill in the mqtt post.

So if I send the following command to my flask app, it sends the text via mqtt.  But if I try via willow, i get an error in willow and the message doesn't come through.  So I must have the wrong json format, but can't find the proper format
curl -X POST -H "Content-Type: application/json" -d '{   "input": {     "text": "turn on basement table lamp"   }, }' http://192.x.x.x:5000/publish

What error are you seeing in willow?

What error are you seeing in willow?

the only error i get from willow that i can see is on my esp box, and its just Response invalid json data

I think that's the response from the server, maybe the server doesn't like what willow is sending?

Do you know how I can see what willow is sending? I'm more concerned with that than the feedback from willow at this point

You could hook the esp up to the serial monitor as described here: https://heywillow.io/development/#set-serial-port
Alternatively you could check the logs at the server side to see what it received, or add logging to show what it didn't like?

You could hook the esp up to the serial monitor as described here: https://heywillow.io/development/#set-serial-port and here: https://heywillow.io/development/#start-serial-monitor Alternatively you could check the logs at the server side to see what it received, or add logging to show what it didn't like?

So I was able to get flask to print what was being posted and looks like this is a sample. {'language': 'en', 'text': 'Turn on the TV.'}.  However, I believe that the single quotes are causing an issue, as its looking for double quotes.  I'll see if I can find a solution for that

If you don't mind a bit of python fiddling, then if you configure the exiting REST endpoint to point at a flask server something like this will pick out the STT results so you can post them to your MQTT broker:
from flask import Flask, request
import asyncio, websockets, ssl, json, os
app = Flask(__name__)

@app.route('/mqtt4willow', methods=['POST', 'GET'])
def rest4willow():
    print(f"request: {request}")
    jdata = request.json
    text = jdata.get('text')
    # post to MQTT here

FYI, here is the eventual flask app that got me working, in case anyone wants to go this route.  I still think an MQTT endpoint option would be good, but this works as a workaround for now.  Thanks everyone!
from flask import Flask, request, jsonify
import paho.mqtt.publish as publish

app = Flask(__name__)

# Define your MQTT broker information
MQTT_BROKER = "x.x.x.x"  # Replace with your MQTT broker address
MQTT_PORT = 1883  # Replace with your MQTT broker's port

@app.route('/publish', methods=['POST'])
def publish_to_mqtt():

    print(request.get_json())
    try:
        data = request.get_json()

        if not data or 'text' not in data:
            return jsonify({'error': 'Invalid JSON data or missing "text" field'}), 400

        text = data['text']
        if not text:
            return jsonify({'error': 'Missing "text" field in the "input" object'}), 400

        topic = "/willow"  # You can modify this to set a default topic
        message = text

        # Debugging: Print the incoming message
        print(f"Incoming Message: {message}")

        publish.single(topic, message, hostname=MQTT_BROKER, port=MQTT_PORT)

        response_data = {'Okay': 'Command Received'}
        #response_data = {'Command Received'}
        return jsonify(response_data), 200

    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
   app.run(host='x.x.x.x', port=5000)

Glad you got it working! I think the error you were seeing was to do with the flask server route method not returning anything, so now your "return jsonify..." call fixes it I guess. To add this type of thing as a built-in facility would mean something like:

adding willow/main/endpoint/mqtt.c with roughly what's in rest.c plus the equivalent of your "publish.single" call
adding config options to allow WAS to maintain the MQTT broker and port data

Very doable and would be useful :)

FYI, as of toverainc/willow-application-server@20f69d7822fa9537ece12f9df983b186d2944463 we started working on command endpoint support in WAS. This is included in the WAS 0.1.0-rc.1 release, and though currently still marked experimental, it is unlikely we will add new endpoints to Willow, or accept PRs to do so. You might want to have a look at adding an MQTT endpoint to WAS, instead of doing it in the Willow C code.

Would it be possible to have an MQTT endpoint be an option, instead of Home Assistant or REST?

Hello,
I also would love having MQTT endpoint implemented in Willow. And the reply from @stintel seems to confirm if such implementation occurs; it will be on WAS component.
In the meantime; maybe could you address your use case by making Willow talking to an MQTT implementation that natively supports REST API endpoints ?
Maybe this one could be interesting: https://docs.emqx.com/en/cloud/latest/api/publish_v5.html
Hope this will be helpful :)
Best regards

So I started on an MQTT endpoint for WAS: toverainc/willow-application-server@63fa17c5f21dc32a196259bde68120b42001e92d
If you want to test it you could use this container image: ghcr.io/toverainc/willow-application-server:feature-mqtt_endpoint. As this does not include the UI, you'll have to run it yourself.


Release ouroboros-consensus-cardano <IP_ADDRESS>

Description
There was a conflict with a version <IP_ADDRESS> what was already released in CHaP. Therefore we are bumping the release to <IP_ADDRESS> instead.

bors r+


syntax error in 23_texture_image.rs and onward

At line 928 in 24_sampler.rs, let (info, mut reader) = decoder.read_info()?; causes a syntax error of expected struct 'Reader', found tuple.

This seems to be caused by the latest version of the png crate.
Though I'd consider this a band-aid fix, as I like to keep dependencies up to date.

I've updated the dependencies for the tutorial.


Set level by env

Good idea this extension!
It's possible to set level alert by environment?
For example, to be free to decide to trigger just a warning in dev mode, test mode, but error prod mode.
The goal is to configure github actions/gitlab CI to just set a warning when push commit but block merged in main if error is triggered.

What do you mean by alert-level?
Phpstan does not has concepts like notices or warnings. Everything is an error.

Ok, I wasn't sure if phpstan allow to set warning instead of error

Thanks for your answer


pull request for e2e test at 12/16/2016 04:45:19

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


feat(component/network): latency checks to global edge/DERP servers (using tailscale)

/tmp/gpud scan

Check from Asia:
⌛ checking network connectivity to edge/derp servers

{"level":"info","ts":"2024-10-17T14:17:35Z","caller":"derp/derp.go:84","msg":"measuring latencies to DERP nodes"}
+-----------+---------------+--------------+
| PROVIDER  |  REGION NAME  |   LATENCY    |
+-----------+---------------+--------------+
| tailscale | Tokyo         | 135.518953ms |
| tailscale | San Francisco | 162.361141ms |
| tailscale | Seattle       | 172.660256ms |
| tailscale | Hong Kong     | 186.865222ms |
| tailscale | Los Angeles   | 192.90225ms  |
| tailscale | Denver        | 198.14393ms  |
| tailscale | Singapore     | 205.379052ms |
| tailscale | Honolulu      | 225.790173ms |
| tailscale | Toronto       | 225.854292ms |
| tailscale | London        | 228.189182ms |
| tailscale | New York City | 228.32153ms  |
| tailscale | Paris         | 229.316372ms |
| tailscale | Chicago       | 230.087345ms |
| tailscale | Dallas        | 243.016964ms |
| tailscale | Miami         | 246.843145ms |
| tailscale | Madrid        | 247.591275ms |
| tailscale | Frankfurt     | 277.455938ms |
| tailscale | Warsaw        | 291.881769ms |
| tailscale | Amsterdam     | 292.571809ms |
| tailscale | Dubai         | 330.59889ms  |
| tailscale | Bangalore     | 332.328797ms |
| tailscale | São Paulo     | 348.24734ms  |
| tailscale | Sydney        | 376.831077ms |
| tailscale | Johannesburg  | 394.437529ms |
| tailscale | Nairobi       | 427.144011ms |
+-----------+---------------+--------------+

Check from US:
⌛ checking network connectivity to edge/derp servers

{"level":"info","ts":"2024-10-17T14:18:37Z","caller":"derp/derp.go:84","msg":"measuring latencies to DERP nodes"}
+-----------+---------------+--------------+
| PROVIDER  |  REGION NAME  |   LATENCY    |
+-----------+---------------+--------------+
| tailscale | New York City | 8.998541ms   |
| tailscale | Toronto       | 19.060602ms  |
| tailscale | Chicago       | 21.435507ms  |
| tailscale | Miami         | 29.443248ms  |
| tailscale | Dallas        | 38.949296ms  |
| tailscale | Denver        | 44.731894ms  |
| tailscale | Seattle       | 62.731356ms  |
| tailscale | Los Angeles   | 64.115506ms  |
| tailscale | London        | 68.405422ms  |
| tailscale | San Francisco | 69.494831ms  |
| tailscale | Paris         | 72.688823ms  |
| tailscale | Amsterdam     | 77.882517ms  |
| tailscale | Frankfurt     | 79.832446ms  |
| tailscale | Madrid        | 87.526568ms  |
| tailscale | Warsaw        | 100.637818ms |
| tailscale | São Paulo     | 111.130095ms |
| tailscale | Honolulu      | 116.181365ms |
| tailscale | Tokyo         | 156.465546ms |
| tailscale | Dubai         | 193.692553ms |
| tailscale | Bangalore     | 209.425136ms |
| tailscale | Hong Kong     | 213.400923ms |
| tailscale | Singapore     | 224.648866ms |
| tailscale | Johannesburg  | 239.468557ms |
| tailscale | Sydney        | 259.728192ms |
| tailscale | Nairobi       | 286.71491ms  |
+-----------+---------------+--------------+

do not name this derp anywhere. we treat these are edge servers for pinging.
todo: map each location to a region based on aws region definitions. assign the machine with the nearest region if not explicitly specified by the user.
todo: get the ASN (https://asnlookup.com/) of the machine and assign the ISP provider of the machine if not explicitly specified by the user.

@xiang90 @cardyok Done
+----------------+------------------+---------------+----------------+--------------+
|    PROVIDER    |       ASN        |  REGION NAME  |   AWS REGION   |   LATENCY    |
+----------------+------------------+---------------+----------------+--------------+
| tailscale-derp | NETACTUATE       | Tokyo         | ap-northeast-1 | 45.730875ms  |
| tailscale-derp | NETACTUATE       | Hong Kong     | ap-east-1      | 69.628833ms  |
| tailscale-derp | NETACTUATE       | Singapore     | ap-southeast-1 | 83.550334ms  |
| tailscale-derp | NETACTUATE       | San Francisco | us-west-1      | 141.274208ms |
| tailscale-derp | NETACTUATE       | Los Angeles   | us-west-1      | 148.22575ms  |
| tailscale-derp | NETACTUATE       | Seattle       | us-west-2      | 153.131417ms |
| tailscale-derp | NETACTUATE       | Denver        | us-west-1      | 164.03975ms  |
| tailscale-derp | NETACTUATE       | Dallas        | us-west-2      | 174.929666ms |
| tailscale-derp | NETACTUATE       | Sydney        | ap-southeast-2 | 183.635083ms |
| tailscale-derp | AS-CHOOPA        | Honolulu      | us-west-2      | 187.728333ms |
| tailscale-derp | NETACTUATE       | Chicago       | us-east-2      | 190.22275ms  |
| tailscale-derp | NETACTUATE       | Toronto       | ca-central-1   | 194.690791ms |
| tailscale-derp | NETACTUATE       | New York City | us-east-1      | 199.599ms    |
| tailscale-derp | NETACTUATE       | Miami         | us-east-1      | 201.81675ms  |
| tailscale-derp | DIGITALOCEAN-ASN | Bangalore     | ap-south-1     | 229.639125ms |
| tailscale-derp | NETACTUATE       | Amsterdam     | eu-west-1      | 248.295458ms |
| tailscale-derp | NETACTUATE       | Frankfurt     | eu-central-1   | 259.755625ms |
| tailscale-derp | NETACTUATE       | London        | eu-west-2      | 269.978333ms |
| tailscale-derp | NETACTUATE       | Madrid        | eu-south-2     | 278.646ms    |
| tailscale-derp | NETACTUATE       | Paris         | eu-west-3      | 281.815208ms |
| tailscale-derp | NETACTUATE       | Dubai         | me-central-1   | 284.744417ms |
| tailscale-derp | NETACTUATE       | Warsaw        | eu-central-1   | 306.22975ms  |
| tailscale-derp | NETACTUATE       | São Paulo     | sa-east-1      | 314.844167ms |
| tailscale-derp | NETACTUATE       | Johannesburg  | af-south-1     | 425.645791ms |
+----------------+------------------+---------------+----------------+--------------+

not checking the asn of the edge server but the machine gpud is on.

@xiang90 Got it. Will do that in a separate PR.


dashboard aggregates with include and exclude filters and NULL values are not correctly calculated

see the following notion link for more information https://www.notion.so/rilldata/dashboard-aggregates-with-include-and-exclude-filters-and-NULL-values-are-not-correctly-calculated-f41b5baeafbc4e9da748bf1cd39a4c2f

This is due to the default null handling in SQL.
@egor-ryashin : Can you please help fix this ?
The fix should be simple  -
In case a user selects a list of values to be exluded and that list doesn't contain null.
We would need to smartly generate filter as -
(( NOT IN ) OR ( IS NULL))


Share access to AppVeyor account

@mathiask88 I'd like to be able to cancel and restart builds on AppVeyor to speed up development. Can you add me to the account? Email me if needed (address in GH profile) (I could not find yours). Thanks!
In case you use your account for other projects and thus cannot share access, we can create a new organizational account. That's also what I'll do if you don't have the time to attend to this. ❤

Hey @vweevers, yes I use this AppVeyor account for another project but I set up a prebuild user role and added the whole Prebuilder team. So everyone should have access to the prebuild stuff in my account now. However we should go for a new account for this organization to be fully independent. But for now you should be g2g.

Awesome! Thanks for the quick action! I'll repurpose this issue to be about opening a new account.

Duplicate of https://github.com/prebuild/prebuild-install/issues/62


Testflight Builds wrong return when empty

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I have read the Contribution Guidelines

Issue Description
While using:
Spaceship::TestFlight::Build.all(app_id: "XXX", platform: "ios" )
Throws a parser error
We don't have already any build for this app_id, but the command throw an error, not an empty collection.
Surelly this will be an issue of iTunes, but I prefer to open this to advice any other users.
There is any workaround to get an empty collection? I'm using this on a script and this will break it.
PD: If i try it with other app_id which have builds, returns correctly this builds.
Complete output when running fastlane, including the stack trace and command used
Spaceship::TestFlight::Build.all(app_id: "XXX", platform: "ios" )

This throws:

Faraday::ParsingError: 743: unexpected token at '<!DOCTYPE html>

<title></title>
<link rel="stylesheet" href="/styles/main.css">

Internal Server Error

Copyright © 2015 Apple Inc. All rights reserved.

Terms of Service

Privacy Policy

Contact Us

'
from ../.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/json-2.0.3/lib/json/common.rb:156:in `parse'

Environment
Please run fastlane env and copy the output below. This will help us help you :+1:
If you used --capture_output option please remove this block - as it is already included there.
✅ fastlane environment ✅
Stack

Key
Value

OS
10.12.5

Ruby
2.3.1

Bundler?
false

Git
git version 2.11.0 (Apple Git-81)

Installation Source
~/.rbenv/versions/2.3.1/bin/fastlane

Host
Mac OS X 10.12.5 (16F73)

Ruby Lib Dir
~/.rbenv/versions/2.3.1/lib

OpenSSL Version
OpenSSL 1.0.2j  26 Sep 2016

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
8.3.3

System Locale

Variable
Value

LANG
es_ES.UTF-8
✅

LC_ALL

LANGUAGE

fastlane gems

Gem
Version
Update-Status

fastlane
2.47.0
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.0.2

slack-notifier
1.5.1

CFPropertyList
2.3.5

claide
1.0.1

colored2
3.1.2

nanaimo
0.2.3

xcodeproj
1.4.4

rouge
1.11.1

xcpretty
0.2.4

terminal-notifier
1.7.1

plist
3.2.0

public_suffix
2.0.5

addressable
2.5.1

multipart-post
2.0.0

word_wrap
1.0.0

tty-screen
0.5.0

babosa
1.0.2

colored
1.2

highline
1.7.8

commander-fastlane
4.4.5

excon
0.54.0

faraday
0.11.0

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20161129

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

fastimage
2.1.0

gh_inspector
1.0.3

declarative
0.0.9

declarative-option
0.1.0

representable
3.0.3

mime-types-data
3.2016.0521

mime-types
3.1

little-plugger
1.1.4

multi_json
1.12.1

logging
2.1.0

jwt
1.5.6

memoist
0.15.0

os
0.9.6

signet
0.7.3

googleauth
0.5.1

httpclient
2.8.3

google-api-client
0.12.0

mini_magick
4.5.1

multi_xml
0.6.0

security
0.1.3

xcpretty-travis-formatter
0.0.4

dotenv
2.2.0

bundler
1.14.3

faraday_middleware
<IP_ADDRESS>

json
2.0.3

io-console
0.4.6

rubyzip
1.2.0

unicode-display_width
1.1.3

terminal-table
1.7.3

generated on: 2017-07-20

Sorry, this is embarrasing.
I was getting a wrong app_id, is not related with the tool
Sorry again and thank you.
This could be closed/deprecated


bug basic javascript waypoint #32

Challenge http://www.freecodecamp.com/challenges/waypoint-use-conditional-logic-with-ifelse-statements has an issue. Please describe how to reproduce it, and include links to screenshots if possible.
function myFunction(){
  var flip = Math.floor(Math.random() * (1 - 0 + 1)) + 0;
  // Create and if else statement here to return "heads" if flip is 0. Otherwise return "tails".
  if (flip === 0) {
    return("heads");
  } else {
    return("tails");
  }
  // Only change code below this line.

}

// Only change code above this line.
// We use this function to show you the value of your variable in your output box.
(function(){return(myFunction());})();

If using double quotes for heads and tails the if statement will not pass.

Using single quotes for heads and tails the if statement will pass
function myFunction(){
  var flip = Math.floor(Math.random() * (1 - 0 + 1)) + 0;
  // Create and if else statement here to return "heads" if flip is 0. Otherwise return "tails".
  if (flip === 0) {
    return('heads');
  } else {
    return('tails');
  }
  // Only change code below this line.

}

// Only change code above this line.
// We use this function to show you the value of your variable in your output box.
(function(){return(myFunction());})();

Hey thanks for reporting this!  This was already reported in #1974 so check out the explanation there, but the fix is pending in Pull Request #2158