OAuth HTTP filter

Title: OAuth HTTP filter
Description:
We have an internal HTTP filter that implements OAuth 2.0 (RFC 6749). I wanted to gauge community interest in upstreaming this. We've been running it for all of our internal sites for about a year now. cc @bplotnick who I think had interest.

We were mostly using the Identity Token portion of OpenID Connect. We just were relying on the IdP to give us some signed token and it just happened to be a JWT. We ignored everything else about OIDC and OAuth 2.0 (standard OAuth 2.0 does not include Identity Tokens IIRC). From everyone we talked to, it seemed like it was a pretty unique use-case, though I guess we'll see when we present at KubeCon.
I assume your filter implements the "Client" part of a given OAuth 2.0 flow. Like redirecting the user and then exchanging the authorization code for a token when the user is redirected back: https://developer.okta.com/blog/2018/04/10/oauth-authorization-code-grant-type#the-authorization-code-flow
That seems pretty useful, though i'd never personally want to implement OAuth 😄
CC: @kaisen and @danielpops

@bplotnick gotcha, yeah you're correct that our filter implements the client-side workflow as described in that link.

This would be quite amazing as we are also looking into implementing such a filter. We currently have to run a separate gatekeeper proxy that handles the auth flow and having this feature would allow us to secure webapps from within envoy rather than within the webapp itself or by using a separate proxy.

In general I would love to see this happen as I think this is a pretty big gap in the ecosystem right now. With that said, is there plan to OSS the backend (token storage, etc.)? Is there a design doc for what this would cover?

@Barborica-Alexandru great to hear! We were also in the "separate proxy" situation, using OpenResty with a Lua script to provide OAuth for webapps, so it was great to consolidate it to Envoy.
@mattklein123 the way the filter works currently is that it makes an HTTP request to a CDS cluster for the actual authentication so it's bring-your-own-authentication-service so that it can plug into whatever backend (for us it's LDAP). The workflow we have is

get your client ID & secret from your auth service and configure it in the filer (we have an integration with our internal secret storage, Knox, which reminds of a separate conversation I wanted to have [1]).
When the filter gets requests it validates the authenticity of the standard OAuth cookies or  redirects the user to the auth service to start an OAuth flow. The filter knows how to handle the callbacks and set cookies, or reject if credentials are invalid.
I'll put together a design document.

[1] We didn't want to write the secret in plaintext for obvious reasons. Any thoughts on expanding SDS to cover general secrets so things like access tokens could be fed from the control plane? Happy to open another issue if you think it's worth a discussion. I'm not finding an existing issue of similar topic.

Definitely ... we are only using NGINX Ingress Controller on our UIs because they are fronted by SSO (using Dex and oauth2 proxy) if Envoy made this simple then we could have a truly Envoy'd platform.

+1 for this. We too are in the "separate proxy" scenario and it is requiring us to keep around an NGINX ingress controller we'd like to remove.

Also a big +1 for this. I’ve utilized the separate proxy flow before and IMO it’s more error prone then if the client-side logic was handled by an envoy filter.

@nickrmc83 thanks for all the resources! Reading through the design & code.

Circling back here, I have a doc nearly complete - hoping to share tomorrow or Monday, just needs some cleanup/diagrams.
I've spent a significant time reviewing the OAuth and OIDC specs and after much consideration, the doc will only propose a design for implementing the Authorization Code OAuth workflow for now, with open discussion points for OIDC and improving the jwt_authn filter to support Discovery now that we have a dynamic forward proxy. The reason this was taking a lot of time (besides other distractions 😬) was trying to figure out how to fit OIDC, but I'd like to start with just OAuth for 2 reasons:

It provides a reasonable 1st milestone for an MVP that can be extended to support OIDC at a later time since OIDC is roughly a superset of features on top of OAuth.
OIDC has a bit more state management for validating the ID token. The Authorization Code OAuth workflow can be completely implemented within an Envoy filter by leveraging secure cookies for handling the state parameter. But OIDC additionally has an optional nonce, (as called out by the Istio folks in their initial design document, and I'm guessing was a motivating factor for moving to ext_authz + authservice, granted nonce is an optional feature).
it provides a nice building block that could be combined with jwt_authn or ext_authz.
An Authorization Code OAuth filter is also what we've built internally so far at Pinterest and matches the initial motivation for opening this issue.

So TL;DR initial MVP would validate or obtain an access token from an IdP via the Authorization Code OAuth 2.0 workflow, which is stored in a secure cookie scheme similar to oauth2-proxy or Istio authn's __Secure-istio_session cookie, and can log out by clearing the cookie. The information can also be proxied through metadata or headers for subsequent filters to work with, and JWTs can be validated with jwt_authn.

Nice, also interesting in following along with this. We recently had to build a separate service that handles the authorization code flow, and speaks to envoy with the ext_authz filter. Would be cool if there was a way to do this all in envoy however.

We're also in a situation here at spotify where we'd need a filter like this for internal services behind oauth2 / okta

dev branch here: https://github.com/derekargueta/envoy/tree/oauth-filter
To keep the scope limited, I plan on initially supporting the authorization code grant for browser-based workflows, storing an HMAC'd token in browser cookies. Subsequent iterations can add support for PKCE, JWT, etc.
one open question I have for now is the handling of state - randomly generated nonce in the 302 redirects that, when redirected back, should match the initial nonce. The caveat is that when redirected back, it is not guaranteed (if anything, highly unlikely) to land back on the same Envoy that generated the nonce. One possibility is to store it in a browser cookie (httpOnly, secure, etc.) so that when the user-agent is redirected back to Envoy, it carries that cookie with it which can be validated against the query parameter.
state is optional so we could technically punt on it for now, but it is highly recommended to protect against CSRF

@derekargueta I don't see any objection to storing the state in a secure browser cookie. In fact, we do the same in the OIDC external_authz filter that we built within our organization.
Do you also plan to work on refreshing the access token as it (nears?) expiry, using the refresh token? The refresh token would be present if the offline_access scope is requested.
Looking forward to have this handled by Envoy directly!

Hi everyone,
In case it's helpful, I wanted to add that there is a related project in the official Istio ecosystem already which can be used in either pod sidecars or at the ingress gateway to provide everything you need for your end users to acquire tokens from OIDC-compliant identity providers using the authcode flow. It can be easily combined with the existing JWT checking capabilities of Istio to require/validate the tokens that it acquires.
The service:

Automatically redirects any unauthenticated end user to the configured identity provider for login
After user login, receives and authcode via a redirect from the identity provider and makes a bank-end to back-end call back to the IDP to exchange the authcode for tokens. It stores the tokens on behalf of the end user.
Transmits either the ID token or the access token (or both) to the downstream application via http headers, without allowing the end user's browser to see those tokens (for security)
Automatically refreshes those tokens using the refresh token whenever possible

The project is open source and is a work in progress, under active daily development. Suggestions, issues, and PRs via github are welcome. https://github.com/istio-ecosystem/authservice
You can also find the development team on https://istio.slack.com in the #oidc-proposal channel.
Specifically related to @derekargueta's question about storing the state and nonce in a browser cookie: yes, this was our approach too, previously. We additionally encrypted the contents of the cookie using a secret which was only known by the backend to prevent the user from seeing or tampering with the values. However, this week we are refactoring that approach and moving the values to backend session storage, so we will not store them in cookies anymore.

@derekargueta I don't see any objection to storing the state in a secure browser cookie. In fact, we do the same in the OIDC ext_authz filter that we built within our organization.
Do you also plan to work on refreshing the access token as it (nears?) expiry, using the refresh token? The refresh token would be present if the offline_access scope is requested.
Looking forward to have this handled by Envoy directly!

Yes refresh tokens will be supported, I’ll update the doc

@derekargueta I don't see any objection to storing the state in a secure browser cookie. In fact, we do the same in the OIDC ext_authz filter that we built within our organization.
Do you also plan to work on refreshing the access token as it (nears?) expiry, using the refresh token? The refresh token would be present if the offline_access scope is requested.
Looking forward to have this handled by Envoy directly!

Yes refresh tokens will be supported, I’ll update the doc

will your branch be merged into master or any formal release?

This is merged. Let's open new issues for follow ups.

For anyone else stumbling across this issue (like me) the documentation of the OAuth2 filter is located here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/oauth2/v3alpha/oauth.proto.html

in case anyone is interested digging PRs, https://github.com/envoyproxy/envoy/pull/11268 is the original impl.


Cannot make WiFi connection?

I have a WiFi network but I cannot seem to connect when trying to connect on the https://hq.pinocc.io/troops main page, what is wrong?

I suspect you also posted on IRC yesterday? There you said the message was 'Couldnt find your Scout :( ', which makes me suspect that it's not a wifi connection issue, but an issue with the local serial connection.
It's been a while since I went through this process, but did you install the Pinoccio serial driver? What OS are you using? Is a serial port listed for the pinoccio in the device manager (Windows) or in /dev (Linux/OSX)?

One issue I've had doing demos in the field is when there is just too much noise due to overlapping WiFi networks, or other signal issues like that. It's not practical for everyone, but you can try switching your access point to a different channel to see if that helps.
If this is preventing you from getting started entirely, you can try setting your lead scout up in bridging mode: http://support.pinocc.io/hc/en-us/articles/202903250


Recommend using the 0x prefix in App.config

Summary
Update the GC Configuration page to recommend using 0x prefix in App.config to avoid a known unfortunate bug.
@dotnet/gc

Thanks for the rewrite, but I am afraid the rewrite is less clear though.
If I were a customer trying to set the GCHeapCount the natural base to use is decimal and I would write 12 there. If I read your message, it isn't clear why I would want to change it to hex and add the 0x prefix. It is more obscure that way and doesn't make anything better. The one with decimal notation will work just fine (on 4.7.2.) and mostly fine on 4.8 (but got misinterpreted as 18, which I wouldn't notice until I really look into it in the debugger).
That's a trap that we want customers to avoid. The reason I added this note is to tell customers about this trap so they can avoid it.
I don't have a strong preference for emphasizing it is a bug, but the message needs to redirect the user to use the 0x hex notation even when it is unnatural to do so.

@cshung I made a different suggestion. Is that more clear?

so, I already heard feedback from customers that the description of how to specify configs sounded confusing. with this additional note it's even more info to take just to specify a config. I think if I were a customer I'd appreciate something less to read. how about changing the description to something like a table? please let me know if this seems clearer to both of you.

config location
.net versions this location applies to
formats
how it's interpreted

app.config
.NET Framework
0xn
n is interpret as a hex value  (also see note below)

env var
.NET Framework, .NET Core
0xn or n
n is interpret as a hex n is interpret as a hex value number in either format

runtimeconfig.json
.NET Core
n
n is interpreted as a decimal value.

Note that you could specify a value with the 0x prefix for app.config but we do not recommend that. Due to an unfortunate bug, on .NET Framework 4.8+ it's interpreted as hex but on previous versions of .NET Framework it's interpreted as decimal. To avoid having to change your config please always use the 0x prefix when specifying a value in your app.config.
An example, if I want to specify 12 heaps for GCHeapCount, this is how I would specify it for an executable named a.exe [give the example how to specify that in all 3 ways]

I think it's a great idea @Maoni0!

looks good. my text about the example sounds too casual for official doc - @gewarren could you please help us make it more suitable for official documentation?
I'd be totally fine if we left this in the GC doc page just because some of this is specific to GC (eg the 4.8 bug). and it's nice to have just one page to go to.

How does this look? Preview


Syntax highlighting messes up

Issue Type: Bug
Sometimes after some kind of edit, the syntax highlight totally messes itself up, and becomes really hard to read to people who rely on it to better understand code:

Answers to questions I expect this to arise:

This is a plain javascript file. Nothing special.
This happens seemingly randomly, just while editing.
Closing & reopening the file keeps this messed up highlighting.
Removing the change that caused this to happen doesn't fix it.
I have no clue what exactly triggers this. Yes, editing, but what/how exactly?..
Other files are fine, until they too become messed up.
Reloading/restarting VScode "fixes" it, until it messes up again (which it will).

VS Code version: Code 1.68.0 (4af164ea3a06f701fe3e89a2bcbb421d2026b68f, 2022-06-08T11:44:16.822Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz (12 x 2904)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.76GB (12.63GB free)

Process Argv

Screen Reader
no

VM
50%

Extensions (32)

Extension
Author (truncated)
Version

overtype
ada
0.2.0

codesnap
adp
1.3.4

ng-template
Ang
14.0.1

font-preview
ctc
1.3.2

vscode-eslint
dba
2.2.2

xml
Dot
2.5.1

EditorConfig
Edi
0.16.4

vscode-great-icons
emm
2.1.86

vscode-firefox-debug
fir
2.9.6

remotehub
Git
0.34.0

todo-tree
Gru
0.0.215

vscode-totalcommander
ipa
1.0.4

rainbow-csv
mec
2.4.0

document
min
2.1.22

vscode-scss
mrm
0.10.0

vscode-docker
ms-
1.22.0

csharp
ms-
1.25.0

remote-wsl
ms-
0.66.3

powershell
ms-
2022.5.1

remote-repositories
ms-
0.12.0

gremlins
nho
0.26.0

vetur
oct
0.35.0

vscode-versionlens
pfl
1.0.10

prisma
Pri
3.15.0

sqlite-viewer
qwt
0.1.5

geo-data-viewer
Ran
2.5.0

vscode-sort-json
ric
1.20.0

mdx
sil
0.1.0

vscode-svg-previewer
vit
0.7.0

volar
Vue
0.37.8

vscode-typescript-vue-plugin
Vue
0.37.8

vscode-import-cost
wix
3.3.0

Please refrain from calling "cannot reproduce" and be done with it. The whole point is that this issue seems random/intermittent and hard to reproduce on demand. And as such, it's not feasible to ascertain whether this problem comes from an extension or not.

Does this reproduce in the latest VS Code insiders build with all extensions disabled?

I encountered with a similar case.
As he describes it, it is purely by chance. So I do not have any reproduceable steps nor any screen shots. In my case, comment lines were not greyed out but maintained their colors as if they were not commented out, which is very confusing.
I restarted the vs code then it was gone.

Color Theme: GitHub Dark Default

Version: 1.68.1
Commit: 30d9c6cd9483b2cc586687151bcbcd635f373630
Date: 2022-06-15T02:58:26.441Z
Electron: 17.4.7
Chromium: 98.0.4758.141
Node.js: 16.13.0
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.18.5-200.fc36.x86_64

@mjbvz

Does this reproduce in the latest VS Code insiders build with all extensions disabled?

I don't know. As said, this is hard to reproduce on-demand. It may go a whole day without messing up, or it may happen within minutes. So hypothetically:

If it DOES still happen in Insiders without extensions - what extra information have we gained? That it still happens? That, we already knew...
If it DOESN'T still happen in Insiders without extensions - who's to say it's actually fixed? It's an intermittent issue, therefor it is mathmatically impossible to check for its absence (just like the famous teapot orbiting Mars).

If a problem concerning this issue hasn't been fixed anywhere between the current stable release, and the master branch, then we should assume it hasn't been fixed. Instead of narrowing down which branch this does and doesn't happen on, we should focus on how to reproduce it first. Then and only then, we can ascertain whether it truly doesn't happen anymore on any given version.
And again, how to reproduce, I don't have an answer to that. I was hoping the experts did.

Sorry but that's not actionable. If it doesn't happen in insiders with extensions disabled, then this issue should be closed. If it does, try narrowing down steps to reproduce the problem so we can investigate it

FYI, Vue.vscode-typescript-vue-plugin has caused similar problems before.

If I try Insiders without extensions, I'm not only testing 2 variables at once (Insiders AND disabling extensions), I'm also in the dark as to what I'm supposed to do. I don't know how to reproduce this reliably. I would need help with that first - then I can try the same steps in Insiders.
In the same way, if this issue doesn't happen for you, @mjbvz, you cannot possibly prove it either. We can only prove that it DOES occur, if it does occur. This is just how provability works, I can't help it either.
So, the question is: what could possibly lead to this problem? What sort of reproduction steps can I try, without changing any other variables?

I've already said what to try: disable extensions. Another user even reported that one of the extensions you had installed (vscode-typescript-vue-plugin) caused similar problem

Just trying stuff is a very unscientific way to track down a problem. Imagine if aviation did troubleshoting like that.
I will repeat myself one more time then: I DO NOT KNOW what to do in order to cause the problem. So whatever it is you want me to try, will never hope to prove that the problem is gone. I can only prove if it still happens, were I to know how to reproduce.
So I ask again: what sort of steps do YOU think I can undertake to get this problem to relaibly reproduce itself?
In the mean time, sure. Fine. I'll disable that extension. And maybe the problem doesn't happen. But then what? The problem hasn't occurred for a few days with this extension enabled. So if the problem seems to be gone, what exactly do we know that we didn't know before?
Like I said, unscientific.


Skill - Petrify

A skill % chance of inflicting the petrify status on the opponent.

The activated skill should not put at pre-battle calc.
It should be rolled RN after MISS RN has been roled in BattleGenerateHitAttributes.
Kernel also offered one common roll RN API as CheckBattleSkillActivte
This PR will be rejected as because it fails to follow the function logic of the vanilla. It is recommended that you modify the code and then resubmit

PoisonPoint skill is also fixed
https://github.com/MokhaLeee/fe8u-cskillsys-kernel/pull/57


Changes to IDM example, sample tests.

RE: DX-24, "Make iOS OAuth2 client testable"
The IDM example has been restructured; sample tests have been added to both Basic and IDM examples.
The README has been changed ever slightly to provide more precise meaning in some cases.

Mark, I added you to reviewers as well, if you have time/desire to take a look at the README. I suspect you may have similar comments to the valuable ones you had for the other samples.

Removed the test, added screenshots.

I haven't suggested many edits to the README. It worked for me. The only bits I stumbled over were on in forgeops-init and using a README on the wrong branch.


there can be only one TIMESTAMP column with CURRENT_TIMESTAMP in DEFAULT or ON UPDATE clause

(affects PHP < 5.6)
caused by https://github.com/rockettpw/jumplinks/commit/cc4281aa1a4545fe1e8206a545d0b2088cc50759

Hi, and thanks for opening this. I have tackled this issue before, and somewhere along the line it created an issue for some folks (can't remember the details right now). I will look into this a little further, but would prefer to leave it as is giving consideration to Jumplinks 2 development (I've switched over to Illuminate Database).

As this exception relates to older versions of MySQL, I will not be fixing this issue. I recommend that you upgrade to MySQL 5.7.


ONNX export in pytorch

--save-onnx does not work.

nn.EmbeddingBag is not supported in ONNX converter
Zflat = Z[:, li, lj] at line 235 of dlrm_s_pytorch.py is translated to torch index which cant be
exported.

proposed solution:

Since in criteo dataset the sparse features are categorical nn.EmbeddingBag can be replaced with nn.Embedding.
since the size of bmm matrix is known, prepare the li,lj indexes with np.tril_indices. then use torch.index_select which will be translated to onnx gather:
li,lj = np.tril_indices(ni,offset,nj)
tril_indexes=[]
for i,j in zip(li,lj):
tril_indexes.append(i*ni+j)
Zflat=torch.reshape(Z,(Z.shape[0],Z.shape[1]*Z.shape[2]))
tril_indexes_tensor=torch.LongTensor(tril_indexes)
Zflat=torch.index_select(Zflat,1,tril_indexes_tensor

After all that, there is still an issue since pytorch onnx translator uses protobuf which doesnt support buffers that exceed 2GB (which is true for criteo dlrm model).
model can be saved without the parameters using "export_params=False"

Thank you for your comments. You are correct that the translation to ONNX has issues due to lack of ONNX support for some of the operators used to describe the model. Let me address some of your suggestions below.

There are tradeoffs associated with using Embedding and EmbeddingBag layers. Perhaps the most critical difference is that the inputs to the layers are defined differently. In particular, EmbeddingBag allows lookups with different number of indices to be easily batched together, while Embedding requires the number of indices in each lookup to be constant within a batch.

For the Kaggle Display Advertising Challenge Dataset this difference is irrelevant because each lookup has a single index in it, but the model is more general and can accept multiple indices per lookup (which can be controlled with a parameter from the command line). That is why we made a conscious choice to use the EmbeddingBag layer in the implementation.

This change seems reasonable. It make the code more compatible with ONNX at the expense of making it slightly more complicated. As you mentioned ultimately you will still hit a 2GB limit for buffer sizes for this dataset.

Therefore, if you are interested in saving protobuf without the parameters (weights/bias) then my advice is to try to use the Caffe2 version with an option "--save-proto-types-shapes", which should save the protobuf of the model including the shape and type of each of the operators. Alternatively, you can use the PyTorch version with an option "--save-model" and "--load-model" to save and load the model with parameters, respectively.


"...Library not initialized"

I've just started implementing WLT (1.5.9) in my project and managed to successfully save and retrieve object poses on fixed locations in the room, regardless of where I started the HoloLens.
However, today when I tried it out in a new physical location, it didn't seem to remember the room between runs, and I got the error
EngineException: Error in call to FrozenWorld.Engine: "FrozenWorld_ClearAnchors: (failed s_state) Library not initialized"
Now, when I go back to my original room which worked before, I get the same error. I've tried restarting Windows, Hololens and removing and reinstalling WLT core, but it's still broken.
It seems that when I restart windows and open the project again, I am able to get the correct anchors for the room, but no fragments, and the error:
vEngineException: Error in call to FrozenWorld.Engine: "hkAnchorGraphFracture::detect: (failed associationSlot < (sizeof(Bits) * 8))  [internal error 0x6DAC5CE2 in Base\hkAnchorGraphFracture.cpp line 150]"
Then, when I restart the project, I get the first error message indefinitely again.
Setup
HoloLens 2
Unity 2020.3.27f
MRTK Foundation 2.8.2
WLT core 1.5.9
Windows Mixed Reality XR plugin
Using Windows Mixed Reality Holographic Remoting
Addition
After clearing all spatial/holographic data and multiple restarts of the Unity editor it seemed to resolve itself. ... And Now it's back again...
I was under the impression that when visiting a new location disjoint from a previous mapped location, a new anchor system would be setup for that new fragment? (https://docs.microsoft.com/en-us/mixed-reality/world-locking-tools/documentation/howtos/usingwlt/persistencetricks#a-walkthrough) What seems to be happening is that the unrecognized location appears to cause confusion when WLT tries to fit the anchors from the old fragment into the new one. The current fragment id is shown as "1" no matter the room, which I guess is the root of the problem. The only solution I've found is to reset all spatial data from the Hololens and restart. But this means that I am never able to run my project in more than one different physical location without resetting all mapping before hand, which makes me unable to make meaningful use of any persistence.

The "Library not Initialized" error means that FrozenWorld_Init was not called.
Check that your remoting session has created a new plugin in the WorldLockingManager here:
https://github.com/microsoft/MixedReality-WorldLockingTools-Unity/blob/204e507df8e7a1b6e53a4f95cfa3710655ec5945/Assets/WorldLocking.Core/Scripts/WorldLockingManager.cs#L786
And that you have successfully disposed of the previous references here when a remoting session ends:
https://github.com/microsoft/MixedReality-WorldLockingTools-Unity/blob/ad94e0cd48fe70431439e71b16d5c4f02d16db4d/Assets/WorldLocking.Engine/Plugin.cs#L72


Deleting all but one recipe in a subfloor results in inconsistent view

Problem Description
Deleting all but the topmost recipe in a level 2 subfloor results in the recipes/subfloors below this subfloor in the level 1 list being shown in the level 2 subfloor list, until you return to level 1.
The expected behavior is either (like previous version) we remain on level 2 subfloor and see only the top recipe, or maybe we are returned to level 1, but not to see an inconsistent mix of both.
Reproduction
Factorio 2.0.10
Factory Planner 2.0.2

Create a floor with three recipes: iron chest, transport belt, inserter
Create a subfloor on the transport belt and add iron gear recipe
Delete iron gear recipe
Observe that the UI remains on "Level 2" and it displays two recipes: transport belt and inserter.

Thanks for the report, this was indeed just a bug, fixed for the next release.


how to use in reverse direction means right to left

    if getCurrentLanguage()=="ar"{
        prestangeInsideView.transform = CGAffineTransform(scaleX: -1, y: 1);
    }


Add better component decoder - fixes #98

This is another approach in order to solve #98 and PR #97. It uses decode-uri-component which doesn't throw an error when it fails to decode. It tries to decode as much from the input as possible. Most of the time, it uses the native decodeURIComponent. Only when that one fails (edge cases for instance), a custom decoder is being used.
Should we document this somewhere?

Fixes #98
Fixes #97

Yeah, can you add a short note at the bottom of the .parse() docs definition.

I updated the docs. I came to realise though that this is actually a breaking change.

Coolio

Nice! :)


[Backport styhead-next] 2024-10-02_01-35-16_master-next_aws-cli

Description
Backport of #9775 to styhead-next.

Backport failed for kirkstone-next, because it was unable to cherry-pick the commit(s).
Please cherry-pick the changes locally.
git fetch origin kirkstone-next
git worktree add -d .worktree/backport-9783-to-kirkstone-next origin/kirkstone-next
cd .worktree/backport-9783-to-kirkstone-next
git checkout -b backport-9783-to-kirkstone-next
ancref=$(git merge-base a057e508a493e2ec5af3c31f72e5d40fc41bcb13 4ffc92dac81cd2f5c6479488557ddc2d1f546db3)
git cherry-pick -x $ancref..4ffc92dac81cd2f5c6479488557ddc2d1f546db3

Backport failed for scarthgap-next, because it was unable to cherry-pick the commit(s).
Please cherry-pick the changes locally.
git fetch origin scarthgap-next
git worktree add -d .worktree/backport-9783-to-scarthgap-next origin/scarthgap-next
cd .worktree/backport-9783-to-scarthgap-next
git checkout -b backport-9783-to-scarthgap-next
ancref=$(git merge-base a057e508a493e2ec5af3c31f72e5d40fc41bcb13 4ffc92dac81cd2f5c6479488557ddc2d1f546db3)
git cherry-pick -x $ancref..4ffc92dac81cd2f5c6479488557ddc2d1f546db3

Backport failed for styhead-next, because it was unable to cherry-pick the commit(s).
Please cherry-pick the changes locally.
git fetch origin styhead-next
git worktree add -d .worktree/backport-9783-to-styhead-next origin/styhead-next
cd .worktree/backport-9783-to-styhead-next
git checkout -b backport-9783-to-styhead-next
ancref=$(git merge-base a057e508a493e2ec5af3c31f72e5d40fc41bcb13 4ffc92dac81cd2f5c6479488557ddc2d1f546db3)
git cherry-pick -x $ancref..4ffc92dac81cd2f5c6479488557ddc2d1f546db3


Update contentOffset on keyboard open/close

Added priorOffset when keyboard opens and also set self.contentOffset to priorOffset when keyboard closes.

Cheers - issue already fixed with SHA 8914c9a8


✨[feature Request]: Enhancing dark mode functionality

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description

When dark mode functionality is turned on only the text color is changing but background is not changing.
Please assign this issue to me under gssoc-ext, and a suitable level labels.
Use Case
Improve UI
Benefits
No response
Add ScreenShots
No response
Priority
High
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24EXTD contributor
[X] I have starred the repository

Hii @Anjaliavv51 !!
Can you add level tag to it and assign the issue to me so that I can start working on it.;..

@Ankitha2130 go ahead and also @Avansh2006 please create your own issue, which doesn't exisits and tag  me

Please add a suitable level label also

@Ankitha2130 i eill add to pr according to ur changes.


[Bug]: Tales of Rebirth SkitViewer

Version details
No Patch released yet
Contact Details
Pegi
What happened?
The skit npc has a problem with list menus, it seems to have one too many new line which breaks the menu inputs. This is even the case with a blank japanese file.

@Mc-muffin This is still and issue, it only works with the modified psp file Lauren is using.

I thought this was mistakenly closed for a second

Finally fixed


Maybe remove bounds all over

Following #161, I'm opening a separate but similar issue, focused on action items from #161.

Do we want to remove bounds from all the experiments? (done for T1 in #168)
Devise a concrete action item from @oliverdial's comment "set the result to bad if the MLE is outside physical bounds by more than a few error bars".
Are there additional action items coming from the discussion in #161?
Setting milestone to "Release 0.1", please change if this is not the case.

@nkanazawa1989 ramarks (https://github.com/Qiskit/qiskit-experiments/pull/427#discussion_r734563474):

We cannot generally remove the boundary, for example, given we have a fit function A cos (omega x + phi), this could be (A, phi) or (-A, phi+pi). In this situation we may want to set A > 0. How it is implemented in qccode?

For fitters with this sort of ambiguity we do a cleanup pass after the fit to pick a specific branch after the fit is done.  This might be an example of a place where the constraint is more helpful and harmful though.

Stale, closing


Syntax errors fail silently in headless mode

Godot version
4.0 - 4.1.rc3
Issue description
syntax errors fail silently in headless mode.
extends MainLoop

const foo

func _initialize():
    print(foo)

extends MainLoop

xfunc _initialize():
    print(100)

extends MainLoop

func _initialize():
    const arr = [1,2,3,4
    print(arr)

This also seems to fail silently.
extends MainLoop

func _initialize():
asfads

seems like syntax errors fail silently in headless mode.

Probably related: #75545.

Duplicate of #75545.


Provide quick fixes

Provide quick fixes for errors such as:
Error: Unrecognized name: label; Did you mean labels?

https://github.com/minodisk/bigquery-runner/commit/366a5728c4ecfe3a1a8181164c93182f46ecbc14

Released in v1.2.0.


Validators - Generate Keys

Closed in favor of #145

Closing in favor of: https://github.com/gnosischain/documentation/issues/227


Remove obsolte Android plugin

Without comment, I'm not sure, which one is used.
The one from maven, or the local source code one ?

To be honest, I'm confused now.
The main question is:
Do we use Android example as demo using latest released version (this pull request #63), or
do we use Android example to test current plugin implementation by code  ? #65
That's why we have to close one of them, this or #65
But there is a solution to have both, but first we should decide this question, to avoid more confusion

now I see your confusion.
I'd go with #65 without changes in settings.gradle, but adding an explanation to remove this line to have a real world example. I've checked quickly, that Gradle doesn't download a plugin in this case to it's cache, so we can use it in our tests as well.
What's your opinion on this topic?

Do we use Android example as demo using latest released version (this pull request #63), or
do we use Android example to test current plugin implementation by code ? #65

Sorry, I made more confusion, I swapped variant A and B the right question are these:
Do we use Android example as demo using latest released version #65 , or
do we use Android example to test current plugin implementation by code ? (this pull request #63)
Please merge one and close the other one. Independent which version you go

I released, that we can specify another settings.gradle just for testing. I'd create a PR for that


Script to retrieve the template versions of infra stacks and images/tags in those templates

The script Input should be the Rancher server version, e.g. v1.6.2
Minimum set of infra stacks to provide info:
healthcheck
network-services
ipsec
vxlan
scheduler
k8s
Potential Example Script Output:
Rancher Server Version: v1.6.2

network-services:v0.2.1
  - rancher/network-manager:v0.7.1
  - rancher/metadata:v0.9.1
  - rancher/dns:v0.15.1
healthcheck:v0.x.x
  - rancher/healthcheck:v0.x.x

**Notes: **
Pick the default version in config.yml of each template
Branches in Catalog
Rancher v1.6.0 and below use master branch
Rancher v1.6.1-v1.6.xx use v1.6-release branch
Rancher v2.0 will use v2.0-relesae branch

@deniseschannon could you effectively QA this by giving the script a try and seeing if it gives you what you want?

Verified this is working as expected.


added code for delete movie api with TDD

added delete API

Added new java class Constants to add all the constants used in files. Removed /api/v1 from all existing endpoints as these are not required.


Options-first Command and EventListener constructors

Fixes #96. Constructors for Command and EventListener currently take an optional object argument for additional options. However, because these both have required function arguments that can have long bodies, it can hurt readability to have options that impact how the function is run at the very bottom of the constructor, especially when working with single-file commands.
This PR adds support for passing the the object argument before the function argument in both these cases, and deprecates the signatures that put the function argument first. Signatures still exist to allow passing no object argument; both these constructors treat that case identically to passing an empty object.

[x] Command constructor
[ ] CommandWithHelp constructor
[ ] EventListener constructor
[ ] Migrate internal usage in default commands

what the deprecation will say
Deprecation: Object-last Client and EventListener constructor signatures
The `Command` and `EventListener` constructors both take an optional object argument containing additional options. #98 introduced new forms for these constructors, which place the options argument before the function argument, and deprecated the old forms which place the options argument at the end of the argument list.

Deprecated forms can be replaced by simply switching the position of the arguments, like so:

```js
// old
new Command('test', message => {
  // ...
}, {
  guildOnly: true,
  permissions: ['manageMessages'],
})
new EventListener('ready', () => {
  // ...
}, {once: true})

// new
new Command('test', {
  guildOnly: true,
  permissions: ['manageMessages'],
}, message => {
  // ...
})
new EventListener('ready', {once: true}, () => {
  // ...
})
```

Omitting the object argument entirely is still supported:

```js
new Command('ping', message => {
  // ...
})
new EventListener('messageCreate', message => {
  // ...
})
```

The object-last constructor signatures of `Command` and `EventListener` will be removed in the 3.0.0 release.

Blocked on #84, I ain't dealing with that EventEmitter constructor until it's reasonable


Fix for "install nginx admin password" task in consul role

In multi-datacenter deployment task "install nginx admin password" from
consul role should run on both datacenters, but only once, so the task
itself has a flag "run_once" and we separate consul deployment to 2
datacenters

Looks good. Let me roll out a 2-datacenter deploy to test and then I'll merge.


Greenkeeper/eslint import resolver webpack 0.10.0

:tada: This PR is included in version 3.2.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Background jobs consolidation

Currently our background jobs need to be restarted manually for VM reboots and manually deployed after fixes. Make it automatic through Octopus deployments

I will be working on #2415 and #2413 as part of this. Changing the title to reflect it.

#2414

#2336
#2332


scheduled_test/test/scheduled_test/nested_task_test is failing

 Issue by floitschG
Originally opened as dart-lang/sdk#14639

After removing deprecated features scheduled_test/test/scheduled_test/nested_task_test is failing.

 Comment by anders-sandholm

Removed Library-ScheduledTest label.
Added Pkg-ScheduledTest label.

 Comment by nex3

Fixed in dart-lang/sdk@2b7caadbc6051937c7e92172c1708923513d6bc9.

Added Fixed label.


Adapt createDashboard to latest changes in cdb.createvis

As described in https://github.com/CartoDB/cartodb.js/pull/1313, there were some changes to cdb.createVis and code in src/api/create-dashboard.js needed to be changed a bit. Since we're passing the skipMapInstantiation option to cdb.createVis here, we need to wait for the load event before we can explicitly instantiate the map (and bind success and error callbacks).
@viddo I'm all ears! Thanks

https://github.com/CartoDB/deep-insights.js/pull/380/files?w=0 for easier review (the default diff is quote noisy, mostly white-space changes anyway). Nothing to add really 👍


[google_sign_in] Fix Exception leaking in debug mode (#64681)

Description
This is related to flutter/flutter#26705 and many others. I already fixed this before on flutter/flutter#37343 submitting this PR https://github.com/flutter/plugins/pull/1937.
I am fixing this again.
Could the owners of these files make some sort of mechanism to avoid this from happening again?

I need some help with the tests, I honestly don't understand why the 'can sign in after init failed before' test is failing. For some reason the MockMethodCallHandler is only being called once. I didn't change anything on this test (just syntactic sugar, but it also fails if I remove it).
Related Issues
Fixes flutter/flutter#64681
Fixes flutter/flutter#45146
flutter/flutter#37343
flutter/flutter#26705
dart-lang/sdk#37268
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[ ] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

@mehmetf could you help me figure out what's the problem with the tests?

All of the referenced issues are closed, referring to an issue that was recently fixed in the Dart VM. Given that, is this PR still necessary?

Thank you for your contribution. I'm going to close this PR for now since there are outstanding comments, just to get this off our PR review queue. Please don't hesitate to submit a new PR if you have the time to address the review comments. Thanks!


windows listview Axis.horizontal not work

Steps to Reproduce

Execute flutter run on the code sample

// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'package:flutter/material.dart';
import 'package:flutter/widgets.dart';

void main() =>
  runApp(
    AA(),
    );
class AA extends StatelessWidget{
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body:  Container(
        decoration: BoxDecoration(
          borderRadius: BorderRadius.all(
            Radius.circular(
              10,
            ),
          ),

        ),
        height:243,

        child: ListView.builder(
            scrollDirection: Axis.horizontal,
            itemCount: 20,
            padding: EdgeInsets.zero,
            itemBuilder: (context, index) {
              return Container(width: 200,height: 100,color: Colors.yellow,margin: EdgeInsets.symmetric(horizontal: flexible(context, 20)),);

            }),
      ),
    );
  }

}

Expected results:Can slide
Actual results: not slide
$ flutter doctor -v
[√] Flutter (Channel stable, 2.5.3, on Microsoft Windows [Version 10.0.19043.1237], locale zh-CN)
    • Flutter version 2.5.3 at D:\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (6 weeks ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[√] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at C:\Users\dansun\AppData\Local\Android\Sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME =  C:\Users\dansun\AppData\Local\Android\sdk
    • Java binary at: C:/Program Files/Android/Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Professional 2022 17.0.0 RC3)
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Professional
    • Visual Studio Professional 2022 version 17.0.31825.309
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 2020.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[√] Android Studio
    • Android Studio at C:/Program Files/Android/Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = C:/Program Files/Android/Android Studio
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19043.1237]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 96.0.4664.45
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 95.0.1020.44

Hi @dansunhappy, by sliding not working are you referring to the drag behaviour not working?


Multiple display styles of documents

1、

2、

3、

4、

@Avey777
The new version v0.7.1 with multiple display styles has been published. Hope you can try it.


race: not working with Alpine based image

Hi,
With the following docker image (save it as demo.docker)
FROM golang:1.6.0-alpine
MAINTAINER<EMAIL_ADDRESS>
RUN apk add --update alpine-sdk \
    && rm -rf /var/cache/apk/*

run
docker build -f demo.docker -t race/demo .

This will fail with:
# runtime/race
race_linux_amd64.syso: In function `__sanitizer::InternalAlloc(unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul,<PHONE_NUMBER>55328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x1681): undefined reference to `__libc_malloc'
race_linux_amd64.syso: In function `__sanitizer::ReExec()':
gotsan.cc:(.text+0xd937): undefined reference to `__libc_stack_end'
race_linux_amd64.syso: In function `__sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul,<PHONE_NUMBER>55328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x5ec8): undefined reference to `__libc_free'
collect2: error: ld returned 1 exit status

If you then disable CGO and run again:
PROJECT_DIR="${PWD}" #assume we are in $GOPATH/src/github.com/dlsniper/demo on the computer
CONTAINER_PROJECT_DIR="/go/src/github.com/dlsniper/demo"
CONTAINER_PROJECT_GOPATH="${CONTAINER_PROJECT_DIR}/vendor:/go"

docker run --rm \
        --net="host" \
        -v ${PROJECT_DIR}:${CONTAINER_PROJECT_DIR} \
        -e CI=true \
        -e GODEBUG=netdns=go \
        -e CGO_ENABLED=0 \
        -e GOPATH=${CONTAINER_PROJECT_GOPATH} \
        -w "${CONTAINER_PROJECT_DIR}" \
        race/demo \
        go test -v -race ./...

It will result in the following output:
go test: -race requires cgo; enable cgo by setting CGO_ENABLED=1

Previously, in go 1.5.3, when running with CGO disabled, this used to fail with:
# testmain
runtime/race(.text): __libc_malloc: not defined
runtime/race(.text): getuid: not defined
runtime/race(.text): pthread_self: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): sleep: not defined
runtime/race(.text): usleep: not defined
runtime/race(.text): abort: not defined
runtime/race(.text): isatty: not defined
runtime/race(.text): __libc_free: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): pipe: not defined
runtime/race(.text): __libc_stack_end: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): exit: not defined
runtime/race(.text.unlikely): __errno_location: not defined
runtime/race(.text): undefined: __libc_malloc
/usr/local/go/pkg/tool/linux_amd64/link: too many errors

To test this just change the base image for the container.
Please let me know if there are any additional details I can share.
Thank you.

CC @dvyukov
As far as I know this is because race.syso assumes a glibc based system.  I don't know that there is a simple fix here.

I can't think of any simple fix. A complex fix would be to remove all dependencies on libc from race runtime (there is an open issue for that).
alpinelinux wiki suggests that there are some ways to run glibc-based programs on alpine:
http://wiki.alpinelinux.org/wiki/Running_glibc_programs
Don't know whether it will help for race detector or not.

hi, all,
just wondering. does anyone have a complete working example of a golang program working with -race ideally not that much different from the official Alpine-based build.
out of interest, are the flags not tested? i would imagine that this sort of issue would be picked up at compile time, unless the Alpine-based is configured to to include it... no idea, just guessing.
anywho, working sample would be much appreciated.

I have managed to build a race_linux_amd64.syso that works on Alpine. Here are the necessary changes: https://github.com/neelance/compiler-rt/commit/32aa655a999d33a4034ec4aea00ed3df1f9e77df I haven't verified that it still works on other Linux platforms.
To build on Alpine:

Clone https://github.com/neelance/compiler-rt/
Go into /compiler-rt/lib/tsan/go/
./buildgo.sh (The test step crashes, but the library works anyways. My guess is that the test does not use a proper Go environment.)
Copy race_linux_amd64.syso to $GOPATH/src/runtime/race/

I'm not sure how to continue from here. Any suggestions on getting that upstream?

Here are instructions on how to contribute to sanitizers:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToContribute
I skimmed through your patch and I think we can upstream something like this. But we need some testing story, otherwise it will break in future.
One good test might be to whitelist all libc symbol dependencies in buildgo.sh. This way we can ensure that we won't silently add a new dependency (ideally in future this list becomes empty).

@dvyukov I see that you did most of the updates of the syso files. I have some more questions:

How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?
Can we somehow add Alpine to the platforms being tested?
We need to fix that crash on the test phase of buildgo.sh. I've investigated it a bit, seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout, but the test is a C program. That was as far as I could get without spending a massive amount of time to understand how that thread sanitizer works.

How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?

I now use golang.org/x/build/cmd/racebuild which builds and tests race runtime for all platforms. But it requires gomote access (I think you need to be a committer for Go repo). If you don't have that access, I will do testing. But test at least on Alpine and on glibc-based Linux.

Can we somehow add Alpine to the platforms being tested?

You need to either run your own Go builder and connect it to dashboard, or we need to setup Alpine linux builder on GCE. @bradfitz, do we want to setup/maintain it?

seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout

It's possible. Does removing -fPIC -fpie from buildgo.sh help? We still need to build the runtime with -pie, but the test itself does not need to be pie.

#17891 tracks adding an Alpine builder. @jessfraz has a (stalled :)) CL in progress.

We now have Alpine builders, so in theory golang.org/x/build/cmd/racebuild could build an Alpine race binary now.
For now I'm just going to be skipping race tests on Alpine so we don't regress on other things.

What is the latest update on this issue?

@djui, there is no update. Nobody on the Go team is working on this, and nobody elsewhere in the community has posted anything here about it. It's all yours if you want to work on it.

Hi @bradfitz I think it's ok to close this issue. It's not fair to leave it open as an issue if it's clear by now this does not work by design without further work.
As you said, if anyone wants to pick it up, great!
For 👎 'ers: Would you agree?

@djui why should the issue be closed if it's still an issue, even if it doesn't have anyone currently working on it? By that logic, most of the current issues should be closed, no?

This caused a moderate amount of pain at my job, as most CI build/test steps happen on Alpine-based images. We had to specifically pull in Debian-based images just for the go test -race steps. So I'd say we do want to keep the issue open, even if there aren't short-term plans to get it done.

@dlsniper My reasoning is focusing on the category:"Is it a bug or a feature" rather than:"If feature, is a requested feature implemented by now." To me, the issue was opened in spirit of a bug, imho, and it became during the conversation that no promises or constraints are violated, thus more of a feature (getting down to: "Should any C stdlib be supported by Go?" or "Is Musl a supported 'arch'?").  I may misclassify this and indeed any of these should be supported, therefore the question for feedback.
So maybe we could open a new ticket, link this ticket, with e.g. race: Make compiling and running race detector work with Musl. wdyt?
@mvdan Understandably, same here.

So maybe we could open a new ticket, link this ticket, with e.g. race: Make compiling and running race detector work with Musl. wdyt?

I can change the title of this issue but I don't think that will speed up the fix for it. Unfortunately, I don't have the time or the knowledge to work on this and it seems that nobody else from all the referenced issues has any plans on fixing this either. However, it seems that people find this issue and it's easy for them to link to it so may I suggest to leave this like they are until someone figures how to fix this? Thank you.

@dlsniper Works for me, Florin.

What does Go 1.11 have to do with this? It never worked on any version.

Looks like maybe when we did go upgrade we might have changed the OS type (need to find that out internally).
Any idea when I should expect a fix for the issue?

@jgheewala as an intermediate suggestion: if you can tolerate the trade-offs of the suggestion described below and your build/CI/CD pipeline supports to use Docker, you could run your tests in a Debian/CentOS/... image running with race-detector but ship/deploy in a non-race detector Alpine image.

ok will do thanks a lot @djui

In case it helps anyone else, I rebased @neelance's compiler-rt changes (https://github.com/golang/go/issues/14481#issuecomment-281972886) on top of compiler-rt@fe2c72c5 (as used by Go 1.13) and have been able to run -race using:
FROM golang:1.13-alpine3.10 as golang

# Make go test -race work on alpine by building (patched) sanitizer manually
# as it is not built by default
# Ref: https://github.com/golang/go/issues/14481#issuecomment-281972886
# SHA: https://github.com/golang/go/blob/go1.13/src/runtime/race/README
COPY 0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch /race.patch
RUN cd / \
    && apk add --no-cache --virtual .build-deps g++ git \
    && mkdir -p compiler-rt \
    && git clone https://llvm.org/git/compiler-rt.git \
    && cd compiler-rt \
    && git reset --hard fe2c72c59aa7f4afa45e3f65a5d16a374b6cce26 \
    && patch -p1 -i /race.patch \
    && cd lib/tsan/go \
    && ./buildgo.sh 2>/dev/null \
    && cp -v race_linux_amd64.syso /usr/local/go/src/runtime/race/ \
    && rm -rf /compiler-rt /race.patch \
    && apk del .build-deps

0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz

how to solve the problem?
runtime/race /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function  __sanitizer::GetArgv()': gotsan.cc:(.text+0x4183): undefined reference to __libc_stack_end' /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function  __sanitizer::ReExec()': gotsan.cc:(.text+0x9797): undefined reference to __libc_stack_end' /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld:

If you use Docker, I have an example with a multi stage build here where the fix mentioned above is built and then overwrites the existing file in the final image.

The fix is submitted to llvm.
We now have 2 reasons to update race runtime: this and the chan synchronization change. @randall77

The last change to http://llvm.org/git/compiler-rt.git is from Oct 22, 2019. So I don't think it contains a recently submitted fix.
Is there somewhere else I should be pulling from? A branch other than master, perhaps?

llvm has moved to gihub: https://github.com/llvm/llvm-project

Current status:
linux/amd64 - working
darwin/amd64 - working
windows/amd64 - working
freebsd/amd64 - Build script fails with "race_freebsd_syso seems to link to libc". Well, yes. Why is that a problem?
netbsd/amd64 - "no space left on device". It can't git clone the llvm-project repository without running out of memory. It's ~2GB. I'll see if we can get a larger disk on the gomotes.
linux/ppc64le - killed during fetch of llvm-project. Also disk space?
linux/arm64 - download & build fine. Tests during race.bash crash randomly.

linux/arm64 - fixed some tests. Now dying because it runs out of OS threads when building various cmd/ binaries.

freebsd/amd64 - Build script fails with "race_freebsd_syso seems to link to libc". Well, yes. Why is that a problem?

This should help:
https://github.com/llvm/llvm-project/commit/0fb8a5356214c47bbb832e89fbb3da1c755eeb73

linux/arm64 - fixed some tests. Now dying because it runs out of OS threads when building various cmd/ binaries.

Perhaps a simplest fix is to export GOMAXPROCS= (or is there some other knob to reduce parallelism?).

linux/arm64: GOMAXPROCS=1 and GOGC=10 almost fixes it. Not quite. It still crashes trying to compile the tests for cmd/compile/internal/ssa.
I think we need a beefier (more ram) builder.
The .syso might be ok though. Maybe I can extract that early.

The .syso might be ok though. Maybe I can extract that early.

Good idea. We could save it earlier. It will be tested during the change presubmit again anyway.

freebsd/amd64 now working with the libc patch.
linux/arm64 also working, hacked around by extracting the .syso early
That just leaves netbsd/amd64 and linux/ppc64le with one definite and one probable disk space issue. I'm pursuing that, may take a while.
I'll prepare the 5 working platforms in a CL.

Humm... how did we get the previous syso files for ppc?...
Generally the way to get rid of these is to #ifdef !SANITIZER_GO CheckASLR. I don't think we need that for Go, but we inherit everything for native world by default...
If there would be a way to strip an object file from everything that's not reachable from few interface functions, that would be nice and probably will make syso files even smaller. But I don't know how to do this easily.
Every once in a while I am thinking about writing a new tsan runtime fork in Go for Go. There is also a next gen algorithm that I think could reduce memory consumption few times and make it faster and remove restriction on number of goroutines and all the nice things. But I can never fully convince myself that it's the right thing to do... or maybe just find time.

Humm... how did we get the previous syso files for ppc?...

Something changed since last fall. It wasn't that code itself, maybe someone turned ASLR on for ppc64le in some config somewhere?

Every once in a while I am thinking about writing a new tsan runtime fork in Go for Go. There is also a next gen algorithm that I think could reduce memory consumption few times and make it faster and remove restriction on number of goroutines and all the nice things. But I can never fully convince myself that it's the right thing to do... or maybe just find time.

It would be nice. Pulling new tsan always seems to be a headache. But rewriting tsan in Go is probably a lot of work, many times the aforementioned headache. Probably not worth it.
Unless you were locked in a room with nothing else to do for a month or so...

Something changed since last fall. It wasn't that code itself, maybe someone turned ASLR on for ppc64le in some config somewhere?

Humm... yes, it seems this bit of code may actually be needed:
#elif SANITIZER_PPC64V2
  // Disable ASLR for Linux PPC64LE.
  int old_personality = personality(0xffffffff);
  if (old_personality != -1 && (old_personality & ADDR_NO_RANDOMIZE) == 0) {
    VReport(1, "WARNING: Program is being run with address space layout "
               "randomization (ASLR) enabled which prevents the thread and "
               "memory sanitizers from working on powerpc64le.\n"
               "ASLR will be disabled and the program re-executed.\n");
    CHECK_NE(personality(old_personality | ADDR_NO_RANDOMIZE), -1);
    ReExec();
  }

Pulling in ReExec may transitively pull in lots of other other things (as you noted already).
Maybe we could do what freebsd does above if SANITIZER_GO to at least warn users:
  if (UNLIKELY(paxflags & CTL_PROC_PAXFLAGS_ASLR)) {
    Printf("This sanitizer is not compatible with enabled ASLR\n");
    Die();
  }

But I don't know if users can actually turn ASLR off if needed.
Potentially ASLR does not affect static Go binaries at all (?). But maybe it's a problem with cgo.
Hard to say what's best without testing...

It would be nice. Pulling new tsan always seems to be a headache. But rewriting tsan in Go is probably a lot of work, many times the aforementioned headache. Probably not worth it.
Unless you were locked in a room with nothing else to do for a month or so...

Yes, that's what I thought as well.
I am actually locked in a room at the moment,  but not with nothing else to do :)

What is the next-gen algorithm?

What is the next-gen algorithm?

It's not described anywhere.

Hello all, a big thank you first of all for the work fixing this issue.
Do you have an approximate time when the Go Docker image will contain the fix? I just tried go test -race using the latest golang:1.14-alpine3.11 and it still fails. Thanks!
Or, if it will take quite some time, is there a way for us to manually build and bundle something in the docker image to have it fixed using your latest changes?

@qdm12 this was merged in master, so it will be part of the upcoming Go 1.15 release due in July, unless it gets reverted.
Only important bug fixes are generally backported to stable Go releases. See https://github.com/golang/go/wiki/MinorReleases.
In particular, while this change is awesome and I'd also like to use it today, the change is probably too invasive and risky to backport to a release like 1.14.3. The first 1.15 beta should be out in just six weeks, so you could try the docker images that come out with that release, too.

@qdm12 We are using
#!/bin/sh
set -eu
set -x

cd /

apk --no-cache add build-base git

# Clone the repo and reset to the right commit
git clone https://git.llvm.org/git/compiler-rt.git /x
cd /x

#https://github.com/golang/go/blob/release-branch.go1.14/src/runtime/race/README
git checkout 810ae8ddac890a6613d814c0b5415c7fcb7f5cca

# Apply the patch and do the build
patch -p1 <<'EOF'
commit 51e2fa48690be675363fbd2b62d775715749f58d
Author: Tomas Volf<EMAIL_ADDRESS>Date:   Thu Mar 5 23:43:38 2020 +0100

    Ported musl compatibility patch

diff --git a/lib/sanitizer_common/sanitizer_allocator.cpp b/lib/sanitizer_common/sanitizer_allocator.cpp
index 8d07906cc..e4681f77e 100644
--- a/lib/sanitizer_common/sanitizer_allocator.cpp
+++ b/lib/sanitizer_common/sanitizer_allocator.cpp
@@ -25,7 +25,7 @@ const char *PrimaryAllocatorName = "SizeClassAllocator";
 const char *SecondaryAllocatorName = "LargeMmapAllocator";

 // ThreadSanitizer for Go uses libc malloc/free.
-#if SANITIZER_GO || defined(SANITIZER_USE_MALLOC)
+#if defined(SANITIZER_USE_MALLOC)
 # if SANITIZER_LINUX && !SANITIZER_ANDROID
 extern "C" void *__libc_malloc(uptr size);
 #  if !SANITIZER_GO
diff --git a/lib/sanitizer_common/sanitizer_common.cpp b/lib/sanitizer_common/sanitizer_common.cpp
index f5f9f49d8..87efda5bd 100644
--- a/lib/sanitizer_common/sanitizer_common.cpp
+++ b/lib/sanitizer_common/sanitizer_common.cpp
@@ -274,6 +274,7 @@ uptr ReadBinaryNameCached(/*out*/char *buf, uptr buf_len) {
   return name_len;
 }

+#if !SANITIZER_GO
 void PrintCmdline() {
   char **argv = GetArgv();
   if (!argv) return;
@@ -282,6 +283,7 @@ void PrintCmdline() {
     Printf("%s ", argv[i]);
   Printf("\n\n");
 }
+#endif

 // Malloc hooks.
 static const int kMaxMallocFreeHooks = 5;
diff --git a/lib/sanitizer_common/sanitizer_linux.cpp b/lib/sanitizer_common/sanitizer_linux.cpp
index 0b53da6c3..ac3152eb8 100644
--- a/lib/sanitizer_common/sanitizer_linux.cpp
+++ b/lib/sanitizer_common/sanitizer_linux.cpp
@@ -26,7 +26,7 @@
 #include "sanitizer_placement_new.h"
 #include "sanitizer_procmaps.h"

-#if SANITIZER_LINUX
+#if SANITIZER_LINUX && !SANITIZER_GO
 #include <asm/param.h>
 #endif

@@ -549,7 +549,7 @@ const char *GetEnv(const char *name) {
 #endif
 }

-#if !SANITIZER_FREEBSD && !SANITIZER_NETBSD && !SANITIZER_OPENBSD
+#if !SANITIZER_FREEBSD && !SANITIZER_NETBSD && !SANITIZER_OPENBSD && !SANITIZER_GO
 extern "C" {
 SANITIZER_WEAK_ATTRIBUTE extern void *__libc_stack_end;
 }
@@ -581,7 +581,7 @@ static void ReadNullSepFileToArray(const char *path, char ***arr,
 }
 #endif

-#if !SANITIZER_OPENBSD
+#if !SANITIZER_OPENBSD && !SANITIZER_GO
 static void GetArgsAndEnv(char ***argv, char ***envp) {
 #if SANITIZER_FREEBSD
   // On FreeBSD, retrieving the argument and environment arrays is done via the
@@ -1060,7 +1060,7 @@ uptr GetMaxUserVirtualAddress() {

 #if !SANITIZER_ANDROID
 uptr GetPageSize() {
-#if SANITIZER_LINUX && (defined(__x86_64__) || defined(__i386__))
+#if SANITIZER_LINUX && defined(EXEC_PAGESIZE)
   return EXEC_PAGESIZE;
 #elif SANITIZER_FREEBSD || SANITIZER_NETBSD
 // Use sysctl as sysconf can trigger interceptors internally.
diff --git a/lib/sanitizer_common/sanitizer_linux_libcdep.cpp b/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
index cd5037182..4e5c04818 100644
--- a/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
+++ b/lib/sanitizer_common/sanitizer_linux_libcdep.cpp
@@ -808,7 +808,7 @@ u64 MonotonicNanoTime() {
 }
 #endif  // SANITIZER_LINUX && !SANITIZER_GO

-#if !SANITIZER_OPENBSD
+#if !SANITIZER_OPENBSD && !SANITIZER_GO
 void ReExec() {
   const char *pathname = "/proc/self/exe";

diff --git a/lib/tsan/rtl/tsan_platform_linux.cpp b/lib/tsan/rtl/tsan_platform_linux.cpp
index 33fa586ca..4c871ec58 100644
--- a/lib/tsan/rtl/tsan_platform_linux.cpp
+++ b/lib/tsan/rtl/tsan_platform_linux.cpp
@@ -62,7 +62,7 @@
 # undef sa_sigaction
 #endif

-#if SANITIZER_FREEBSD
+#if SANITIZER_FREEBSD && !SANITIZER_GO
 extern "C" void *__libc_stack_end;
 void *__libc_stack_end = 0;
 #endif
EOF

cd lib/tsan/go
./buildgo.sh

install -Dt /out/usr/local/go/src/runtime/race/ race_linux_amd64.syso

and then copy it over the original one
COPY --from=compiler-rt       /out /

Or you can just wait for the 1.15 :)

Okay thanks for letting us know :+1: I noticed there is a golang:latest, is this based on the master branch? If so, is there one based on the master branch and alpine perhaps? Anyway, stability/production wise, I'd stick with the release in July :wink:

@qdm12 this was merged in master, so it will be part of the upcoming Go 1.15 release due in July, unless it gets reverted.
Only important bug fixes are generally backported to stable Go releases. See https://github.com/golang/go/wiki/MinorReleases.
In particular, while this change is awesome and I'd also like to use it today, the change is probably too invasive and risky to backport to a release like 1.14.3. The first 1.15 beta should be out in just six weeks, so you could try the docker images that come out with that release, too.

Is this confirmed working in golang 1.15? I don't see anything in the release notes, and my google search didn't turn up anything helpful. Thanks!

Yes, we are using it successfully with the (rolling) 1.15-alpine tag.

I can confirm it works on golang:1.15-alpine, although not on golang:1.15-alpine3.12 for example.
Good job!

For those forced to use an older image, I added
RUN apk add --update --no-cache alpine-sdk
to make it work. Seems similar to this gist: https://gist.github.com/miry/fece267c7faba904c360


Failed at the<EMAIL_ADDRESS>install script

Facing this issue while trying to install as root
sh-3.2# npm install -g puppeteer

<EMAIL_ADDRESS>install /usr/local/lib/node_modules/puppeteer
node install.js

ERROR: Failed to set up Chromium r818858! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
[Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/puppeteer/.local-chromium'] {
errno: -13,
code: 'EACCES',
syscall: 'mkdir',
path: '/usr/local/lib/node_modules/puppeteer/.local-chromium'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: node install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /var/root/.npm/_logs/2020-11-19T05_52_30_535Z-debug.log

you have no write permission, the 'mkdir' is not working
you can try install with --save option, once you have a local npm project, which you get with 'npm init'

the problem may be caused by your internet can't download the newest Chromium，so you can use
npm install puppeteer --ignore-scripts  to skip download
and then，when you use the script ，you should set the executablePath like
const path = require('path');
// the browser path
const path = path.resolve('C:\Program Files\Google\Chrome\Application\chrome.exe')

const browser = await puppeteer.launch({
    executablePath: path
})

Thanks guys issue is fixed now.

thanks, @xxg233 I have the same issue now fixed.

the problem may be caused by your internet can't download the newest Chromium，so you can use
npm install puppeteer --ignore-scripts to skip download
and then，when you use the script ，you should set the executablePath like
const path = require('path');
// the browser path
const path = path.resolve('C:\Program Files\Google\Chrome\Application\chrome.exe')

const browser = await puppeteer.launch({
    executablePath: path
})

Hello
Thank you for your solution. I did the first command (npm install puppeteer --ignore-scripts) and it worked well. But I do not know how to run this code/command. Could you please explain more? I am not familiar with that.

the problem may be caused by your internet can't download the newest Chromium，so you can use
npm install puppeteer --ignore-scripts to skip download
and then，when you use the script ，you should set the executablePath like
const path = require('path');
// the browser path
const path = path.resolve('C:\Program Files\Google\Chrome\Application\chrome.exe')

const browser = await puppeteer.launch({
    executablePath: path
})

Hello
Thank you for your solution. I did the first command (npm install puppeteer --ignore-scripts) and it worked well. But I do not know how to run this code/command. Could you please explain more? I am not familiar with that.


Adding /O=system:masters to admin certificate

Issue #1125. Make RBAC authorization plugin work out of the box.
"When bootstrapping, superuser credentials should include the system:masters group, for example by creating a client cert with /O=system:masters. This gives those credentials full access to the API and allows an admin to then set up bindings for other users."

ci check this

I signed up the CLA

Re-kicking ci b/c it failed with a random error.

should we also add /O=system:nodes to nodes certificate ?
got lots of errors like this:
I0614 09:50:37.538099       1 rbac.go:87] RBAC DENY: user "kube-node-test132s2" groups [system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 09:50:37.538306       1 rbac.go:87] RBAC DENY: user "kube-node-test134s2" groups [system:authenticated] cannot "list" resource "pods" cluster-wide
I0614 09:50:37.538605       1 rbac.go:87] RBAC DENY: user "kube-node-test132s3" groups [system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 09:50:37.539282       1 rbac.go:87] RBAC DENY: user "kube-node-test132s3" groups [system:authenticated] cannot "list" resource "services" cluster-wide
I0614 09:50:37.539847       1 rbac.go:87] RBAC DENY: user "kube-node-test132s3" groups [system:authenticated] cannot "list" resource "services" cluster-wide
I0614 09:50:37.540981       1 rbac.go:87] RBAC DENY: user "kube-node-test132s3" groups [system:authenticated] cannot "list" resource "pods" cluster-wide
I0614 09:50:37.542211       1 rbac.go:87] RBAC DENY: user "kube-node-test132s3" groups [system:authenticated] cannot "list" resource "nodes" cluster-wide

still got similar error when /O=system:nodes added to nodes certificate

I0614 10:39:15.781192       1 rbac.go:87] RBAC DENY: user "system:node:kube-node-test134s1" groups [system:nodes system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 10:39:15.781240       1 wrap.go:75] GET /api/v1/endpoints?resourceVersion=0: (283.366µs) 403 [[hyperkube/v1.6.4+coreos.0 (linux/amd64) kubernetes/8996efd] <IP_ADDRESS>:34075]
I0614 10:39:15.889356       1 rbac.go:87] RBAC DENY: user "system:node:kube-node-test134s2" groups [system:nodes system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 10:39:15.889396       1 wrap.go:75] GET /api/v1/endpoints?resourceVersion=0: (274.358µs) 403 [[hyperkube/v1.6.4+coreos.0 (linux/amd64) kubernetes/8996efd] <IP_ADDRESS>:28741]
I0614 10:39:15.948654       1 rbac.go:87] RBAC DENY: user "system:node:kube-node-test131s2" groups [system:nodes system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 10:39:15.948700       1 wrap.go:75] GET /api/v1/endpoints?resourceVersion=0: (277.308µs) 403 [[hyperkube/v1.6.4+coreos.0 (linux/amd64) kubernetes/8996efd] <IP_ADDRESS>:33939]
I0614 10:39:15.949330       1 rbac.go:87] RBAC DENY: user "system:node:kube-node-test131s3" groups [system:nodes system:authenticated] cannot "list" resource "endpoints" cluster-wide
I0614 10:39:15.949394       1 wrap.go:75] GET /api/v1/endpoints?resourceVersion=0: (320.574µs) 403 [[hyperkube/v1.6.4+coreos.0 (linux/amd64) kubernetes/8996efd] <IP_ADDRESS>:36388]

i found these RBAC DENY for kube-node on the same physical machine as the kube-master,
but kubelet use <IP_ADDRESS>:8080 if on the same machine with kube-master, and use kube-master[0]:8443 if on different machine.

seems to be caused by kube-proxy, but this file api.go is deleted in latest source code:

E0615 05:45:08.256402       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: User "system:node:kube-node-test134s2" cannot list endpoints at the cluster scope. (get endpoints)
E0615 05:45:09.258716       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: User "system:node:kube-node-test134s2" cannot list endpoints at the cluster scope. (get endpoints)
E0615 05:45:10.260443       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: User "system:node:kube-node-test134s2" cannot list endpoints at the cluster scope. (get endpoints)

if you get here, please see #1351


ClientData from dict?

How about creating ClientData from a dict, and having a from_json to decode from a str?
https://github.com/Yubico/python-fido2/blob/492eac92201cef9929f47f3e9eeb95aef056baf7/fido2/client.py#L45
(In my case, for example, the api layer already takes care about json decoding)

The reason that ClientData requires bytes (or base64 encoded data) is that the exact binary serialization used is important to maintain the sha256 digest. De-serializing this data, and then re-serializing it to get the raw bytes has a large chance of introducing changes in the data (for example the order of the fields, any whitespace changes, and so on), which would cause the digest to no longer match, and the signature validation to fail. For this reason, we don't allow it.

Is that also true for AuthenticatorData, AttestedCredentialData and other structures?

Yes, the binary representations of these are signed during attestation.

De-serializing this data, and then re-serializing it to get the raw bytes has a large chance of introducing changes in the data (for example the order of the fields, any whitespace changes, and so on), which would cause the digest to no longer match, and the signature validation to fail. For this reason, we don't allow it.

I don't think, you can prevent that. As the AuthenticatorData and other objects have serialization method create and provide deserialized data, such as rp_id_hash, flags, etc. Anybody can extract they individually, serialize and the deserialize and create the object again.

Yeah, maybe the wrong choice of wording on my part. What I meant was that where possible, we should make it hard for a consumer of the library to accidentally mess things up. The example of re-creating the ClientData from a dict which was previously deserialized from JSON is a prime example of this, as it may result in a different byte representation for the ClientData, which woulc cause the server to reject the operation. The only clue as to what was happening would be an "Invalid signature" error, so debugging the issue could be difficult.


Michel pi master

Hey 😃
Did you close your PR because I did not answer within 2 days?
We could talk about the changes. Besides those regarding the changes of the framework targets and the addition of Fody.
Those are no go.
Your PR could be a bit more explaining what and why you did what you did.
Can not merge https://github.com/michel-pi/GameOverlay.Net/pull/85/files#diff-f31c97ef8a0b1944b2c4fd9169dfa7845b8bc1937c7579c9d32ad3783fa9748eR148 since those properties do not reflect the current windows state when changed after init.


Confusing handling of named arguments

Enso reconciles currying, named arguments and argument default values. This however introduces some trade-offs.
TL;DR
There is a specification of named arguments behavior and to quote it:

Named arguments are applied in the order they are given.
This means that if you positionally apply to an argument foo and
then try to later apply to it by name, this will fail due to currying of functions.

@JaroslavTulach interprets that as a confirmation the current behavior is correct and all we can do is to improve error reporting as Radek concludes too.
Original Report
Here I encountered one such issue where the various trade-offs make the behaviour very confusing to the user.
from Standard.Base import all

fun a b c =
    IO.println "a = "+a.to_display_text
    IO.println "b = "+b.to_display_text
    IO.println "c = "+c.to_display_text
    a+b+c

main =
    IO.println '\n1:'
    IO.println <| fun "A" "B" "C"
    IO.println '\n2:'
    IO.println <| fun  b="B" "A" "C"
    IO.println '\n3:'
    IO.println <| fun "A" "C" b="B"
    IO.println '\n4:'
    IO.println <| Panic.recover Any <| (fun "A" "C" b="B").length
    IO.println '\n5:'
    IO.println <| fun "A" "C" b="B" "D"

yields us:
1:
a = A
b = B
c = C
ABC

2:
a = A
b = B
c = C
ABC

3:
named-args.fun[named-args.enso:3-22]

4:
(Error: (No_Such_Method.Error named-args.fun[named-args.enso:3-22] UnresolvedSymbol<length>))

5:
a = A
b = C
c = D
Execution finished with an error: Type error: expected a function, but got ACD.
        at <enso> named-args.main<arg-1>(named-args.enso:19:19-39)
        at <enso> named-args.main(named-args.enso:19:5-39)

Let's unpack what is happening here.

We just provide all arguments positionally - all is well.
We provide the b argument as named - it is correctly reordered.
We provide the 3 arguments, but now the named b argument comes last. This is the most interesting example showing the issue. Due to how our argument resolution works (and this is not fully avoidable if we want to support currying, at least not without some careful limitations), what the system sees: I see argument "A", I pass it to parameter a, then I see "C", I've got a function waiting for an argument named b - OK all looks good - let's take it. Then I see a named argument b="B". OK but I have a function taking a parameter c only - no match - the resolver "suspends" this argument, hoping that once more arguments are provided, maybe our function will return a new function taking more arguments and one of its names will match my pending b? Maybe. For now I return a value containing a Function waiting for one parameter - that is what is printed. Note that this 'suspended pending argument' b="B" is hidden and it is impossible to see that we are waiting for this argument.
Actually I want to come to this in a moment, below.
Here we see the same behaviour as in (3), just continued. What happened is that we did provide another argument "D". So my function from step (3) was waiting for one more parameter (named c). I got an unnamed argument "D" - OK - that sounds good - I provide this argument for the parameter c. This runs my function fun with args a="A", b="C" and c="D" - it executes and returns a Text value "ACD". But the resolution continues - I'm still pending a named argument b="B". Ok but what the function returned is a Text value, not a Function - so a type error is invoked.

The case (5) alone is problematic, as I get a weird type error which does not really tell me what went wrong. That case is actually an instance of #5219.
But (4) shows something even more concerning. What I do is I expect fun to return me a Text, so I call length on it. What happens then is that my resolution is currently suspended. It resolved to a function that is waiting for one parameter named c and is also waiting to apply one named argument b="B". The object returned is a Function holding all that 'state'. I call length on this and I get a message that a Function does not have a method called length. Well, fair - functions don't have those. But I have not expected a Function, I've provided 3 arguments to my function fun and I expect it to return a Text. Instead, I see I'm getting a function - so this looks like a usual case of "not enough arguments" - if I have a function f x y and provide just IO.println (f 1) I get a function. But wait - I look at the code - fun is expecting 3 parameters and I did provide 3 arguments!. The issue is that the argument b="B" did not match and is silently suspended, with no way to see it.
I've just lost about 30-40 minutes trying to track down a case in my code when this was happening. It is a very confusing kind of error message, because it suggests we do not have enough arguments, but we have all the arguments that we need!

I'm not sure what would be the proper solution for this.

I think from the user perspective - ideally, we should make sure that named arguments work even if reordered. I'm not sure how viable that is because we are limited by other requirements, like ensuring named arguments propagate through currying.
If we cannot make such reordered named arguments work, we should probably at least try to make sure the error message that is raised is clearer - the current one is really confusing. This would likely be the same issue/solution as #5219. Likely if we get something like: "Unexpected named argument b", it could be a bit easier to track down what the issue is.

This behaviour is consistent with point (2) of the Named Arguments section in the function arguments design: https://enso.org/docs/developer/enso/syntax/function-arguments.html#named-arguments
This probably rules out (1) - as I suspected - it is unlikely possible to reconcile reordering named arguments with currying.
However still, I think we need to look into (2) - currently the user experience of using named arguments can become a rabbit hole.
Maybe actually we could somehow 'group' arguments to user-facing components that do not resort on currying and still have (1)? I'm not a fan of doing this as that would make the language less uniform. However from the UX perspective - we definitely want to have clear error messages and intuitive handling of arguments - and the current state is quite far from that - as shown here or in #5219 - i.e. when you misspell an argument name.
This may not be as high priority though, as the IDE should generally manage arguments of nodes and so it should be much harder to trigger these errors in the IDE. I imagine the only scenario is when the library version has changed between runs of a workflow and an argument was renamed there.

For case 3: After we've supplied "A" and "C" (as parameters a and b), we return a function expecting a parameter c. There is a suspended named parameter, but it is the wrong name, so we would expect it to say that the argument to the partially-applied function is the wrong name, right? So just having b="B" there is immediately an error.
So that in case 4, we don't even get to the point of trying to call length, since we already hit the error with b="B".

For case 3: After we've supplied "A" and "C" (as parameters a and b), we return a function expecting a parameter c. There is a suspended named parameter, but it is the wrong name, so we would expect it to say that the argument to the partially-applied function is the wrong name, right? So just having b="B" there is immediately an error.
So that in case 4, we don't even get to the point of trying to call length, since we already hit the error with b="B".

Yes, that would be ideal. I think there are some challenges to that (we need to ensure that some function returned further downward the chain will not be taking this parameter).
We can relatively easily check this if the return value evaluates to something that is not a Function anymore. Then, if we still have 'pending' named arguments but have something that is not a Function, instead of a Type error, we could display this more specific error.
But the general case and one shown in (4) is actually not that simple. The issue is currying - at the point where we are, we do not know if after we get the last parameter we are waiting for (c), if the function fun will not return a new function, expecting the parameter b. So we cannot error immediately :/
e.g.
fun2 a b c =
    my_b = b
    if c == "D" then b-> a+my_b+b+c else z-> a+b+c+z

As demonstrated in fun2, the condition cannot be known a-priori:
fun2 "A" "C" (b="B")

Here we don't know what the parameter be and if b will be valid or not:
If we do
fun2 "A" "C" (b="B") "D"

Then b="B" will happily match, we'll get back "ACBD".
But if we do
fun2 "A" "C" (b="B") "X"

then this will reduce to a function waiting for parameters named z but having a pending named argument b="B" - again we do not know anything because we can get one more argument and maybe we will get back a curried function that does take "B" - determining this in general cases is essentially equivalent to the halting problem and also can be non-deterministic if we have side effects.

I guess what we can do, is detect cases where a Type error is thrown or whenever a "Function value with pending unresolved named arguments" is encountered, and handle the errors specially.
In particular, I'm not sure that the entity "Function value with pending unresolved named arguments" should be of type Function. I'm not sure if it can be any other type, but at least IMO we should print in its textual representation that there are these pending arguments.
This is really hard, I'm not sure what the right solution should be but we definitely should try to improve this as currently the errors can be very misleading and basically unreadable.

Can we include looking at partially applied constructors within the type checking in this ticket please.

Can we include looking at partially applied constructors within the type checking in this ticket please.

These seem to be rather completely unrelated issues, shouldn't we create a separate ticket for it?

Or are they actually related?

I think they are related.

all we can do is to improve error reporting

If there is an argument that requires some type, but gets Function, it is a sign of ...

partially applied

... arguments. Of a function or ...

constructor

... at that moment we should look at FunctionSchema and describe what arguments need to be provided or what are the over-saturated arguments and report that to the user. When I heard about the partially applied constructor problem, I immediately thought the fix is going to be at the same place.

Or are they actually related?

I think they are related.

I see, ok. It will be great to have both of these issues improved 🙂

The z argument is not displayed on the third line, even though it has beem set z=23

As designed. We don't display arguments with default values. You voted +1.

I'm not sure how to solve this.

I believe it could be solved by removing the possibility to re-order arguments. But as @kustosz wrote:

it would be very annoying in practice as the point of named arguments is to not think about order

I am not sure it is the most important point, I believe the point of named+defaulted arguments is to not have to specify them all.
I believe that, if we remove the ability to specify arguments in random order, we can solve these issues. However it is a significant change to the language and needs to be discussed with major stakeholders. I don't think it would affect the IDE experience, so it may be acceptable, but I happily leave the negotiations up to you, Radek.

my two cents: specifying w out of place should be an error - default parameters being able to be specified out of order is fine, but they shouldn't be allowed to be specified out of order when the arguments come from different declarations. i think this is especially true if w does not show up in the docs in the first place, in which case i think it would be counterintuitive to accept w before z
... alternatively another possibility may be to add metadata about function parameters to the type, to make the entire call possible to statically analyze?

The z argument is not displayed on the third line, even though it has beem set z=23

As designed. We don't display arguments with default values. You voted +1.

I though this applies only to argument with default value. Then that's ok. But here the value is overridden with value provided by the user. And in that case not displaying it does not seem ok.

... the value is overridden with value provided by the user...

OK, that can be fixed. You can report a bug for me.

my two cents: specifying w out of place should be an error - default parameters being able to be specified out of order is fine, but they shouldn't be allowed to be specified out of order when the arguments come from different declarations

One thing to mention is currying. There should be no way to tell a difference between f(a, b, c) and g(a)(b)(c) which shall behave the same if f is curried version of g or vice versa. Named arguments are particularly nasty from this perspective as one can use f c=3 and then that has to work with g c=3.
However g c=3 is an application of an over-saturated argument and the whole runtime is quite complicated because of supporting that. Moreover it creates additional problem of properly reporting an error as g anything=something is valid until g gets saturated & evaluated.
If we want to improve the error messages, then (I believe) we have to restrict the freedom people have when providing arguments to a function.

... the value is overridden with value provided by the user...

OK, that can be fixed. You can report a bug for me.

Ok: https://github.com/enso-org/enso/issues/7723

my two cents: specifying w out of place should be an error - default parameters being able to be specified out of order is fine, but they shouldn't be allowed to be specified out of order when the arguments come from different declarations

One thing to mention is currying. There should be no way to tell a difference between f(a, b, c) and g(a)(b)(c) which shall behave the same if f is curried version of g or vice versa. Named arguments are particularly nasty from this perspective as one can use f c=3 and then that has to work with g c=3.
However g c=3 is an application of an over-saturated argument and the whole runtime is quite complicated because of supporting that. Moreover it creates additional problem of properly reporting an error as g anything=something is valid until g gets saturated & evaluated.
If we want to improve the error messages, then (I believe) we have to restrict the freedom people have when providing arguments to a function. E.g. f c=3 is an error, if a or b is missing. Or (if a and b have default values), then f c=3 is OK, but the values of a and b are immediately set to their default values and can no longer be modified.

I like your idea. I think looking at the practical use of the language, it is really important to get good, easy to understand, error messages.
Your PR has already improved this vastly, but I think we are not yet 'there'. What you are proposing here could make it much better.
As you mentioned, this would not change the semantics from the GUI perspective, since there we specify the arguments in the right order 'anyway'. It may make the life of text users slightly harder, but IMHO the benefit of better error messages far exceeds the benefit of being able to reorder named arguments.


Add new playback strategy - basicStrategy

📺 What
Adds a new playback strategy - basicStrategy.

Tickets: IPLAYERTVV1-11669

🛠 How
Adds new strategy file. This strategy uses the devices native playback capabilities, but differs from the nativeStrategy in that it does not contain specific playback state or work-arounds for non-compliant browsers. Playback is fully event driven, and any state is determined directly from the browsers media element.
✅ Testing
Test Guidelines

Test engineer sign off
:x:

Tested with extensive manual testing.
✅ Acceptance criteria [Optional]
[MANDATORY for cross-team reviews]

[] Feature parity with existing Native Strategy

Mind updating the wiki on configuration when this is in? https://github.com/bbc/bigscreen-player/wiki/Playback-Strategy

Mind updating the wiki on configuration when this is in? https://github.com/bbc/bigscreen-player/wiki/Playback-Strategy

Mind updating the wiki on configuration when this is in? https://github.com/bbc/bigscreen-player/wiki/Playback-Strategy

👍 yep, will add this to the ticket to do. Thanks.

Mind updating the wiki on configuration when this is in? https://github.com/bbc/bigscreen-player/wiki/Playback-Strategy

👍 yep, will add this to the ticket to do. Thanks.


Update wireguard to 1.6.3

Please update my adapter ioBroker.wireguard to version 1.6.3.
This pull request was created by https://www.iobroker.dev c0726ff.

RE-CHECK!


XamlUIBasics SemanticZoom Sample broken when built with newer Microsoft.NETCore.UniversalWindowsPlatform

The SemanticZoom.ZoomedOutView mode does not render correctly when building with newer versions of Microsoft.NETCore.UniversalWindowsPlatform.
I also consider all of the Windows-universal-samples to be invalid at this point because most of the samples are being built against the 5.0 version of Microsoft.NETCore.UniversalWindowsPlatform.  These samples should reveal issues with newer 6.x Platform SDK builds so that Microsoft can see the problems that have been introduced so they can be fixed before vaulting such problems onto the developer community and ultimately Windows 10 users.
There are many other issues related to newer versions of Microsoft.NETCore.UniversalWindowsPlatform, that work perfectly with the 5.0 version.
Many bugs have been introduced starting with the 6.x releases of Microsoft.NETCore.UniversalWindowsPlatform.  One of the more egregious is the way ZipFile.ExtractToDirectory no longer honors the FutureAccessList.  Please start building with the latest releases!
The reason some Microsoft staffers appear mystified when presented with issues by other developers on here is because the external developers have swapped in 6.x versions of Microsoft.NETCore.UniversalWindowsPlatform when building these samples.  Hence they see issues Microsoft staffers do not.

Lastly, the rendering issue introduced with


Implement auth client to provision Istio identity for consumer instance

Issue moved to istio/istio #1491 via ZenHub