Spyder freezes when large MaskedArrays are in memory

I've experienced a long-standing issue with Spyder that dates back to Spyder 2.1.13 (and likely before) and is still present in the current 3.0.0b1. It is likely linked to Spyder interactions with the cdms2 module that uses numpy.ma for its backend array manipulation.
After loading a ~large [50,50,300,360] ~1GB matrix into memory (from a netcdf file - using cdms2.open), the responsiveness of the console disappears.. And intermittently responds to inputs. This continues to occur until the matrix is reduced in size, i.e. var = var[0:1,:,:,:]
The issue disappears when using pure numpy, so var = numpy.ma.ones([50,50,300,360]) doesn't yield the same lack of responsiveness.
The issue is not related to resource limitations on the machine (redhat6.7, 128GB, Xeon E5-2643 quad core).
As it's related to Spyder interactions with the cdms2 module, I'm not sure the best way to get to the bottom of this issue.. Following #1958 and #1968 I have adjusted the auto refresh feature of the variable explorer (turned this off) but this doesn't appear to solve the problem.
What further information is required to attempt to get to the bottom of this issue? For completeness:
Spyder version: 3.0.0-b1
Python version: 2.7.10
Qt version: 4.8.4
PyQT4 version: 4.11.3
numpy version: 1.9.0
uv-cdat/cdms2 version: 2.4.0-rc1
@dnadeau4 pinging you here..

Did you try to completely disable the variable explorer (ie close the pane) ? There is nothing special about cdms in particular in spyder.

@Nodd good call.. Closing the variable explorer starts to speed things up considerably.. Ok so my issue appears to be with the variable explorer really slowing things down.. Is this noted somewhere in another open issue?

@durack1, thanks for taking the time to open this issue and letting us know about this problem.
This is a very interesting use case to improve Spyder responsiveness with big data. We have done a lot of work lately to better handle big DataFrames and NumPy arrays when they are opened.
However, the problem here could lie in the fact that we're not optimizing how the array is represented in the Variable Explorer. What I mean is that in the Value column we are just putting the full repr of the array (i.e. what's printed when you call print(array)) instead of something simpler (like it's first 10 or 20 elements).
But to test this hypothesis, I need you to give some sample code (using cdms2) I can test on my side to fix this problem :-)

@ccordoba12 no problem, the easiest example would be just load a variable from a netcdf file using cdms2, while you have a variable explorer pane open.
So the code:
import cdms2 as cdm
fileHandle = cdm.open('AusCOM1-0.Salt.Omon.so.00011231-00501231.nc')
var = fileHandle('so')
fileHandle.close()

Should reproduce the laggyness that I've experienced - the variable will require ~1GB available memory to load.. You can grab the file above from here (it's 355MB)

@ccordoba12 we should use that file for future tests! :wink:

@goanpeca @ccordoba12 if you would like an even larger test file, I'd be more than happy to provide this!

@ccordoba12 @goanpeca let me know if you have any trouble getting access to cdms2 and installing this along with netcdf4..

@durack1 thanks for the heads up :-)

@durack1, could you also upload a smaller nc file? I mean one that doesn't cause Spyder to freeze? That would be really helpful too :-)

@ccordoba12 it'd have to be pretty small.. I've been experiencing this freezing issue even with smaller matrices.. I'm currently on travel but will drop a smaller file on the webserver when I'm back in a week or so..

Ok, no problem. I'm working on other things right now, but I hope to address this issue for beta2.

@ccordoba12 great - what it the timeline for beta2? I'm not holding things up if I get this data to you next week am I?

Don't worry, we are two or three weeks away of it :-)
El 19/10/15 a las 07:26, Paul J. Durack escribió:

@ccordoba12 https://github.com/ccordoba12 great - what it the
timeline for beta2? I'm not holding things up if I get this data to
you next week am I?
—
Reply to this email directly or view it on GitHub
https://github.com/spyder-ide/spyder/issues/2748#issuecomment-149199662.

@ccordoba12 apologies for the delay.. Here is a much smaller file that doesn't appear to trigger the laggyness issues. In the example below I have re-enabled the variable explorer pane in Spyder 3.0.0b1 and it seems to work fine - it loads two variables from a netcdf file using cdms2.
So the code:
import cdms2 as cdm
fileHandle = cdm.open('DurackandWijffels_GlobalOceanSurfaceChanges_1950-2000.nc')
saltChange = fileHandle('salinity_change')
thetaoChange = fileHandle('thetao_change')

You can grab the file above from here (it's 607KB)

@durack1, please give us a smaller than the one you uploaded first (the one called AusCOM1-0.Salt.Omon.so.00011231-00501231.nc).
That file seems to require 10 gigs of ram (not 1, I tested it on my virtual machines! :-), so I can't use it for testing.

@ccordoba12 the file should be fine - you could load a smaller subset using:
>>> import resource
>>> import cdms2 as cdm
>>> import numpy as np

>>> print 'Max mem: %05.3f GB' % (np.float32(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)/1.e6)
Max mem: 0.155 GB
>>> fileHandle = cdm.open('AusCOM1-0.Salt.Omon.so.00011231-00501231.nc')
>>> print 'Max mem: %05.3f GB' % (np.float32(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)/1.e6)
Max mem: 0.155 GB
>>> var = fileHandle('so',time=slice(0,1))
print 'Max mem: %05.3f GB' % (np.float32(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)/1.e6)
>>> Max mem: 0.155 GB
>>> fileHandle.close()
>>> print 'Max mem: %05.3f GB' % (np.float32(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)/1.e6)
Max mem: 0.155 GB

Obviously then changing the number of timesteps loaded by altering your indexes (e.g. var = fileHandle('so',time=slice(0,10))) will then get progressively larger matrices loaded into memory. The single time slice example above should need just ~0.2GB or so..

I wonder if we should show the repr of objects smaller than say 1MB, and keep a whitelist for types that we know have no problems (like pandas dataframe). It would avoid freezing or crashing spyder because of unknown big objects.

@Nodd, it's a good idea. But how to determine reliably an object's memory?

Sorry, it's not its memory the problem, but the size of its repr. How could we account for that?

@ccordoba12 @Nodd in the case of a numpy array, why not use numpyArray.shape?

@durack1, that just gives its size :-) repr is the string printed when you do something like:
>>> a = np.array([1, 2, 3])
>>> a
array([1, 2, 3])

Most numpy repr's are very efficient, but some (like record arrays and it seems masked arrays) seem to not be :-)

@ccordoba12 but such shape info can then be used to query a small subset of the array - using the very efficient indexing syntax - so the example above var.shape = (50,50,300,360), could then define a more targeted (and much smaller) temporary matrix in which repr can then be performed.

We can guess that the bigger an object is, the bigger its repr could be. If we know that a type has a simple repr, we can force to always show it.
My proposal is not perfect, it's just a workaround to avoid recurring problems with the variable explorer and big objects.
As for the size, I vaguely recall that python has a sizeof equivalent. It may fail for some types, but again it can be better than nothing.

I discovered that this problem was generated because the repr of masked arrays is terribly inefficient.
So the fix is to use as the Value of masked arrays (i.e. what appears in the fourth column of the Variable Explorer) a simple string (i.e. just "Masked array") instead of the variable's repr.
And then the problem goes away :-)

@ccordoba12 great, looking forward to testing this in beta2.. @goanpeca any chance code-folding will also find itself into beta2?

@durack1, not yet sorry :-(
I think it will be for beta3

@goanpeca no problem, looking forward to testing code-folding in beta3 then!

@ccordoba12 Maybe it would be worth opening a bug report for numpy ?

@Nodd, sure. Could you do it, please?

@Nodd could you do it, please :-), and add a link to this issue :p

Yeah I'll do it, but first I'll have to check what "the repr of masked arrays is terribly inefficient." means.

@Nodd, it means that when you run
repr(ma)

where ma is a masked array, it takes a lot of time to return the result (at least for the arrays provided by @durack1)

@Nodd those files are available through the links above here (large 3D matrices) and here (smaller 2D matrices)


Automated Package Upgrade

:tada: This PR is included in version 5.0.435 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Initializers are private

Documentation states that the generated structs will have public initializers, which would be required when attempting to instantiate messages within a module, but they are private.

Are you referring to the convenience initializers that take fields as arguments? Those have been removed; only the parameterless initializer remains.
Unfortunately the documentation that you linked to has become very out of date; I just filed #425 to track fixing it up.
To initialize a message you have a couple approaches:

Create a new one as a var and then set its fields:

var foo = MyGeneratedMessage()
foo.someField = 5
foo.someOtherField = "hello"

...but if you're picky about your mutable/immutable state, you may not want to have a var hanging around longer than you need it.

Use the static with factory method on all message types which captures the initialization in a block so that the finished message can be stored in a let:

let foo = MyGeneratedMessage.with {
  $0.someField = 5
  $0.someOtherField = "hello"
}

@davidmuzi Hi, can you find the Deserialization method? init(protobuf: Data) throws, I m using 0.9.901

init(protobuf: Data) throws has been renamed to init(serializedData: Data) throws in the recent versions.

Apologies for bringing this issue up again, but why initializers with fields as arguments were removed, and instead we have the with static method alternative? It seems to me that initializers are more idiomatic Swift code.
Thanks!

We remove them for a few reasons: code bloat, proto evolution (as the proto changes with time (fields added, etc.), it can break code as the order of the fields to the initializer could change, etc.

@thomasvl Thanks for the answer.
But without an initializer, I think one can just define an invalid message? Or is it always with default values and hence allowed?

There is always a init().
Protobuf defines all message fields to have their default value unless you set them to something else.  So the message is always valid from that pov.


latest-41.20241113.1 not available for aurora-dx

Describe the bug
I tried using the commands in the release notes to rebase to this version as I need a newer kernel.
I ran
IMAGE_NAME=$(jq -r '.["image-name"]' < /usr/share/ublue-os/image-info.json)
sudo bootc switch --enforce-container-sigpolicy ghcr.io/ublue-os/$IMAGE_NAME:latest-41.20241113.1

and I got this error
ERROR Switching: Pulling: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: reading manifest latest-41.20241113.1 in ghcr.io/ublue-os/aurora-dx: manifest unknown

What did you expect to happen?
I expected to use the image
Output of bootc status
No staged image present
Current booted image: ghcr.io/ublue-os/aurora-dx:stable
    Image version: 41.20241117.3 (2024-11-17 15:52:27 UTC)
    Image digest: sha256:e78b70e68f068ad8e4331e636f4a488ff820fa5bbcdfa39a6643f0749414e790
Current rollback state is native ostree

Output of groups
phire wheel docker incus-admin libvirt

Extra information or context
No response

Remove the date afterwards.
The date pins to that specific image.

This specific date was a partial build, there's no aurora-dx here (just looked), I'd grab another day or just choose one from the ujust rebase helper to get you on the latest channel.

Thanks, I thought I was on latest. It has a new enough kernel. Unfortunately it did not fix the underlying issue. This issue is solved though.


Use command+S to submit query

In CartoDB you can use Cmd-S on a Mac (not sure on windows) to submit the query, it would be nice here too.

I've got a branch up that would close this issue, but I'm waiting until I can get crtl/cmd+arrow key to cycle through the history as well to submit a PR.


Google OpenID Connect does not work

I saw #477 and #520 but I was not able to use Google OpenID Connect support. Is anyone at this moment?
When I tried yesterday, there seemed several weird problems found that are hard for me to fix or workaround.
1: nonce is being rejected by server
As @eshellman says at #477, AuthCanceled will be thrown, probably because Google's server sends HTTP 400. I inserted some debug statements and found JSON message from Google.
{
  "error" : "invalid_request",
  "error_description" : "Parameter not allowed for this message type: nonce"
}

OpenIdConnectAuth class inserts the "nonce" parameter every time (auth_complete_params()), while I'm not sure if it is legitimate for the spec.
Maybe GoogleOpenIdConnect needs to revert the behavior. Possibly I'm just wrong with how to use python-social-auth.
    def auth_complete_params(self, state=None):
        params = super(OpenIdConnectAuth, self).auth_complete_params(state)
        # Add a nonce to the request so that to help counter CSRF
        params['nonce'] = self.get_and_store_nonce(
            self.ACCESS_TOKEN_URL, state
        )
        return params

2: RS256 is required while open_id.py is assuming HS256
open_id.py is assuming RS256 while the server is forcing HS256? It looks even after nonce is deleted (for debugging), jwt_decode() now fails itself.
This topic will be related to this issue too: https://github.com/omab/python-social-auth/issues/641
3: Lack of documentation.
Right now only the (possibly) useful information will be https://github.com/omab/python-social-auth/pull/520
No documentation seems in http://django-social-auth.readthedocs.org/en/latest/backends/google.html (at least right now) It does not mention "OpenID Connect" at all.
It will be very helpful if there are some explanations about how to get it work.
Thanks!

:+1:

i need the python script and password list
On Fri, Jun 19, 2015 at 5:40 AM, Masahiro Wakame<EMAIL_ADDRESS>wrote:

[image: :+1:]
—
Reply to this email directly or view it on GitHub
https://github.com/omab/python-social-auth/issues/644#issuecomment-113503760
.

FWIW, I implemented a simple django project with Python Social Auth that demonstrates this behavior.
https://github.com/dmiyakawa/psa_exp
Maybe I'm not using the library, so pointing it out would be also helpful.

:+1:

Any updates on this? Seems like OpenID has been phased out in favor of OpenID Connect. Hence support for OpenID Connect is imperative.

+1

My progress with this issue thus far:

I did the nonce fix

also got the RS256 algorithm error

Added ID_TOKEN_JWT_DECODE_KWARGS = {'algorithms': ['RS256']} to settings to fix that

Got a different algorithm error

Installed cryptography (pip install cryptography) which fixed that

Got this error TypeError: Expecting a PEM-formatted key

I have no idea where to even start fixing that.

@unklphil I just implemented OpenIdConnect authorization for some proprietary platform today which also only used RS256 for signing JWTs. I had to do something like this in ```def validate_and_return_id_token`` to get it working:
from cryptography.hazmat.backends import default_backend
from cryptography.x509 import load_pem_x509_certificate

key = load_pem_x509_certificate(self.setting('ID_TOKEN_DECRYPTION_KEY'), default_backend())
public_key = key.public_key()
decode_kwargs = {
    'algorithms': ['RS256'],
    'audience': client_id,
    'issuer': self.ID_TOKEN_ISSUER,
    'key': public_key,
    'options': {
        'verify_signature': True,
        'verify_exp': True,
        'verify_iat': True,
        'verify_aud': True,
        'verify_iss': True,
        'require_exp': True,
        'require_iat': True,
    },
}

your_social_ID_TOKEN_DECRYPTION_KEY should contain the certificate. I used something like this to load it:
in_file = open("public-key-for-jwt-signing.pem", "rb")
your_social_ID_TOKEN_DECRYPTION_KEY = infile.read()

I hope that helps.

The related PRs were merged or ported to the new lib social-core.


Telemetry plugins should not crash on invalid config

Metadata exchange, stats, stackdriver none of these plugins should crash on failed config validation.
They should increment an invalid config counter and disable them selves.
AttributeGen already does this.
It is ok to get this into 1.6.1

@esnible changing behaviour of telemetry plugins.

Not fully done. We need to update plugins to continue working with sane defaults if possible.


Bonus Carnage healing

using carnage heals for 25 hp per target hit

:tada: This PR is included in version 1.6.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Viz test problems

There's some stuff working just by chance in our controller tests:

[ ] the hostname is set to the default example.com even though the code is using the header HTTP_HOST
[ ] not everything is being cleaned between tests, at least "derived visualizations with no canonical". This causes "test races". Just activate this test and try to run the whole file.
[ ] The method delete_user_data is quite fragile. Almost any interaction with named maps makes it stop processing deletions.

There are some limitations of rack test that can make removing the host header a bit harder than I expected: https://github.com/jnicklas/capybara/issues/36
Need to review the query builder tests, that probably don't need anything related to rack, sessions and all that.


Leftover changes

This branch contains the leftover changes from PR #88 after my cleanup. I either haven't tested them or I still don't understand what they do.

@jtligon and @jeredfloyd: please review.

@jeredfloyd Thanks! Did you actually try out the Bluetooth sharing functionality with Jelling? If so, thoughts?

I did not try out the Share functionality yet; I've only checked that the View loads without crashing the app and attempted to resolve any runtime messages that were logged.


Display only available options for years mode

Expected behaviour
I am using the datepicker for years only in this instance.  I have available years from 2016 to 2022 only.  When the picker opens it should display 2016 in the top left corner as the first option and end with 2022 as the last option.  I would not like to see the seven disabled years prior to the first available year (as seen in the image below), which also causes years 2021 and 2022 to have to be paged to.  Is there a way to do this so that the seven pickable years are displayed and the users don't have to page to find the last two?

Actual behaviour
As mentioned above the picker displays seven undesired (yet disabled) years prior to the minDate, it also displays past the maxDate, but this is less of a concern, altho if we could remove both that would be optimal.  I have experimented with css
.year.disabled{display:none;}
which does remove the disabled, however instead of 2021 and 2022 showing on the initial picker view we still have to page to the next to be able to select them.

Datepicker version used
1.10.0
Example code
Because I have many and the positioning was not working I initialize the pickers in the each loop where i can set the container to the parent.
$('.yearpicker').each(function () {
                var container = $(this).parent();

                $(this).datepicker({
                    format: "yyyy",
                    viewMode: "years",
                    startView: "years",
                    yearStart: 1,
                    startDate: new Date(2016, 0, 1),
                    minViewMode: "years",
                    autoclose: true,
                    container: container,
                    orientation: 'auto',
                    minDate: new Date(2016, 0, 1), // January 1, 2016
                    maxDate: new Date(2022, 11, 31), // December 31, 2022
                });

            });

Thank you, i appreciate your assistance.

Update
I have been able to find a workaround where i basically build the calendar on show.  This seems to solve my issue, I'll leave it here in case anyone else needs similar functionality.

            $('.yearpicker').each(function () {
                var container = $(this).parent();

                $(this).datepicker({
                    format: "yyyy",
                    viewMode: "years",
                    startView: "years",
                    minViewMode: "years",
                    maxViewMode: "years",
                    autoclose: true,
                    container: container,
                    orientation: 'auto',
                }).on('show', function (event) {
                    var currentYear = new Date().getFullYear();
                    var startYear = 2016;
                    var endYear = 2022;
                    var numYears = endYear - startYear + 1;

                    // Set the initial view to startYear
                    $(this).datepicker('update', new Date(startYear, 0, 1));

                    // Clear out all existing years
                    $('.datepicker-years .datepicker-switch').text(startYear);

                    // Generate and append years
                    var yearsHtml = '';
                    for (var i = 0; i < numYears; i++) {
                        var year = startYear + i;
                        yearsHtml += '' + year + '';
                    }
                    $('.datepicker-years tbody').html('
' + yearsHtml + '
');
                });

            });


Sceneform Tools (Beta) Plugin does not correctly import gltf files with pbrSpecularGlossiness material

I tried importing a .gltf file with pbrSpecularGlossiness material. I did not get any error, and the .sfa and .sfb files were generated. However, both the .sfa and .sfb files do not contain any specular information.
Also, the app/build/sceneform_sdk/default_materials/gltf_material.sfm file contains information for just the pbrMetallicRoughness material, and not for pbrSpecularGlossiness.
It will be great if pbrSpecularGlossiness material is supported as well.
Google Sceneform Tools (Beta) Version: 1.0.0
Android Studio 3.1.1
Build #AI-173.4697961, built on April 4, 2018
JRE: 1.8.0_152-release-1024-b01 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.13.4

Thanks for the request!  We don't yet support gltf extensions in Sceneform (pbrSpecularGlossiness or unlit), but we are working to expand the material variations that we support for importing, so stay tuned.

pbrSpecularGlossiness is now supported starting with version 1.4.


Sign-in with AAD Auth: AADSTS50000: There was an error issuing a token or an issue with our sign-in service.

Describe the issue
I'm trying to create a BC container with AAD auth using the script below. The same script works fine in two different tenants, but in our company tenant it shows the There was an error issuing a token or an issue with our sign-in service during SSO to the BC container.
Scripts used to create container and cause the issue
$appIdUriPrefix = "selfservice-ppitest"
$tenantName = "cosmoconsult.com"
$appIdUri = "https://$appIdUriPrefix.$tenantName"

Connect-AzureAD
$ids = Create-AadAppsForBC -appIdUri $appIdUri -publicWebBaseUrl "https://mybcvm.westeurope.cloudapp.azure.com/myaadcont2" -useCurrentAzureAdConnection -SingleTenant -IncludeExcelAadApp -IncludePowerBiAadApp -IncludeEmailAadApp -IncludeApiAccess

New-BcContainer -accept_eula -accept_outdated -containerName bcaad19 `
    -artifactUrl (Get-BCArtifactUrl -version 19.5 -type Sandbox) -licenseFile "<mylicenseUrl>" `
    -useSSL -useTraefik -authenticationEMail "<myemail>" `
    -auth AAD -AadTenant $tenantName -AadAppId "$($ids.SsoAdAppId)" -AadAppIdUri $appIdUri

Full output of scripts
BcContainerHelper is version 3.0.4
BcContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Datacenter - ltsc2019
Docker Client Version is 20.10.14
Docker Server Version is 20.10.14
Downloading application artifact /sandbox/19.5.36567.38753/w1
Downloading C:\Users\vmadmin\AppData\Local\Temp\4\fd090985-fa83-480f-b38f-90b3f3d3b83d.zip
Unpacking application artifact to tmp folder using Expand-Archive
Downloading platform artifact /sandbox/19.5.36567.38753/platform
Downloading C:\Users\vmadmin\AppData\Local\Temp\4\110b46fd-6efe-4fab-9ada-75dd99f83ef0.zip
Unpacking platform artifact to tmp folder using Expand-Archive
Downloading Prerequisite Components
Downloading c:\bcartifacts.cache\sandbox\19.5.36567.38753\platform\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
Downloading c:\bcartifacts.cache\sandbox\19.5.36567.38753\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Downloading c:\bcartifacts.cache\sandbox\19.5.36567.38753\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Fetching all docker images
Fetching all docker volumes
Enabling SSL as otherwise all clients will see mixed HTTP / HTTPS request, which will cause problems e.g. on the mobile and modern windows clients
Using image mcr.microsoft.com/businesscentral:10.0.17763.2686
PublicDnsName is mybcvm.westeurope.cloudapp.azure.com
Creating Container bcaad19
Style: sandbox
Multitenant: Yes
Version: 19.5.36567.38753
Platform: 19.0.36528.38726
Generic Tag: <IP_ADDRESS>
Container OS Version: 10.0.17763.2686 (ltsc2019)
Host OS Version: 10.0.17763.2686 (ltsc2019)
Using process isolation
Using locale en-US
Adding special CheckHealth.ps1 to enable Traefik support
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file https://ccppi.blob.core.windows.net/lic/devlic-bc19-de
Downloading C:\ProgramData\BcContainerHelper\Extensions\bcaad19\my\license
Additional Parameters:
-e webserverinstance=bcaad19
-e publicdnsname=mybcvm.westeurope.cloudapp.azure.com
-l "traefik.protocol=https"
-l "traefik.web.frontend.rule=PathPrefix:/bcaad19"
-l "traefik.web.port=443"
-l "traefik.soap.frontend.rule=PathPrefix:/bcaad19soap;ReplacePathRegex: ^/bcaad19soap(.*) /BC$1"
-l "traefik.soap.port=7047"
-l "traefik.rest.frontend.rule=PathPrefix:/bcaad19rest;ReplacePathRegex: ^/bcaad19rest(.*) /BC$1"
-l "traefik.rest.port=7048"
-l "traefik.dev.frontend.rule=PathPrefix:/bcaad19dev;ReplacePathRegex: ^/bcaad19dev(.*) /BC$1"
-l "traefik.dev.port=7049"
-l "traefik.snap.frontend.rule=PathPrefix:/bcaad19snap;ReplacePathRegex: ^/bcaad19snap(.*) /BC$1"
-l "traefik.snap.port=7083"
-l "traefik.dl.frontend.rule=PathPrefixStrip:/bcaad19dl"
-l "traefik.dl.port=8080"
-l "traefik.dl.protocol=http"
-l "traefik.enable=true"
-l "traefik.frontend.entryPoints=https"
--env customNavSettings=ValidAudiences=9ffc2893-7dea-4c10-8f3f-403a608af0ec;https://api.businesscentral.dynamics.com,DisableTokenSigningCertificateValidation=True,ExtendedSecurityTokenLifetime=24,ClientServicesCredentialType=NavUserPassword,PublicODataBaseUrl=https://mybcvm.westeurope.cloudapp.azure.com/bcaad19rest/odata,PublicSOAPBaseUrl=https://mybcvm.westeurope.cloudapp.azure.com/bcaad19soap/ws,PublicWebBaseUrl=https://mybcvm.westeurope.cloudapp.azure.com/bcaad19
Files in C:\ProgramData\BcContainerHelper\Extensions\bcaad19\my:
- AdditionalOutput.ps1
- CheckHealth.ps1
- license
- MainLoop.ps1
- SetupVariables.ps1
- updatecontainerhosts.ps1
Creating container bcaad19 from image mcr.microsoft.com/businesscentral:10.0.17763.2686
de05a6b257c03fd2ee214b76a060301eb6c508d8efc0fc82c3deb4df41276887
Waiting for container bcaad19 to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/19.5.36567.38753/w1
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\19.5.36567.38753\platform\ServiceTier\Program Files
c:\dl\sandbox\19.5.36567.38753\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\sandbox\19.5.36567.38753\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\sandbox\19.5.36567.38753\platform\WindowsPowerShellScripts\WebSearch
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Determining Database Collation from c:\dl\sandbox\19.5.36567.38753\w1\BusinessCentral-W1.bak
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Copying Web Client Files
c:\dl\sandbox\19.5.36567.38753\platform\WebClient\Microsoft Dynamics NAV
Copying Client Files
c:\dl\sandbox\19.5.36567.38753\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\sandbox\19.5.36567.38753\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\sandbox\19.5.36567.38753\platform\LegacyDlls\systemFolder
Copying ModernDev Files
c:\dl\sandbox\19.5.36567.38753\platform
c:\dl\sandbox\19.5.36567.38753\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\19.5.36567.38753\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\19.5.36567.38753\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\19.5.36567.38753\w1\Extensions
Copying Applications
C:\dl\sandbox\19.5.36567.38753\platform\Applications
Starting Business Central Service Tier
Importing license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 205 seconds
Installation complete
Initializing...
Setting host.containerhelper.internal to <IP_ADDRESS> in container hosts file
Starting Container
Hostname is bcaad19
PublicDnsName is mybcvm.westeurope.cloudapp.azure.com
Using AccessControlService Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint 69E69B33BB8B4E756D631E53EE1796E23D1D04C8
DNS identity mybcvm.westeurope.cloudapp.azure.com
Modifying Service Tier Config File with Instance Specific Settings
Modifying Service Tier Config File with settings from environment variable
Setting ValidAudiences to 9ffc2893-7dea-4c10-8f3f-403a608af0ec;https://api.businesscentral.dynamics.com
Setting DisableTokenSigningCertificateValidation to True
Setting ExtendedSecurityTokenLifetime to 24
Setting ClientServicesCredentialType to NavUserPassword
Setting PublicODataBaseUrl to https://mybcvm.westeurope.cloudapp.azure.com/bcaad19rest/odata
Setting PublicSOAPBaseUrl to https://mybcvm.westeurope.cloudapp.azure.com/bcaad19soap/ws
Setting PublicWebBaseUrl to https://mybcvm.westeurope.cloudapp.azure.com/bcaad19
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Enabling Financials User Experience
Using license file 'c:\run\my\license'
Import License
Dismounting Tenant
Mounting Tenant
Setting AadTenantId to cosmoconsult.com
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating bcadmin as SQL User and add to sysadmin
Creating SUPER user
Container IP Address: <IP_ADDRESS>
Container Hostname  : bcaad19
Container Dns Name  : mybcvm.westeurope.cloudapp.azure.com
Web Client          : https://mybcvm.westeurope.cloudapp.azure.com/bcaad19/?tenant=default
Dev. Server         : https://mybcvm.westeurope.cloudapp.azure.com
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting bcaad19-default to <IP_ADDRESS> in container hosts file

Files:
http://mybcvm.westeurope.cloudapp.azure.com:8080/ALLanguage.vsix
http://mybcvm.westeurope.cloudapp.azure.com:8080/certificate.cer

Container Total Physical Memory is 16.0Gb
Container Free Physical Memory is 8.2Gb

Initialization took 70 seconds
Ready for connections!
Reading CustomSettings.config from bcaad19
Creating Desktop Shortcuts for bcaad19
Container bcaad19 successfully created
Because of Traefik, the following URLs need to be used when accessing the container from outside your Docker host:
Web Client:        https://mybcvm.westeurope.cloudapp.azure.com/bcaad19?tenant=default
SOAP WebServices:  https://mybcvm.westeurope.cloudapp.azure.com/bcaad19soap
OData WebServices: https://mybcvm.westeurope.cloudapp.azure.com/bcaad19rest
Dev Service:       https://mybcvm.westeurope.cloudapp.azure.com/bcaad19dev
Snapshot Service:  https://mybcvm.westeurope.cloudapp.azure.com/bcaad19snap
File downloads:    https://mybcvm.westeurope.cloudapp.azure.com/bcaad19dl

Use:
Get-BcContainerEventLog -containerName bcaad19 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName bcaad19 to get debug information about the container
Enter-BcContainer -containerName bcaad19 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName bcaad19 to remove the container again
docker logs bcaad19 to retrieve information about URL's again

Screenshots

Additional context

Thanks for the update.
Are you logging in with @cosmoconsult accounts?
Have you tried to remove the -singletenant?

Just tried but still getting the error.

Also tried successfully in a completely fresh CDX environment. App Registrations are identical between the two tenants.
Any ideas on which AAD settings we could try to compare?

Will try to repro this (Just came home from Directions NA)

We found it.
We had a policy where app registration permissions need consent when they are not in an allowed list.
So although the app registration showed that no admin consent is needed:

It only worked after we granted admin consent on all permissions.
Thanks anyway and enjoy your weekend after your many Direction Sessions :)

Thanks:-)


Using multiple InputMasks on a same page, only the last one works

Hi.
This behavior started from v1.0.16. Rolling back to v1.0.15 fixes it.
If I use two or more input mask on the same page, only the last one works as expected. Taking this minimal working example:
<InputMask id="id1" data-mask="00000000" validateOnKeyPress="false" @bind-Value="_someModel.Field1"/>
<InputMask id="id2" data-mask="00000000" validateOnKeyPress="false" @bind-Value="_someModel.Field2"/>

public partial class SomePage : ComponentBase
{
    private SomeModel _someModel = new();
}

public class SomeModel
{
    public string Field1 { get; set; } = "";
    public string Field2 { get; set; } = "";
}

The InputMask with id1 doesn't respect the mask, I can input letters, more than 8 characters length, etc.
Using only one InputMask also works just fine.

Hello, I apologize for the inconvenience, a new nuget package has just been published...


BigQuery: many new samples

Introduces a variety of sample snippets across the product.  ORC format, Parquet format, CMEK, schema manipulations.
Also, wraps common include block used by all samples in a no-op function, to ensure that samples have a consistent indent depth.

I forgot this was dependent on some changes to the client library that I haven't submitted yet.  Will ping this once those changes are in.

Dependent changes went in with https://code-review.googlesource.com/c/gocloud/+/31670


[pre-sve]TC-CNET4.4 Step 2 bad response to ScanNetworks (Scan Networks) from Ameba server

./chip-tool networkcommissioning scan-networks 1111 0 --Ssid 574E38303254 --Breadcrumb 1
the Sdid  specified matches the current network  read from
./chip-tool networkcommissioning read networks 1111 0
[1655488863.152775][41969:41974] CHIP:DMG: Received Command Response Data, Endpoint=0 Cluster=0x0000_0031 Command=0x0000_0001
[1655488863.152900][41969:41974] CHIP:TOO: Endpoint: 0 Cluster: 0x0000_0031 Command 0x0000_0001
[1655488863.153041][41969:41974] CHIP:TOO:   ScanNetworksResponse: {
[1655488863.153157][41969:41974] CHIP:TOO:     networkingStatus: 0
[1655488863.153252][41969:41974] CHIP:TOO:     wiFiScanResults: 0 entries
I would expect to see the network information for the connected network.
SHA 1e3f12041013dd71a53d035dbf2f6d142fc06c4e
Platform Ameba

this relates to https://github.com/project-chip/connectedhomeip/issues/19729

TC-CNET-4-4-2.log

@chrisbeach59, according to the test step, Step 2: "TH sends ScanNetworks Command to the DUT with SSID argument value as 'Userwifi_ssid' and Breadcrumb argument value 1"
I think this means you should pass in the ASCII value of ssid instead of hex. I have tested it with Ameba platform and got the expected result.
Yet, in the example verification step, the hex value is passed in, might be a mistake?


Fixing more relative import statements

I found them when working on making the package Py3 compatible.
Those relative import statements seem to work fine on Py2.7 too.

Coverage remained the same at 91.61% when pulling ed7d9ceb349f405ca40f43835a5fd6fc1beed98b on pklaus:relative-imports into 288ac27084b47166ac662402ea340d0aa25d8f56 on curtacircuitos:master.

Looks great.
Merged, Thanks.


Steps to download and install and offline version of python language server

Hi
I am using VSCODE with the Microsoft Python extension. For IntelliSense I wanted to use Microsoft-python language server (instead of default Jedi server)
Our project code base is IP protected and can only be accessed on Linux remote machines.
Hence we have VSCODE and python extension installed in our Linux environment, on these machines we don't have internet access.
Can someone please help me to learn how I can download and install an offline version of Microsoft python language server? so that I can use it with our VSCODE installation with python extension?
Today I am facing this error message :

To my knowledge, the extension downloads using VS Code's builtin proxy system to handle these cases. Have you configured a proxy to facilitate downloads, or are you saying you have no internet whatsoever?

@jakebailey
Yes since our environment is IP protective we are not allowed to access internet from these machines.
I can install something manually though (after moving the packages from non IP area to IP area)

I see. You can feasibly manually download and copy files by running VS Code on another machine with python and moving the languageServer.<version> folder to the other machine.

Though I will point out how odd it is that the extension is able to know which version to download, as that too is an internet access.

I see. You can feasibly manually download and copy files by running VS Code on another machine with python and moving the languageServer.<version> folder to the other machine.

So, this folder language server. will be inside the extensions folder ?

Inside the ms-python.python directory, yes.

Hi @jakebailey
Thanks for sharing the info, I followed the steps below to install the language server but looks like there is a failure at the start of the client.
Steps:

Downloaded the Python-Language-Server-win-x<IP_ADDRESS>.nupkg
Created the following directory called languageServer in the ~/.vscode/extensions/ms-python.python-2020.5.86806/
Unzipped the contents of the downloaded pkg into the directory from step 2 .
Changed the access of the following using chmod +x.
-rwxrwxr-x 1 msourabh users  73K Mar 20 16:00 Microsoft.Python.LanguageServer.pdb
-rwxrwxr-x 1 msourabh users 176K Mar 20 16:00 Microsoft.Python.LanguageServer.exe
-rwxrwxr-x 1 msourabh users 250K Mar 20 16:00 Microsoft.Python.LanguageServer.dll
Disabled jedi

Relaunched vscode ,

I am seeing the following messages in the output for python :

I am seeing the following message in output for python language server

Am I missing any steps ?

Your logs imply you're not on Windows.

Yes, that is correct this is not Windows. For Linux do I need to follow different steps>
I took reference from here: https://github.com/microsoft/python-language-server/issues/1698

Each build is platform specific; you'd want to get the linux one, or do my suggested method of letting the extension download the LS and copying it from one to the other (rather than a direct extraction and chmod).

I apologize for this silly question, but can I get a link or some info from where I can download the following package: Python-Language-Server-linux-x<IP_ADDRESS>.nupkg ? At present I am unable to install this from any of the linux machines

https://github.com/microsoft/vscode-python/blob/master/src/client/activation/languageServer/languageServerPackageService.ts
https://github.com/microsoft/vscode-python/blob/master/src/client/activation/languageServer/platformData.ts
URLs: https://github.com/microsoft/python-language-server/issues/1916. Specific version is not guaranteed to exist, we do not keep all versions and old ones are getting trimmed.
URL the same as Windows, package name construction by channel and platform see the code. Example
https://pvsc.blob.core.windows.net/python-language-server-stable/Python-Language-Server-linux-x<IP_ADDRESS>.nupkg

Closing.

Thanks I was able to download and install in IP protected area.
I am still facing issues related to IntelliSense I will open another issue ticket for that


Add domain youscope

@coolkrokogamer First stage of verification passed, official domain is groenehartscholen.nl as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://helpdesk.groenehartscholen.nl/

School official website URL: https://groenehartscholen.nl/lyceum/
The school street address, including city and country: Tolstraat 11, 2405 VS Alphen aan den Rijn in the Netherlands
An URL of a page on the official website where the school offers an IT-related long-term (>1 year) course: https://groenehartscholen.nl/lyceum/vakken/ -> Button "INFORMATICA"
A proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.:
They have the domain in their url for every subject on This  page. For informatica for example they have https://sites.google.com/youscope.nl/informatica
CSG Groene Hart is a secondary school in the Netherlands. The student are around 12 to 18 years old. Youscope.nl is the domain students and teachers on this school get. The teachers have [4 or 3<EMAIL_ADDRESS>and the students have [5<EMAIL_ADDRESS>Thanks for your response!

@coolkrokogamer  Thank you very much for the clarifications. Pull request merged.


alpha: remove QuaternionHelper

:tada: This PR is included in version 2.0.0-alpha.2 :tada:
The release is available on:

GitHub release
npm package (@alpha dist-tag)

Your semantic-release bot :package::rocket:


Increase the max allowed httpHeaderSize

According to https://stackoverflow.com/questions/64112724/how-to-increase-max-header-size-for-graphdb-standalone-server we could be modifying the default 4096 limit on HHTP-POST for triples to ingest via sparql through setting
graphdb.connector.maxHttpHeaderSize = 65536

This should be doable through inserting something like -Dgraphdb.connector.maxHttpHeaderSize=${GDB_MAX_HEADER:-65536} into the statement at the end of graphdb/kgap/entrypoint-wrap.sh (at https://github.com/vliz-be-opsci/k-gap/blob/main/graphdb/kgap/entrypoint-wrap.sh#L31)
this in combination with

introducing this env variable in dotenv-example and/or README
introducing setting it in the sample docker-compose.yml

Hopefully that covers the issue mentioned in https://github.com/vliz-be-opsci/py-RDF-store/issues/12

was solved through PR #20


img per infra + updated client versions

What this PR does / why we need it:
Images per Infrastructure and updated client version for kubectl, gardenctl and aliyun
Which issue(s) this PR fixes:
Fixes #37
Special notes for your reviewer:
Release note:

Images per Infrastructure and updated client version for kubectl, gardenctl and aliyun

Awesome @jfortin-sap , thank you.
I noticed some CLIs have versions, some not. What's our strategy? Explicit versions are better, because you then have reproducible builds, but then someone needs to update the CLIs regularly. Maybe you/SRE could do so? Maybe only for the core components and the Gardener extensions teams for the CLIs?
Ah, the latest kubectl is v1.20.4 and I believe it finally solves the problem with the managed fields, so we better get the version that has the fix for that in, please.

@vlerenc Can we merge? Updated kubctl version as requested, and created a separate task for explicit CLI versions https://github.com/gardener/ops-toolbelt/issues/44

Sure, whatever you think. I am not a reviewer here, so if you have tested it @jfortin-sap, please go ahead or ask your colleagues to review this PR.

@plkokanov Do you approve?

I noticed some CLIs have versions, some not. What's our strategy? Explicit versions are better, because you then have reproducible builds, but then someone needs to update the CLIs regularly. Maybe you/SRE could do so? Maybe only for the core components and the Gardener extensions teams for the CLIs?

@vlerenc I also think It would become too much maintenance if all of the versions have to be constantly updated.
Currently only tools installed via curl and git support versioning, however It shouldn't be a problem to extend others (pip, apt and whatever comes in the future) to also support a version flag. Then we could specify versions for things like the azure cli, gcp cli, openstack and so on.

I also think It would become too much maintenance if all of the versions have to be constantly updated.

Well, pinned versions are generally safer @plkokanov. How about doing that once every time we support a new Kubernetes version?

Makes sense, let me open a new issue to track this. We should also sync/update iaas cli versions with gardenctl versions.


[python] added plot_split_value_histogram function

@henry0312 can you review this PR? thanks!

Sure, but I'm on a trip now, so I'll review in two days 🙋♂
2019年4月30日(火) 23:19 Guolin Ke<EMAIL_ADDRESS>
@henry0312 https://github.com/henry0312 can you review this PR? thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/LightGBM/pull/2043#issuecomment-487970618,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADJ2GQJEPUO6J7KBOX7F3LPTBINJANCNFSM4G43U53Q
.

--
Tsukasa OMOTO
<EMAIL_ADDRESS>


rounded is not documented in Box

https://docs.nativebase.io/box uses rounded in the example but it does not appear to be documented anywhere as to what the values allowed are.
It appears to be pulling it from radii based on the source code.

Thanks for reporting this @trajano, we will add the props here https://docs.nativebase.io/utility-props#borders-radius with the next Docs update or release.

Thanks for Reporting this bug Rounded is now included in docs under utility-props.


[RESOURCE] Follow This

Resource title
Follow This
URL
https://www.follow-this.org/
Relevant tags (1 per line)
Finance
Collective Action
Organizations
Resource Summary
Follow This is a grassroots non-profit organization with over 10,000 members. Based in Amsterdam, the organization empowers responsible shareholders to push Big Oil to go green.
Relevance
No response
Other comments
No response

Looks good! Updated the description to add a bit more context


get a conformance certification for the library

https://openid.net/certification/rp_testing/
using the profiles:

basic
config
https://rp.certification.openid.net:8080/test_list

https://openid.net/certification/rp_testing/
http://openid.net/certification/rp_submission/
http://openid.net/wordpress-content/uploads/2016/12/Certification-Submission-Examples.pdf

Note that for each test, it’s normal to (1) do a WebFinger lookup, (2) retrieve the OP configuration, (3) perform dynamic registration, (4) send an authentication request using the chosen response_type, and (5) process the authentication response. When preparing to submit a certification result, you’ll also (6) capture an RP log file and/or screen shot documenting how your RP handled the response.

certification requests are sent, waiting for the conformance

done


Sync: Ripple(d) 1.10.0

High Level Overview of Change

Context of Change

Type of Change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Refactor (non-breaking change that only restructures code)
[ ] Tests (you added tests for code that already exists, or your new feature included in this PR)
[ ] Documentation update
[ ] Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
[ ] Release

API Impact

[ ] Public API: New feature (new methods and/or new fields)
[ ] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
[ ] libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
[ ] Peer protocol change (must be backward compatible or bump the peer protocol version)

#4320 https://github.com/XRPLF/rippled/commit/9346842eed474793823468db010ce3c6a4f20859
#4218 https://github.com/XRPLF/rippled/commit/b7f588b789d132cc8dc0acff05b781bee452e827
#4218 https://github.com/XRPLF/rippled/commit/c3acbce82dc5db8765cfc9dfeae36954cec0312f
#4218 https://github.com/XRPLF/rippled/commit/066f91ca07be9a31fad601844af057a0bd0b2508
#4370 https://github.com/XRPLF/rippled/commit/4f95b9d7a69e95e40126822514e5369fcd57fb1d (Conflict - Modified)
#4496 https://github.com/XRPLF/rippled/commit/e75cd4931378f7e0b94afdb9fdee8d45e4159531 (NO - Markdown)
#4472 https://github.com/XRPLF/rippled/commit/1f417764c3293adfd60e98c9d27da096f474d13d (NO - Pkg Manager)
#4489 https://github.com/XRPLF/rippled/commit/c5003969dee3fe7fbc1e0a14afab1ccbb8782d82 (Conflict - Not Modified)
#4501 https://github.com/XRPLF/rippled/commit/8d482d3557fb206eab5c7bbfc57216238b516d14
#4454 https://github.com/XRPLF/rippled/commit/436de0e03aaad76ca3326a4d5a048e5441d80422 (NO - Workflow)
#4508 https://github.com/XRPLF/rippled/commit/b21a05d465961d50bee7f7858aa2b6ed063c0315 (NO - Markdown)
#4510 https://github.com/XRPLF/rippled/commit/c7ef4c978345bacf3894545e7c9d09e9f8a7b0bb (NO - Workflow)
#4498 https://github.com/XRPLF/rippled/commit/67238b9fa66c5f99fe2a969b2a8851a4bca2c74f (NO Conan)
#4404 https://github.com/XRPLF/rippled/commit/78076a69038be72102bcf90076f69f79ce78987c (Conflict - Modified)
#4511 https://github.com/XRPLF/rippled/commit/629ed5c691fc454140a532d872c2420a93a6d564 (NO - Workflow)
#4447 https://github.com/XRPLF/rippled/commit/3620ac287e5a0b7206504a2032a6d4fb185bcf97
#4527 https://github.com/XRPLF/rippled/commit/ce997a6de8b82aa979632960bbf37efc44fd99bc
#4533 https://github.com/XRPLF/rippled/commit/32f8ae1af1f7ef34897baea355d8b2fc59aaf6a9 (Conflict - Modified)
#4535 https://github.com/XRPLF/rippled/commit/adf672ff83d473043485f8a48d39bb5ecbd892b5 (NO - README)
#4514 https://github.com/XRPLF/rippled/commit/adde0c2d11f60d0dff66e086ac32ff7847088a29 (NO - BUILD)
#4538/#4531 https://github.com/XRPLF/rippled/commit/f709311762f4a4c97dbc47748980242ae9d1c466
#4555 https://github.com/XRPLF/rippled/commit/c9a586c2437bc8ffd22e946c82e1cbe906e1fc40
#4558 https://github.com/XRPLF/rippled/commit/5644c8704fc2120509cfcb8c9c15f205d7344972
#4536 https://github.com/XRPLF/rippled/commit/5d011c7e6b399b988ad193d199e6e3dba92e49ad
#4565 https://github.com/XRPLF/rippled/commit/77c0a62a7472dbc4dc9ed97d3141840a3acb5ce8

There's 2 build errors that prevent this from being merged

I think we'll merge memdb first then this


Failed to load Controllers automatically on 1.5.0

If controller components are in different TS files, and can not be loaded  automatically.  Inlined controller class in server.ts  file is fine.

Yes. This is right. I just realized we are missing the entry-point documentation from the new docs (here).
The logic behind the entry-point is to accumulate all the references of your files in a single file which is the file you will run with deno. You can find this documentation in our legacy documentation here.

So let's say, in the picture you attached. You will create a entry-point file anywhere, in this file you will include all your references
import { ........ } from<EMAIL_ADDRESS>import { MyController } from "../MyController.ts" // for example
import { MyService } from "../MyService.ts" // for example

const controllers = [MyController];
const services = [MyService];

.....

new MandarineCore().MVC().run();

and then you would deno run that entry-point file. Let me know if there are further questions

@buttercubz Do you think this can be improved through the CLI or Mandarine's core itself? It's kind of hacky because Deno will need the imports anyway so I don't see any possible way with javascript & V8 to automatically detect the imports to use, but I think there's a work to be done here. Any thoughts @buttercubz ? Any thought is very well appreciated

@buttercubz Do you think this can be improved through the CLI or Mandarine's core itself? It's kind of hacky because Deno will need the imports anyway so I don't see any possible way with javascript & V8 to automatically detect the imports to use, but I think there's a work to be done here. Any thoughts @buttercubz ? Any thought is very well appreciated

We have already created a system to handle the dependencies without the use of the import map, we have that version in case the deno ends the support for the import map, maybe that same system can work in mandarine cli

@andreespirela
Can you detail the problem for me to implement a prototype in mandarine cli?

@buttercubz Thanks, will do!

@andreespirela  Got. I'm a Spring Boot Develper. :)   and  I think Mandarine.TS  will load all TS files from the src/main/mandarine and load the components automatically like Spring Boot App.    Any plan to support this feature?  For a Java developer, and I really like Mandarine.TS.  👍

@linux-china Great to hear that! Mandarine was mainly inspired on Spring boot but now we are taking a different approach to give it a Mandarine flavour. Yes, we want to support the auto-scan feature to reference all the needed files like you described, we are not sure yet how we will approach it because of the way Javascript  and specially the V8 engine work, but it’s definately on the roadmap.

Closing as no further comment was added and seems like the user could solve it with the mentioned indications


[SPARK-10100][SQL] Eliminate hash table lookup if there is no grouping key in aggregation.

This improves performance by ~ 20 - 30% in one of my local test and should fix the performance regression from 1.4 to 1.5 on ss_max.

Merged build triggered.

Merged build started.

Test build #41309 has started for   PR 8332 at commit 4dc723d.

Test build #41309 has finished for   PR 8332 at commit 4dc723d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41309/
Test PASSed.

LGTM. merging to master and branch 1.5.


Broken link

The "See Download and Setup" link in the readme is broken.

Already an existing issue #7989 with a pull request that fixes the issue: #7990.  Just waiting for a CLA to be signed.

Thanks for filing the bug @Bihaqo , and thanks for the redirect @ClarkZinzow !
Closing this out.


Tracking issues of OpenDAL WASM support

WebAssembly (WASM) is a modern binary instruction format for a stack-based virtual machine. It is designed as a portable compilation target for high-level languages like C/C++ and Rust, enabling deployment on the web for client and server applications. WASM aims to enable near-native performance for web applications, providing a more efficient and secure alternative to traditional JavaScript. Its key features include fast loading times, efficient execution, and the ability to run alongside JavaScript, allowing developers to build complex client-side web applications that perform better than those written solely in JavaScript.

By adding WASM support for opendal, we will allow users to use opendal in targets like wasm32-unknown-unknown. In a short word, users can use opendal in web browser!
We have addressed most blockers by https://github.com/apache/incubator-opendal/pull/3796 and adding a basic test for s3 https://github.com/apache/incubator-opendal/pull/3802. It's time for us to move forward.

This tracking issue is used to track our current progress of wasm support. Welcome to join in us!
Tasks
Add OPFS support
After opendal is ready for WASM, we can start to work for opfs now!

[ ] https://github.com/apache/incubator-opendal/issues/2442

Add support for more services
Please run the following command to check if opendal can build under wasm32 arch:
cargo build --target wasm32-unknown-unknown --no-default-features --features=services-s3

Visit https://github.com/apache/incubator-opendal/pull/3796 for possible fix or ask directly in this issue.
It's possible that some services can't support wasm at all, like fs. Please also provide your feedback here. We will collect them into a seperate doc.

[ ] alluxio
[ ] atomicserver
[x] azblob in https://github.com/apache/incubator-opendal/pull/3806
[ ] azdls
[ ] azfile
[ ] b2
[x] cacache (can't support)
[ ] cloudflare_kv
[ ] cos
[ ] d1
[ ] dashmap
[ ] dbfs
[ ] dropbox
[x] etcd (can't support)
[x] foundationdb (can't support)
[x] fs  (can't support)
[ ] ftp
[x] gcs (cant's support for https://github.com/apache/incubator-opendal/issues/3803#issuecomment-1868235037)
[x] gdrive in https://github.com/apache/incubator-opendal/pull/3808
[ ] ghac
[ ] gridfs
[x] hdfs (can't support)
[ ] http
[ ] huggingface
[ ] ipfs
[ ] ipmfs
[ ] libsql
[ ] memcached
[ ] memory
[ ] mini_moka
[ ] moka
[x] mongodb (can't support)
[x] mysql (can't support)
[ ] obs
[ ] onedrive
[ ] oss
[x] persy (can't support)
[x] postgresql (can't support)
[x] redb (can't support)
[x] redis (can't support)
[x] rocksdb (can't support)
[x] s3 (supported in #3802)
[ ] seafile
[x] sftp (can't support)
[x] sled (can't support)
[x] sqlite (can't support)
[ ] supabase
[ ] swift
[x] tikv (can't support)
[ ] upyun
[ ] vercel_artifacts
[ ] webdav
[ ] webhdfs

Testing

[ ] Integrate behavior test for wasm target

Existing issues

[ ] wasm-pack test --node doesn't work for s3_read_on_wasm
[ ] #3809

FS style target can supported by wasm32-wasi.
I will take handle of this

FS style target can supported by wasm32-wasi.
I will take handle of this

You start a new issue for this. It's out of the goal of this tracking issue.

For now, I think the service based on the TCP socket is not working because the wasm32-unknown-unknown is not support TCP yet, the wasm32-unknown-emscripten can support TCP socket but need some hackc on Server
The TCP service is some service like following bellow:

MySQL
PostgreSQL
MongoDB
Redis

Can we be used on wasmtime?

Can we be used on wasmedge?

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.
The WASI runtime is not support for the wasm-bindgen which is dependent by reqwest now.
If we want to support the wasi runtime, we may need the other HTTP client which is compatible with the WASI spec

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

Thanks, let's remove them. I just found that support wasm32-unknown-unkown is totally different thing from support wasm32-wasi. Lesson learnt.

For now, the memory service can both on wasi(means can run in wasmtime and wasmedge, tested) and wasm32. The fs services need remove the tokio::fs

For the record,
wasm-pack test --chrome --headless and wasm-pack test --chrome is both working.
But need some special S3 settings like
[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "POST",
            "PUT",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

incompatible crate found<EMAIL_ADDRESS>cargo tree -i ring
ring v0.17.5
├── jsonwebtoken v9.2.0
│   └── reqsign v0.14.6
│       └── opendal v0.43.0
├── rustls v0.21.9
│   ├── hyper-rustls v0.24.2
│   │   └── reqwest v0.11.22
│   │       ├── opendal v0.43.0
│   │       └── reqsign v0.14.6 (*)
│   ├── reqwest v0.11.22 (*)
│   └── tokio-rustls v0.24.1
│       ├── hyper-rustls v0.24.2 (*)
│       └── reqwest v0.11.22 (*)
├── rustls-webpki v0.101.7
│   └── rustls v0.21.9 (*)
└── sct v0.7.1
    └── rustls v0.21.9 (*)

When I tried to compile with services-gcs, error reported:

Let's mark gcs as unsupported since gcs will require jsonwebtoken (and ring) for JWT.

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: https://github.com/briansmith/ring/issues/657#issuecomment-509877297
the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

it seems surprising that ring would be failing in that way

Thanks for raising this up! I re-visit the gcs support on wasm, and confirmed that gcs is able to compile on wasm32-unknown-unknown. I will start a PR to address this.

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: briansmith/ring#657 (comment)
the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

Thank you for pointing that out!
I've tried to compile ring before but it kept failing. Finally, I realized that the llvm comes with macos doesn't have a backend of wasm. I reinstalled llvm and it compiled successfully. Sorry for the confusion!

Hi,
I am very interested in trying the wasm version of opendal for my project But I am not a professional rust developer though (I know something about javascript/typescript but none of rust), so I have some questions/ suggestions:

any doc to use the wasm version? by importing any library in js/typescript? any npm package?
would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .
would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.

Thank you so much and happy new year!

It is still in a early stage and has not yet reached the stage where it can be released as a lib.
Maybe you could take a look at OpenDAL node.js binding.

It is still in a early stage and has not yet reached the stage where it can be released as a lib.
Maybe you could take a look at OpenDAL node.js binding.

Hi @fyears, thank you for your interest! As @Young-Flash mentioned, our WebAssembly support is still in the early stages and not yet ready for practical use.

any doc to use the wasm version? by importing any library in js/typescript? any npm package?

Not yet. WASM support is still in active development, and we need to address existing bugs such as https://github.com/apache/incubator-opendal/issues/3810 before proceeding.
We haven't released it yet. (However, you can create a wasm library by running cargo add opendal and building it.)

would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .

We are using web-sys now, and I'm not sure if web-sys support this use case.

would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.

We will not support the fs service on wasm, which means there's no need to provide hooks either. We have a plan to support opfs: https://github.com/apache/incubator-opendal/issues/2442. If users require IndexedDB, we can offer native support for it.

in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Nice question. We want this as well, but I'm not sure how to implement it. We can re-visit this after we have build a usable opendal wasm lib.

Hi @fyears, thank you for your interest! As @Young-Flash mentioned, our WebAssembly support is still in the early stages and not yet ready for practical use.

any doc to use the wasm version? by importing any library in js/typescript? any npm package?

Not yet. WASM support is still in active development, and we need to address existing bugs such as https://github.com/apache/incubator-opendal/issues/3810 before proceeding.
We haven't released it yet. (However, you can create a wasm library by running cargo add opendal and building it.)

would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .

We are using web-sys now, and I'm not sure if web-sys support this use case.

would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.

We will not support the fs service on wasm, which means there's no need to provide hooks either. We have a plan to support opfs: https://github.com/apache/incubator-opendal/issues/2442. If users require IndexedDB, we can offer native support for it.

in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Nice question. We want this as well, but I'm not sure how to implement it. We can re-visit this after we have build a usable opendal wasm lib.

web-sys

After a quick search, it seems that it has a clear mapping to js web api. However I think the most dirty work comes to the data crossing boundaries: wasm (prepare the net calls) -> js (fetch hook) -> wasm (reading the result)
Let's wait for the stable wasm releases first... LOL
Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Interesting use case! I personally want OpenDAL to be usable in web browsers and Electron as well.

I've played with the HTTP operator and it works well. Except for the CORS problem mentioned before. It needs the support from server to add CORS header (like the S3 operator)

may i ask is it usable now?


Update instead to 3.0.0

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask's name and version.

Thank you for the contribution. It was merged directly as e104ac12bf0bb0da0f8061d96e8313bd4c125a09 to keep commit history cleaner. Your contribution is still credited to you.


Fix error judgement for MainFragmentLocatedRegion and MostlyUsedDataRegion in distributed plan

Description
The previous judgement of in ExchangeNodeAdder.java may be wrong, because MostlyUsedDataRegion may be ahead of MainFragmentLocatedRegion.

Codecov Report
Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (bb20b5f) 49.01% compared to head (7e8ddd8) 49.02%.
Report is 2 commits behind head on master.

Files
Patch %
Lines

...e/plan/planner/distribution/ExchangeNodeAdder.java
35.29%
11 Missing :warning:

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #11636   +/-   ##
=========================================
  Coverage     49.01%   49.02%
- Complexity    24898    24904    +6
=========================================
  Files          2815     2815
  Lines        176117   176128   +11
  Branches      21125    21129    +4
=========================================
+ Hits          86322    86342   +20
+ Misses        89795    89786    -9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix sitemap generation.

I'm not sure if this is the optimal solution. I definitely not like duplicating the route paths. Any ideas how to simplify this and make it less error-prone are welcome!
On a side note, our tests didn't catch this. Suggestions are welcome for this too.
Fixes #1125.

Yup, that was my idea too, but I didn't have the time to look into it more. I suspect we can.
Feel free to take a stab at it.


cmd/compile: compiler allows unreachable statements

I noticed this in some badly re-factored complex code, but it reduces nicely into this toy program. Basically the compiler allows unreachable statements that could be detected statically at compile time. It would be nice if code like this failed to compile.
package main

import "fmt"

func main() {
	fmt.Println("foo", foo())
}

func foo() int {
	fmt.Println("now you see me")
	return 0
	fmt.Println("now you don't")
	return 1
}

While questionable, that's technically valid Go 1 code, so it has to be accepted by the compiler.  It's also already caught by cmd/vet:
$ go vet issue12920.go
issue12920.go:12: unreachable code

Let me add that to try to ensure that all Go compilers agree on which programs can be compiled, the notion of an unreachable statement is actually defined by the language spec: https://golang.org/ref/spec#Terminating_statements .

Permitting early returns helps debugging a large function where you might
want to quickly disable the last portion by putting in a return.
The alternative of adding an if 0 (which also introduces deadcode) or
commenting out/deleting aren't as easy.


Network roundtrips doubled for API requests

We have a SPA that uses pop-ups to establish web sessions (tracked in first-party cookies) to other user-defined domains and then uses CORS XHR/Fetch POST requests to execute queries on those remote systems.
In trying to enable the Storage Access API, we have created an iframe that requests access.  However, the access request is scoped only to that specific iframe.
Somehow we need the ability to request access for the many network requests we will be making.
Will Storage Access Headers solve this?
On the surface it does look like it would functionally allow things to work by returning the correct headers.  However, the requirement to double every network request (each one being first sent without cookies and then with cookies) will have terrible performance consequences particularly when the latency between browser and data source can be higher than 300ms. (e.g. 5 sequential network request = 1.5 seconds of extra waiting).
Two possible changes

Automatic Storage Access for CORS

The best option would be to address this by considering CORS to automatically request storage access for all API calls if configured with Access-Control-Allow-Credentials?  The single OPTIONS request made at the beginning of the CORS workflow should be sufficient to determine that the data source requires cookies, and every API call thereafter should automatically be granted storage access if there is already a pending grant (i.e. created via an iframe calling requestStorageAccess).

Storage Access Headers Once per Window for API calls

As a compromise, the storage access headers could be required only for the first request sent to the given target API - and all further requests to the same top-level domain would benefit from the request.

Thanks for the feedback @petipp, I discussed this with @cfredric and we think that both suggested alternatives might be viable, but would like to explore the second option first as it gives 3p embed developers a little more control over consistency and security of their apps.
cc @bvandersloot who was previously arguing for a closer integration with CORS and @arturjanc who has opinions on security things

@johannhof Can you outline in a little more detail how the "second option" would work here? Would we store the fact that a particular URL opted into receiving credentialed requests by setting the Activate-Storage-Access header, and avoid the extra round trip on subsequent requests to that URL from the same document? (I think this would have to be scoped to a document rather than window because we don't want other cross-site content embedded in the same window to inherit the capability.)
Re: "automatic storage access for CORS", Fetch already has the credentials toggle that lets the caller control whether cookies will be attached to the request, and requires server opt-in by setting Access-Control-Allow-Credentials: true and Access-Control-Allow-Origin. So I imagine it wouldn't be difficult to figure out how to integrate with CORS if that's what we decided to pursue.

@arturjanc Yeah we're coming up with a new proposal right now, in my mind this might be an explicit developer opt-in ala  Activate-Storage-Access: Sticky, we can discuss the desired scope of it :)
Right, I don't think it's hard to limit this access to CORS request, we're already doing that with RWS / rSAFor today. I don't think that CORS in itself is sufficient for opt-in though, given the lack of preflights for most practically used requests.

@johannhof   That's fair.  FWIW, CORS would be sufficient for us because our API is POST-based so there's always a preflight.

Yes, definitely sure.  This is from the OPTIONS call:

Ah, yeah, that makes sense!

Hi @petipp,
We're thinking of the scope of the "stickiness" discussed above.
Making the activation sticky for an entire site or origin would be too broad probably – that would mean that a single endpoint could downgrade security for all other endpoints, potentially maintained by completely different teams.
On the other extreme, if we key the activation for the exact URL, that could result in a lot of retries if you use a lot of unique query params.
Would it be reasonable for your usecase to maintain stickiness for the exact URLs? including the query params and/or URL fragment

Hi @sandormajor
In the typical (optimal) case, our initial load sends two requests to different URLs under a common path (a/b/C, a/b/E).  User interaction (and less-optimized initial load cases) will make many more requests using the second path.
So exact matches would be okay over time - but initial load is a very critical workflow for the user's perception of performance.
Would it be possible to aggregate them by a common path?  Would that then require an additional (optional) to be returned with Activate-Storage-Access?  e.g. something like this:
Activate-Storage-Access: retry; allowed-origin="https://foo.bar"; allowed-scope="a/b"

@petipp Would you be willing to share a typical flow of requests for your use case with real(-ish) looking URLs?
We're trying to understand whether scoping should support wildcards (e.g. so that anything under /a/b would receive credentialed requests) or if you could, for example, explicitly list individual endpoints for which storage access would apply (e.g. allowed-scope=/a/b/endpoint1,/a/b/endpoint2,...). The latter is more cumbersome, but also reduces the potential for misconfigurations (say, a developer setting allowed-scope=/) which would expose the application to cross-site vulnerabilities.

@arturjanc
This would be what we would typically do:
GET https://host:port/a/b/service/v2/GetServerInfo
POST https://host:port/a/b/service/v2/GetResponse
POST https://host:port/a/b/service/v2/GetResponse
POST https://host:port/a/b/service/v2/GetResponse
So if we could make the initial GET handle the SAH permissions for the POST requests then this would be perfect.  So for our case a list of specific other endpoints would also work fine.  I would prefer if the paths could be relative (e.g. ./GetResponse) just in case a reverse proxy is used.

Hi @petipp
Please take a look at https://github.com/privacycg/storage-access-headers/pull/22 if it would work for you

Hi @sandormajor
Sorry for the delay.  Yes, this looks like it would work for us.  Though we currently only need to be able to provide a single URL, it would also be good to know that it would accept multiple URLs (comma-separated?).  Is that included in the proposal?  It didn't seem to be mentioned in the document.

Hi @petipp,
Thank you for the feedback! Yes, it is going to be an inner-list which has a space-separated syntax like this:
reuse-for=("/baz.html" "https://embeddee-origin.example/foo/bar")


Change the namespace

The namespace should be changed from Tymon to something new.
In my opinion PHPOpenSourceSaver would be a bit long. Probably PHPOSS would be better. Feel free to post your ideas.

@Messhias I have changed the namespace and created a pull request. I have added you as reviewer :)

I just. saw it, but I cannot merge it unfortunately :(

@Messhias Can you check https://github.com/PHP-Open-Source-Saver/jwt-auth/invitations if you had accepted the invitation? Additionaly I have added you as member, without the team. Probably you need to relogin into github to get it work.

I took a look right now and it's showing me as "admin", I'll give it a try
to login again.
Best Regards,
Fabio William Conceição
Remote: https://remote.com/fabiowilliam https://remote.com/fabiowilliam
Tel/Whastapp: +351 93 212 1477
Skype: fabioo.william.conceicao
LinkedIn:
https://www.linkedin.com/in/fabio-william-conceição-379b9823/
https://www.linkedin.com/in/fabio-william-conceição-379b9823/
*Github: *https://github.com/Messhias/ https://github.com/Messhias/
Upwork: https://www.upwork.com/o/profiles/users/_~0126d10487b9843f68/
https://www.upwork.com/o/profiles/users/_~0126d10487b9843f68/
Em ter., 21 de set. de 2021 às 10:38, eschricker @.***>
escreveu:

@Messhias https://github.com/Messhias Can you check
https://github.com/PHP-Open-Source-Saver/jwt-auth/invitations if you had
accepted the invitation? Additionaly I have added you as member, without
the team. Probably you need to relogin into github to get it work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PHP-Open-Source-Saver/jwt-auth/issues/3#issuecomment-923808487,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAI3QQ5AEKU6RNGA4W6ZNWTUDBG7TANCNFSM5EGYLILQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


saw-tools: update license

Motivation for this change
When trying to use saw package, I get:

Package ‘saw-tools-0.1.1-20150731’ in /nix/store/brydnggk5q9dr7d226fnda0yalipk7q7-source/pkgs/applications/science/logic/saw-tools/default.nix:54 has an unfree license (‘unfreeRedistributable’), refusing to evaluate.

However, the project is actually licensed with BSD 3 (https://github.com/GaloisInc/saw-script/blob/master/LICENSE)
Things done
Change license of nix expression to reflect actual license of project.

Please amend the commit and PR title to follow https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#submitting-changes

Please amend the commit and PR title to follow https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#submitting-changes

Thank you Ben for the pointers.
I updated as requested.

Thank you Ben for the pointers.
I updated as requested.


Owner- and group-rules failing in V2 search resolvers

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
14
Amplify CLI Version
7.6.26
What operating system are you using?
Mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
Custom Cognito attributes
Amplify Categories
api
Amplify Commands
Not applicable
Describe the bug
When making a search query with owner- or group rules, the query is failing and not returning any results.
For the ForeignNational model below, a user with a custom:person_id should be able to read any ForeignNational whose relatives field includes the claim.
But here is a user with person_id of 9af7... searching for his relatives and it’s empty. In the same query, I'm doing a get query for a known relative Zdf... to show that the user does indeed have rights to their record..

Here is the identical query made as a member of the Admins group, showing the expected response:

In the Query.searchForeignNationals.req.vtl resolver that the CLI generates, I can tell that it is making the following query to OpenSearch:
  "bool": {
    "must": [
      {
        "bool": {
          "should": [
            {
              "terms_set": {
                "companyID.keyword": {
                  "terms": ["___xamznone____"],
                  "minimum_should_match_script": { "source": "1" }
                }
              }
            },
            {
              "terms_set": {
                "id.keyword": {
                  "terms": ["9af7e450-a8c2-4808-bfef-dfb878ba628d"],
                  "minimum_should_match_script": { "source": "1" }
                }
              }
            },
            {
              "terms_set": {
                "relatives": {
                  "terms": ["9af7e450-a8c2-4808-bfef-dfb878ba628d"],
                  "minimum_should_match_script": { "source": "1" }
                }
              }
            },
            {
              "terms_set": {
                "relativeAdmins": {
                  "terms": ["9af7e450-a8c2-4808-bfef-dfb878ba628d"],
                  "minimum_should_match_script": { "source": "1" }
                }
              }
            }
          ]
        }
      },
      { "match": { "relatives": "9af7e450-a8c2-4808-bfef-dfb878ba628d" } }
    ]
  }
}

Perhaps this filter is malformed? I don't know enough about Query DSL to diagnose.
Expected behavior
Search queries should correctly respect the owner and group rules in models.
Reproduction steps

Create model with owner and group auth rules
Search for records employing those rules

GraphQL schema(s)

# Put schemas below this line

type ForeignNational implements Person
  @model(
    queries: { get: "getForeignNational", list: null }
    subscriptions: null
  )
  @searchable
  @auth(
    rules: [
      {
        allow: groups
        groups: ["Admins", "Team", "Employers"] # TODO: Remove employer
      }
      {
        allow: groups
        groupsField: "companyID"
        groupClaim: "custom:company_access"
        operations: [read, update, delete]
      }
      {
        allow: owner
        ownerField: "id"
        identityClaim: "custom:person_id"
        operations: [read, update, delete]
      }
      {
        allow: owner
        ownerField: "relatives"
        identityClaim: "custom:person_id"
        operations: [read]
      }
      {
        allow: owner
        ownerField: "relativeAdmins"
        identityClaim: "custom:person_id"
        operations: [read, update, delete]
      }
    ]
  ) {
  id: ID!
  firstname: String
relatives: [String!]
  relativeAdmins: [String!]
...
}

Log output

# Put your logs below this line

Additional information
No response

Here is my use cases and observations.
So I have a model like this:
type Asset
    @model(subscriptions: null)
    @auth(
        rules: [
            # Users of the owner org can do all the operations
            { allow: groups, groupsField: "org_owner" }
            # Users in group specified in "set_name" field will have read only access
            { allow: groups, groupsField: "set_name", operations: [read] }
        ]
    ) {
    id: ID! @primaryKey
    org_owner: String @index(name: "byOrganization", queryField: "assetsByOrganization")
    set_name: String @index(name: "byContentSetName", queryField: "assetsByContentSet")
    ItemName: String @default(value: "No Name")
}

The authorization rules works fine for all the operations.
But
As you can see I set indexes on fields used for the groups. When doing operations with assetsByOrganization and assetsByContentSet there is a problem with implementing the auth rules. That is because the auth rules are implemented as filters in auth resolver.
Example for simple listing:
#if( !$isAuthorized )
  #set( $authFilter = [] )
  #set( $role0 = $util.defaultIfNull($ctx.identity.claims.get("cognito:groups"), []) )
  #if( $util.isString($role0) )
    #if( $util.isList($util.parseJson($role0)) )
      #set( $role0 = $util.parseJson($role0) )
    #else
      #set( $role0 = [$role0] )
    #end
  #end
  #if( !$role0.isEmpty() )
    $util.qr($authFilter.add({ "org_owner": { "in": $role0 } }))
  #end
  #set( $role1 = $util.defaultIfNull($ctx.identity.claims.get("cognito:groups"), []) )
  #if( $util.isString($role1) )
    #if( $util.isList($util.parseJson($role1)) )
      #set( $role1 = $util.parseJson($role1) )
    #else
      #set( $role1 = [$role1] )
    #end
  #end
  #if( !$role1.isEmpty() )
    $util.qr($authFilter.add({ "set_name": { "in": $role1 } }))
  #end
  #if( !$authFilter.isEmpty() )
    $util.qr($ctx.stash.put("authFilter", { "or": $authFilter }))
  #end
#end

You can see the auth resolver adding those rules for both org_owner and set_name group fields
It results in this filter statement in DynamoDB query (user is in group called std_lib in my case):
{
    "expression": "(contains(:or_0_org_owner_in, #org_owner) OR contains(:or_1_set_name_in, #set_name))",
    "expressionNames": {
        "#org_owner": "org_owner",
        "#set_name": "set_name"
    },
    "expressionValues": {
        ":or_0_org_owner_in": {
        "L": [
            {
            "S": "std_lib"
            }
        ]
        },
        ":or_1_set_name_in": {
        "L": [
            {
            "S": "std_lib"
            }
        ]
        }
    }
}

But for the "indexed" queries, 1 of the group fields become a primary key, which cannot be used in filter expression to DynamoDB. So I have noticed this field is just not being added in auth resolver (generated automatically) which makes the DynamoDB request valid BUT the authorization rules are now broken and you often get not items from DB because now there is no the second part of the filtering expression (there is no "OR" now).
As a workaround I have modified the auth resolver specifically for my assetsByContentSet case:
$util.qr($ctx.stash.put("hasAuth", true))
#set( $isAuthorized = false )
#if( $util.authType() == "User Pool Authorization" )
    #set ($userGroups = $util.defaultIfNull($ctx.identity.claims.get("cognito:groups"), []))
    #set( $authFilter = [] )
    #set ($targetGroup = $ctx.args.set_name)
    #set ($userIsInTargetGroup = $userGroups.contains("${targetGroup}") )
    #if( $userIsInTargetGroup )
        #set( $isAuthorized = true )
    #else
        #if( !$userGroups.isEmpty() )
            $util.qr($authFilter.add({ "org_owner": { "in": $userGroups } }))
        #end
    #end
    #if( !$authFilter.isEmpty() )
        $util.qr($ctx.stash.put("authFilter", { "or": $authFilter }))
    #end
#end
#if( !$isAuthorized && $util.isNull($ctx.stash.authFilter) )
    $util.unauthorized()
#end
$util.toJson({"version":"2018-05-29","payload":{}})

Here I knowthat there will be always a required argument set_name and a user can be in a group with the same name, I can authorize those users right away without any filtering. Otherwise I add the filtering for the second group field org_owner.
I think this approach can be somehow generalized for any group filed (array or string) that happen to be a primary key when generating auth resolvers.
Even before I wen into this issue with indexes, there is another fail case I hit. Initially, I had no indexes but one of my group fields was also a sort key of the table. And there was a similar problem regarding the authorization rules.

So my search appears to work if I rewrite the resolver to generate match queries instead of terms_set, such as the following instead of the filter pasted above:
{
  "bool": {
    "must": [
      {
        "bool": {
          "should": [
            { "match": { "companyID.keyword": "___xamznone____" } },
            {
              "match": { "id.keyword": "9af7e450-a8c2-4808-bfef-dfb878ba628d" }
            },
            {
              "match": { "relatives": "9af7e450-a8c2-4808-bfef-dfb878ba628d" }
            },
            {
              "match": {
                "relativeAdmins": "9af7e450-a8c2-4808-bfef-dfb878ba628d"
              }
            }
          ]
        }
      },
      { "match": { "relatives": "9af7e450-a8c2-4808-bfef-dfb878ba628d" } }
    ]
  }
}

But something tells me match is going to have unintended consequences here.

Hey @lseemann :wave: I was able to successfully reproduce this issue using the following users:

calling from the user:

calling from an admin:

Marking as a bug 🙂


[BUG] Upgrade elasticsearch-oss to 7.9.1 & ODFE to the latest version (1.10.1)

Describe the bug
During testing I found this regression bug in Elasticsearch v7.8.0. It was fixed in ES 7.8.1, see PR.
To Reproduce
Steps to reproduce the behavior:
It's not possible to upgrade only Elasticsearch without Open Distro for Elasticsearch, for example:
apt-get install opendistro-alerting
The following packages have unmet dependencies:\n opendistro-alerting : Depends: elasticsearch-oss (= 7.8.0) but 7.8.1 is to be installed
E: Unable to correct problems, you have held broken packages.

Expected behavior
Successful installation.
Config files
No.
OS (please complete the following information):

OS: All

Cloud Environment (please complete the following information):

Cloud Provider: All

Additional context
ODFE just released new version (1.10.1), see here.
Originally posted by @to-bar in https://github.com/epiphany-platform/epiphany/issues/1140#issuecomment-702062434

This task fixes issue introduced in #1140

On block because of https://github.com/epiphany-platform/epiphany/issues/1741

With PR #1718 we have introduced Jinja template for jvm.options file. This file defines JVM heap size (Xmx). The default is 1 GB but Elasticsearch doc says:

When moving to production, it is important to configure heap size to ensure that Elasticsearch has enough heap available.

So it is very likely that one could increase the JVM heap size. But for epicli upgrade we do not support defining custom values for new configuration settings. The setting specification.jvm_options has been introduced in v0.8.0rc1.
Current code (for upgrades) always uses the default value, i.e 1 GB (see PR #1763), so any customized value is overwritten during the upgrade. To solve this issue we can try to get the existing setting from remote host (from jvm.options file) to preserve the custom value instead of overwriting it with the default.
More info: https://www.elastic.co/guide/en/elasticsearch/reference/7.9/heap-size.html
This JVM heap size issue can be solved in separate PR.

@przemyslavic @to-bar Was it tested? If so, please move it to the correct column.

Tested mainly by @to-bar.


PyPREP excludes more channels from RANSAC than MATLAB PREP

After fixing the correlation, deviation, and channel picks code to match MATLAB PREP, I finally started trying to dig into differences in RANSAC interpolation between implementations. In the process, however, I noticed that the channel picks still weren't lining up between PyPREP and MATLAB PREP despite my earlier efforts.
Some deeper digging revealed that this is because PyPREP and MATLAB PREP are starting with different pools of "good" channels: PyPREP with 58 and MATLAB PREP with 60! Apparently, MATLAB PREP doesn't exclude channels bad by HF noise prior to RANSAC: only channels bad by NaN, flat, deviation, correlation, or dropout (note: NaN and flat channels already removed earlier in code):
%% Bad so far by amplitude and correlation (take these out before doing ransac)
noisyChannels = union(noisyOut.noisyChannels.badChannelsFromDeviation, ...
    union(noisyOut.noisyChannels.badChannelsFromCorrelation, ...
          noisyOut.noisyChannels.badChannelsFromDropOuts));

Presumably this is because RANSAC applies a 50 Hz lowpass filter before doing correlations anyway, so HF noise wouldn't necessarily be a priori grounds for exclusion. Does this make sense, or does it look like a bug in the MatPREP code? Is this a behaviour we'd like to mimic in PyPREP?

Apparently, MATLAB PREP doesn't exclude channels bad by HF noise prior to RANSAC: only channels bad by NaN, flat, deviation, correlation, or dropout (note: NaN and flat channels already removed earlier in code)

interesting, I would have assumed all channels previously identified as bad would be excluded.

Presumably this is because RANSAC applies a 50 Hz lowpass filter before doing correlations anyway, so HF noise wouldn't necessarily be a priori grounds for exclusion.

that makes sense to me 🤔 is there anything on this in the paper?

is there anything on this in the paper?

I checked, and the paper gives no indication that this is happening:

We use the RANSAC (random sample consensus) method (Fischler and Bolles, 1981) to select a random subset of (so far) good channels to predict the behavior of each (excluded from subset) channel in small non-overlapping time windows (5 s by default). The implementation is based on functions in BCILAB (Kothe and Makeig, 2013).
The bad-by-RANSAC channels behave in a manner poorly predicted by the other channels. Before applying RANSAC, we remove channels designated as noisy by other methods.

From that it sounds like HF channels would be dropped too. However, the HF noise detection procedure is only explained in the paragraph after RANSAC, so that might suggest it's not meant to be part of the pre-exclusion process. Also, I checked the git blame for those lines and they've been that way since at least 2014, with the only notable change being the addition of bad-by-dropout channels at some point ~6 years ago.
My personal instinct would be to side with the code here, since the rationale that they're being low-pass filtered at 50 Hz before RANSAC anyway (and haven't failed any of the other bad-by criteria) makes sense. The HF noise channels are still included in the final NoisyChannels output so they would still get ignored when doing the actual re-referencing. What do you think?

My personal instinct would be to side with the code here, since the rationale that they're being low-pass filtered at 50 Hz before RANSAC anyway (and haven't failed any of the other bad-by criteria) makes sense. The HF noise channels are still included in the final NoisyChannels output so they would still get ignored when doing the actual re-referencing

I agree with this instinct. In any case after we manage to get really close to the original PREP we can start asking the questions of improving further the original technique. But as of now I think the goal is to match as close as possible the original PREP, right?. Unless is a really weird mistake with no explained rationale yet (like the weird median calculation jasmainak found) , I think we should side with the code.

okay, thanks both - I agree to side with the matprep code then :+1:


Keybinding Copy not working - v 0.191.0

Issue in atom/atom
https://github.com/atom/atom/issues/6382#issuecomment-93463679

Thank you for reporting. I may fixed problems and published v0.6.0.
Please try it? :)


Fix dependency vulnerabilities

Resolve 3 dependency vulnerabilities (1 low, 1 moderate, 1 high).

You are welcome @kmartens27


Unable to open the notebooks

I get the below error:
Unreadable Notebook: /Julia/DataFrames_JuliaCon2020/advanced.ipynb NotJSONError('Notebook does not appear to be JSON: '\n\n\n\n\n\n<html lang="...')

You probably want the actual .ipynb file, rather than the Web page: https://raw.githubusercontent.com/bkamins/JuliaCon2020-DataFrames-Tutorial/master/indexing_part2_advanced.ipynb

Dang! Thanks!


Change stream/record image without changing preview

You can use both without preview.
But this will change rotation of all (preview if you have it, video record and streamed video). Only change rotation of streamed video is not supported.
Originally posted by @pedroSG94 in https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/707#issuecomment-713374230
Changing streamed video without changing preview is now supported?
I'm using sensorManager and prepareVideo to set the sensor rotation, but I want to keep the preview unchanged, so I can keep it locked in portrait mode.

I explained how to implement this feature here:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/531#issuecomment-963531623
If you want use a service as declared here:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/531#issuecomment-964735702
No, It isn't possible because change glinterface is not supported as I declared in the first link (read last line of limitations).
For now, you have 3 options: Use the first use case (without service), develop it yourself or wait for the feature.