S3 datapipes

Changes

Added S3FileLister and S3FileLoader IterDataPipes.
Added pybind11 build for s3 io cpp files and python scripts.

TODO

[x] clean up setup files and link pybind11 in CMAKE_PREFIX automatically.
[x] remove aws-cpp-sdk dependency at build with BUILD_S3 env var & pop exceptions when missing dependencies at usage.
[x] new api changes for list_files.
[x] clean up cpp files (naming, new structure, new logic etc.)
[x] expose timeouts, regions.
[x] thorough tests

[x] different correct usage: bucket (with or without / at last), folder (with or without / at last), prefix, item.
[x] different incorrect usage: non-existing files, wrong s3 urls, etc.
[x] region changes
[x] choice of public datasets

[x] benchmarks

[x] performance test

[x] README.md

[x] user guide & recommendations
[x] dependencies

Can we get this into the torch 1.11.0 release?

Can we get this into the torch 1.11.0 release?

TorchData is going to do a release after torch 1.11 (March maybe). If this PR is landed before that, we can squeeze it into TorchData release

Note that, to use S3 IO, we'll need pybind11 and "aws-sdk-cpp:s3;transfer" installed. We also need the BUILD_S3 env var set to "ON" to trigger build.

Note that, to build torchdata with S3 IO, we'll need pybind11 and "aws-sdk-cpp:s3;transfer" installed. We also need the BUILD_S3 env var set to "ON" to trigger build.

If so, we need to add pip install pybind11 as well to run CI correctly. But, I am not sure how we are gonna install "aws-sdk-cpp:s3;transfer"?

how we are gonna install "aws-sdk-cpp:s3;transfer"

I've written a README on how to install dependencies and build and use S3 IO datapipes. torchdata/datapipes/iter/load/README.md

git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
cd aws-sdk-cpp/
mkdir sdk-build
cd sdk-build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3;transfer"
make
make install # may need sudo

Add a separate step before building torchdata should be fine. https://github.com/pytorch/data/blob/e53afb723de8ce761c0fabc8c8f5c066e83cb50a/.github/workflows/ci.yml#L47-L48

git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
cd aws-sdk-cpp/
mkdir sdk-build
cd sdk-build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3;transfer"
make
make install # may need sudo

Add a separate step before building torchdata should be fine.
https://github.com/pytorch/data/blob/e53afb723de8ce761c0fabc8c8f5c066e83cb50a/.github/workflows/ci.yml#L47-L48

Added another step to install aws-sdk-cpp.
I wonder why install test requirements before building instead of after building and before tests?

Added another step to install aws-sdk-cpp.
I wonder why install test requirements before building instead of after building and before tests?

You need to compile dependency before building TorchData, am I right? Except, you want to do a more fancy feature to compile aws and provide S3 IO whenever such DataPipe is used. (I think it's doable but needs to do some research, we should land this PR first then figure it out).

Added another step to install aws-sdk-cpp.
I wonder why install test requirements before building instead of after building and before tests?

You need to compile dependency before building TorchData, am I right?

I meant that I noticed the following step (already existing in ci) is before building torchdata:
      - name: Install test requirements
        run: pip3 install expecttest fsspec iopath==0.1.9 numpy pytest rarfile

Should we move it to after building torchdata and before testing?

CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage

Looks like the CI still doesn't install ninja before installing torchdata. Any idea? @ejguan?

Without SDK installed any usage of S3 pipes fails with
#17 [13/13] RUN python test.py
#17 sha256:a0ceb2208da477f7ae261e437253fd7fe4763562b03c7bcce76b1d46b57163b5
#17 1.951 /usr/local/lib/python3.8/dist-packages/torchdata-0.3.0a0+2627793-py3.8-linux-x86_64.egg/torchdata/_extension.py:30: UserWarning: torchdata C++ extension is not available.
#17 1.951   warnings.warn("torchdata C++ extension is not available.")
#17 1.952 Traceback (most recent call last):
#17 1.952   File "test.py", line 8, in <module>
#17 1.952     s3_lister_dp = S3FileLister(IterableWrapper(input), region="us-west-2")
#17 1.952   File "/usr/local/lib/python3.8/dist-packages/torchdata-0.3.0a0+2627793-py3.8-linux-x86_64.egg/torchdata/datapipes/iter/load/s3io.py", line 29, in __init__
#17 1.952     self.handler = torchdata._torchdata.S3Handler(request_timeout_ms, region)
#17 1.952 AttributeError: module 'torchdata' has no attribute '_torchdata'
#17 ERROR: executor failed running [/bin/sh -c python test.py]: exit code: 1

Should be nice error instead

Should be nice error instead

Hi @VitalyFedyunin, would you recommend we raise a ModuleNotFoundError exception, similar to what is done here?
Something like this?
--- a/torchdata/datapipes/iter/load/s3io.py
+++ b/torchdata/datapipes/iter/load/s3io.py
@@ -24,6 +24,11 @@ class S3FileListerIterDataPipe(IterDataPipe[str]):
     """

     def __init__(self, source_datapipe: IterDataPipe[str], length: int = -1, request_timeout_ms=-1, region="") -> None:
+        if not hasattr(torchdata, '_torchdata'):
+            raise ModuleNotFoundError(
+                "Torchdata must be built with BUILD_S3=1 to use this datapipe."
+            )
+
         self.source_datapipe: IterDataPipe[str] = source_datapipe
         self.length: int = length
         self.handler = torchdata._torchdata.S3Handler(request_timeout_ms, region)

@ejguan , let's make the mergins process faster and catch PyTorch 1.11.0 release.
The current issues are:

Windows requires a different set of commands to install aws-cpp-sdk (the final step also requires administrative privileges). How should we do that?
The style lint is failing for weird reasons un-related to torchdata, and mypy lint is complaining about the inevitable torchdata._torchdata usage. Shall we set to ignore the mypy lint with some mypy.ini settings?
Building fails for MacOS py37/38, Ubuntu py37, which I can't reproduce. I tried to build in Ubuntu py37 env, and my build and test both succeeded. Could this be a resource issue?
The test failures for the remaining MacOS py39, Ubuntu py38/39 are also weird, because I tested locally and the ValueError should be raised to pass the tests. I changed it to BaseException to see if it catches the error.

Please help us speed up the process, as we all want to include S3 IO datapipes in the coming torchdata release. Thank you!

@ejguan Could you please trigger the CI again to check if the mypy error is cleared, and the correct python versions are found at building? Thanks.

The test failures for the remaining MacOS py39, Ubuntu py38/39 are also weird, because I tested locally and the ValueError should be raised to pass the tests. I changed it to BaseException to see if it catches the error.

Can I have a code pointer which Error you are expecting?

The logic for error is:

passing in the input url (incorrect in this test) to list the files:
https://github.com/ydaiming/data/blob/s3-datapipes/torchdata/csrc/pybind/S3Handler/pybind.cpp#L22
https://github.com/ydaiming/data/blob/s3-datapipes/torchdata/csrc/pybind/S3Handler/S3Handler.cpp#L431
when parsing the incorrect input, invalid_argument will be thrown.
https://github.com/ydaiming/data/blob/s3-datapipes/torchdata/csrc/pybind/S3Handler/S3Handler.cpp#L435
https://github.com/ydaiming/data/blob/s3-datapipes/torchdata/csrc/pybind/S3Handler/S3Handler.cpp#L139
Then pybind11 will throw ValueError on the PyThon side.
https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html

The tests passed in my environment, I'm not sure what's the different in the CI system.

Windows requires a different set of commands to install aws-cpp-sdk (the final step also requires administrative privileges). How should we do that?

What do you mean different set of commands? Can you list the command you want to use? IIRC, you can specify shell: bash in the workflow to run the command

For Ubuntu and MacOS, the commands are as following, and already added:
          git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
          cd aws-sdk-cpp/
          mkdir sdk-build
          cd sdk-build
          cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3;transfer"
          make
          make install

while on Windows, the following commands are used:
git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
mkdir sdk_build
cd sdk_build
cmake ..\aws-sdk-cpp -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3:transfer"
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe" ALL_BUILD.vcxproj  -p:Configuration=Release # depending on where MSBuild is
msbuild INSTALL.vcxproj -p:Configuration=Release # with administrative privilege

reference: https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/setup-windows.html

My bad, looks like sudo is still required. I checked early commits and saw successful builds. Could you re-trigger the ci again? If possible, we can stop the ci for commit fe63369.

The correct python version isn't picked still. Any other ideas?

@ejguan My bad, the test for incorrect filenames should take input not file_urls. The test should pass now.
And if the test passes now, the only thing remaining is about the CI.

For windows, you may add another step to install msbuild via https://github.com/microsoft/setup-msbuild

For windows, you may add another step to install msbuild via https://github.com/microsoft/setup-msbuild

Added the step. The last step for installation still requires administrative privilege. We may need a way for that.
Also, the ubuntu py38/39 are passing. We've got us a solid checkpoint.

@ydaiming Could you send me your email via slack then I will see how to let you run the CI without my manual approval.

@ejguan Could you trigger the CI now to see if the runtime Python is actually indicated as the same for setup-python? Thanks.

@ejguan Looks like 6 Domain tests failed in [MacOS, Windows] x [py37, 38, 39]. Samples can't be drawn for [eurosat]. Is this issue related to my S3 IO PR in any way?

@ejguan I rebased my branch based on the main branch, which causes a diverge (non-fast-forward). I will need to create a new branch and a new PR https://github.com/pytorch/data/pull/318 for the rebased code.

@ejguan I rebased my branch based on the main branch, which causes a diverge (non-fast-forward). I will need to create a new branch and a new PR #318 for the rebased code.

I did rebase for you and forced update your branch. LMK if anything wrong with the rebase with your change.

@ejguan
Thanks for rebasing the branch! A small suggestion is removing the setup.cfg which looks redundantly added.


iOS toolchain error when running flutter doctor

I have gone through the set up as per the instruction on flutter.io and i keep getting an error on the iOS toolchain section.
[!] iOS toolchain - develop for iOS devices (Xcode 10.0)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 10.0, Build version 10A255
✗ Verify that all connected devices have been paired with this computer in Xcode.
If all devices have been paired, libimobiledevice and ideviceinstaller may require updating.
To update, run:
brew uninstall --ignore-dependencies libimobiledevice
brew install --HEAD libimobiledevice
brew install ideviceinstaller
• ios-deploy 2.0.0
• CocoaPods version 1.5.3
Every time i run brew install --HEAD libimobiledevice i get another error and i am at a loss on how to fix it.
==> ./autogen.sh
Last 15 lines from /Users/dewy/Library/Logs/Homebrew/libimobiledevice/01.autogen.sh:
checking dynamic linker characteristics... darwin17.7.0 dyld
checking how to hardcode library paths into programs... immediate
checking for pkg-config... /usr/local/opt/pkg-config/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libusbmuxd >= 1.1.0... no
configure: error: Package requirements (libusbmuxd >= 1.1.0) were not met:
Requested 'libusbmuxd >= 1.1.0' but version of libusbmuxd is 1.0.10
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables libusbmuxd_CFLAGS
and libusbmuxd_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details
I have tried a couple of ideas and did some googling but couldn't find an answer

Having the same problem. Let me know if you resolve it.

Solved by this: https://stackoverflow.com/a/52604913/3480445

@DamionA does @danielwii's suggestion help?

That has worked a treat, thanks @danielwii

the message shows after the flutter upgraded to 1.2.1
[!] iOS toolchain - develop for iOS devices (Xcode 10.1)
    ✗ Verify that all connected devices have been paired with this computer in Xcode.
      If all devices have been paired, libimobiledevice and ideviceinstaller may require updating.
      To update with Brew, run:
        brew update
        brew uninstall --ignore-dependencies libimobiledevice
        brew uninstall --ignore-dependencies usbmuxd
        brew install --HEAD usbmuxd
        brew unlink usbmuxd
        brew link usbmuxd
        brew install --HEAD libimobiledevice
        brew install ideviceinstaller

the method provided by @danielwii doesnt work

the message shows after the flutter upgraded to 1.2.1
[!] iOS toolchain - develop for iOS devices (Xcode 10.1)
    ✗ Verify that all connected devices have been paired with this computer in Xcode.
      If all devices have been paired, libimobiledevice and ideviceinstaller may require updating.
      To update with Brew, run:
        brew update
        brew uninstall --ignore-dependencies libimobiledevice
        brew uninstall --ignore-dependencies usbmuxd
        brew install --HEAD usbmuxd
        brew unlink usbmuxd
        brew link usbmuxd
        brew install --HEAD libimobiledevice
        brew install ideviceinstaller

all the commands listed above has been performed, not work
the method provided by @danielwii doesnt work

same problem

Same here...

Same here...

[!] iOS toolchain - develop for iOS devices (Xcode 10.1)     ✗ Verify that all connected devices have been paired with this computer in Xcode.       If all devices have been paired, libimobiledevice and ideviceinstaller may require updating.       To update with Brew, run:         brew update         brew uninstall --ignore-dependencies libimobiledevice         brew uninstall --ignore-dependencies usbmuxd         brew install --HEAD usbmuxd         brew unlink usbmuxd         brew link usbmuxd         brew install --HEAD libimobiledevice         brew install ideviceinstaller
HELP!
it runs in my simulator tho!

Hey guys, it seems this issue can be fixed (at least ios toolchains issue) by plugging in a iphone 6 or 6s (older works maybe IDK) then try flutter doctor again...

同样的问题

我重启电脑就好了，过一会儿又报错，但是貌似不影响使用

try this:https://github.com/nesger/FlutterNote/issues/1#issuecomment-444806569


[Bug][Devlake Api] Can't GET projects which have "/" in their name

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
We have a use case of using a project naming convention like this: my/project/name, and we are not longer able to GET /project/my/project/name, or the escaped version of it, /project/my%2Fproject%2Fname
What do you expect to happen
We would expect to either of the formats to work.. we are coming from v0.18.0, where format "my/project/name" used to work, but now it no longers retrieves the project:

The interface indicates this should be supported, and I was also able to create a new project using the /:

How to reproduce

online devlake locally
create a project named "my/test/project" on the interface
attempt to open it

Anything else
No response
Version
v1.0.0-beta2
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I guess it is related to this change? https://github.com/apache/incubator-devlake/commit/e9c25792fea4226a5f7d9b9e611e6b0062b1c7e0#
But then how to be able to have / in the project name but also have a /check in the end of the other route? :thinking:

Weird, escaped project name should work.

Need triage

I managed to get the route working again with '/' in the project name, but I think the '/check' route will continue to fail if we keep using ':projectName' instead of '*projectName'.
My draft pull request: https://github.com/apache/incubator-devlake/pull/7281

Will be fixed in v1.0.0-beta5


[INVESTIGATE] Can we supply a custom root certificate on Windows?

Description
Instead of asking the users to ignore the default Windows root certificate, we can maybe supply a custom one instead (https://stackoverflow.com/questions/68896243/how-to-properly-configure-node-js-to-use-self-signed-root-certificates -- not with a self-signed one of course).
Would be a convenient solution for less tech-savvy users

If the root cause is let's encrypt, it might be simpler to move to our new host name for the plugin. It will be proxied by cloudflare thus using a different non-letsencrypt certificate. Before we rush things we need to confirm some things:

find someone that can reliably reproduce the issue
reproduce the issue with a key created on naos.ugent.be and using that as hostname
using that same key with naos.dodona.be (also proxied by cloudflare) should then work


Add config file for mod_static.

Codecov Report

Merging #95 into develop will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop      #95      +/-   ##
===========================================
+ Coverage    52.76%   52.79%   +0.02%
===========================================
  Files          200      200
  Lines        20791    20791
===========================================
+ Hits         10970    10976       +6
+ Misses        8607     8599       -8
- Partials      1214     1216       +2

Impacted Files
Coverage Δ

bfe_balance/bal_gslb/sub_cluster.go
48.14% <0%> (-7.41%)
:arrow_down:

...e_modules/mod_trust_clientip/mod_trust_clientip.go
42.85% <0%> (-3.3%)
:arrow_down:

bfe_http2/server.go
71.83% <0%> (+0.32%)
:arrow_up:

bfe_websocket/server_conn.go
76.51% <0%> (+0.75%)
:arrow_up:

bfe_spdy/server_conn.go
78.53% <0%> (+0.88%)
:arrow_up:

bfe_spdy/server_write_sched.go
80% <0%> (+1.53%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a000af6...e7e07f1. Read the comment docs.


SqlMagic.autolimit doesn't add a limit clause to the query

What happens?
We're using jupysql to query a trino cluster through jupyter notebook. I want to autolimit the query results that we query to the cluster, as it's intended to be used for an interactive analytics workload.
I expect the SqlMagic.autolimit to add a LIMIT clause to the query, but it doesn't, i can see the original query in the Trino Admin UI.
To Reproduce

Create a trino sqlalchemy connection. trino_connection = create_engine("trino://trino_cluster....")
set an autolimit. %config SqlMagic.autolimit=500
Use the connection: %sql trino_connection
Run a query: %sql SELECT * FROM BIG_TABLE
Get the result sets or See the query in the trino UI. You'll get the  SELECT * FROM BIG_TABLE without the LIMIT Clause

OS:
Linux
JupySQL Version:
0.10.10
Full Name:
Takieddine KADIRI
Affiliation:
Société Générale Assurances

yeah, the effect of autolimit is not to add a LIMIT to the statement but to send the query and only fetch the top X results, are you seeing performance issues?
you can see the relevant code here.
the main reason not to add LIMIT, is that some DBs don't support it so we'd have to figure out a way to make it portable. I guess it wouldn't be too hard using sqlglot, but given that we support many databases, adding stuff like this ends up breaking features for some users.
perhaps a more reasonable solution would be to add LIMIT only to the dbs that we're sure they support it, but even detecting which db the user is connected to is challenging and we've encountered edge cases.
bottom line is that we've decided not to modify user's code to prevent these issues. if you have time, feel free to open a PR, perhaps there is another approach that I'm overlooking

Ah i see, The doc mislead me. Thank you for the quick feedback.
For the interactive analytics workload, we're also using Apache Superset, that manage to systematically LIMIT the query. That's why i was looking for this in jupysql
Maybe i could use jupysql for a slighly different use case, where the desired behavior is to collect the entire query results for further analysis with pandas or for doing joins with another data sources. I'll tests the user bahaviors and the cluster performance with jupysql and autopandas=True and let you know if the LIMIT clause is really needed here.

sounds good, thanks for your feedback!


refactoring functions resetToProduction & importSchema

The importSchema() and resetToProduction() functions are written in a more readable way. Specifically, I wrote a single line of code inside the try-catch block and used error instead of console.error() 🥷

Thanks for your contribution. It looks like you're changing back-ticks to double quotes, and renaming a variable. For consistency with the rest of the project, I'm going to close this PR and leave this function as-is.


Add endpoint that returns HTTP 200 when the cache is ready

On large disks >4TB the LRU index takes a few minutes to load and also the endpoint /status suffers from lock contention on high-loads (sometimes /status takes 10 seconds to respond).
This new endpoint just returns OK when the cache is up / ready.

@mostynb Good, is this a constant endpoint ? can you show me a curl example ?

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is the null SHA256 hash, so if your bazel-remote instance is listening on port 8080 then you can do something like this:
curl --head --fail http://localhost:8080/cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
You can also redirect stdout and stderr to /dev/null if you only want to use curl's exit code and don't want the console output.
This is handled here: https://github.com/buchgr/bazel-remote/blob/337c0dbe1dbdc3f95551dfd462c62acebb4b028f/server/http.go#L306
You can see that it does not require a cache lock, and will return OK even if the cache is empty.

Thank you! I think this should be enough then :+1:

Added a note about this to README.md: https://github.com/buchgr/bazel-remote/commit/a7d442dace50102973bcd91e5e15b45b9d0e1fac


update mypy to 1.13

[x] Closes #9624

Noticed, that our NestedSequence definition was wrong and outdated.
Also, the normalize paths method never worked properly for >1D sequences. Fixed this as well.

The pydap / bs4 error is unrelated I think.

I don't seem to be able to force merge this, can someone do this plz?


Question: Add a custom header to OAuth authentication

Hi,
I need to set a custom header parameter for my OAuth Authentication. Something like "X-User-Id".
Haven't found anything for additional parameters for OAuth, or maybe I missed something?
Sorry if that's a noob question.
I'm using Keycloak on the server side.
My current configuration looks like this:

@Bean
public SecurityContext securityContext() {
return SecurityContext.builder().securityReferences(defaultAuth()).build();
}
private List defaultAuth() {
return Arrays.asList(new SecurityReference(SECURITY_SCHEME_OAUTH2, defaultScope().toArray(new AuthorizationScope[] {})));
}
private Predicate postPaths() {
return regex("/.*");
}
private Predicate springBootActuatorJmxPaths() {
return regex("^/(?!env|restart|pause|resume|refresh).*$");
}
private List defaultScope() {
AuthorizationScope authorizationScope = new AuthorizationScope("openid", "Basic Open ID Connect Scope");
List authorizationScopes = new ArrayList<>();
authorizationScopes.add(authorizationScope);
return authorizationScopes;
}
@Bean
public Docket postsApi(List securityContexts) {
return new Docket(DocumentationType.SWAGGER_2).groupName("public-api")
.apiInfo(apiInfo()).select().paths(postPaths())
.apis(RequestHandlerSelectors.basePackage("com.example"))
.paths(springBootActuatorJmxPaths())
.build()
.securitySchemes(Collections.singletonList(oauth()))
.securityContexts(securityContexts)
;
}
@Bean
List grantTypes() {
List grantTypes = new ArrayList<>();
grantTypes.add(
new ImplicitGrant(
new LoginEndpoint(oAuthServerUri + "/realms/" + REALM_NAME + "/protocol/openid-connect/auth"),
"access_token"
)
);
return grantTypes;
}
@Bean
SecurityScheme oauth() {
return new OAuthBuilder()
.name(SECURITY_SCHEME_OAUTH2)
.scopes(defaultScope())
.grantTypes(grantTypes())
.build();
}
@Bean
public SecurityConfiguration securityInfo() {
return SecurityConfigurationBuilder.builder()
.clientId(clientId)
.realm(REALM_NAME)
.appName(serviceName)
.scopeSeparator(" ")
.build();
}

@hakamairi You can use the globalOperationParameters method in the docket. Look at # 22 in the docs

@dilipkrish thanks for your answer.
I already tried that, and as for 2.8.0 it adds parameters per REST API call, not the Authentication part.

I think you'd have to extend sawgger-ui for that.

Thanks for your answer @dilipkrish
I've asked on swagger-ui for that https://github.com/swagger-api/swagger-ui/issues/4268#issuecomment-369432531
Short answer is that they pointed me to requestInterceptor, does springfox provide any support for this feature?
I was only able to find some mentions of this here https://github.com/springfox/springfox/issues/2166
No documentation though.

@hakamairi, I believe if you want to customize the options are to bring your own customized swagger-ui. The bundled swagger-ui only gets you so far

Hi , any idea how to add XSRF-TOKEN to keycloak in Docket. i see the request includes XSRF token but my post request is failing either null or invalid csrf token.

Havent done it myself, so I cant say without researching. If you did find out please report back @selvaebi


Should do a release

4.8.0 is busted on everything because of that quotes issue. Should ship a 4.8.1 ASAP to unbreak there :)

It seems that e257f450c044616fefbd15e9a180387aaaa572a1 accidentally fixed a bug in bin/chef-zero. It's a syntax error so it impacts all platforms.

Here is the specific line: https://github.com/chef/chef-zero/commit/e257f450c044616fefbd15e9a180387aaaa572a1#diff-a2a4711aebb6fc27c21c8481c60db638R103

nothing accidental about it, but yes, we should do a new release.

@thommay I only mention accidental because the commit for e257f45 doesn't mention that a bug fix is included. Makes it seem like it's only style.

yeah, the fix itself was in https://github.com/chef/chef-zero/commit/1e9200bd055a0c15836adf0c6c75a8c20ad77887

Ah, seen. I misread the diff when I looked at it originally. Cheers!

I did a release.


Don't (re)set value in IndividualUser setters

@joaofnds missed this when doing PR review but noticing now when troubleshooting the import script for the new user/org models.
We should have it like OrganizationUser just return the value don't set e.g. this.firstName = firstName within the setter. Otherwise the setter is recalled (again & again...). Mongoose somehow prevents a loop from running endlessly but it's still called many times so to be avoided.

Superceded by #574


Static image do not load if the browser is not load from the top

If you refresh your webpage in the middle of the page, the display: static element won't start loading.

You could try this jQuery(window).scroll();
I fixed jquery.lazyload.js same problem with this.


Suggestion: Give user ability to set UUIDs

Currently importing things that depend on one another (i.e. cross-references) is hard, because we need to:

Import ThingA
Catch the ID of ThingA
Modify the payload of ThingB to include a beacon with a reference to Thing A
Import ThingB

If we allowed the user to set the UUID on Thing A, they could just assign UUIDs as they please and then import ThingA and ThingB without having to either catch an ID or modify a payload.
@bobvanluijt Thoughts?

Yes! This would be fantastic. I would allow all UUID versions (if possible) except for Nil UUID types.

I would like to move this up if possible, it comes in handy for some use cases.

Sure, what do you offer in return? 😛
Kidding. In all seriousness, sure. I don't think I'll get to it today, but I can still prfioritize this highly.


Tokens

Context
Hecate has long had the ability to generate and track tokens in a generic way, however it has been tied primarily to session tokens. This goal of this PR is to add more generic handling of both session and user created tokens
TODO

[ ] Add API to list tokens
[ ] Add tests for listing tokens
[ ] Add user UI to list & manage tokens

cc/ @ingalls

Redo user page with tabs

Token Listing Page

Move account security to it's own tab in anticipation of OTP support


[BUGFIX] Many Fixes/Additions

Proposed Semver Increment Bump: [MAJOR/MINOR/PATCH]
None (These should have been pushed last night with the configuration system, but I forgot, except for the clean function fix, but I figured I'd add it here anyway)
Changes Proposed in this Pull Request (List new items in CHANGELOG.MD)

Configuration command uses new configuration system
Events Added for Configuration System (Delete && Create)
Fixes clean function not removing client.user.email

(If Applicable) What Issue does it fix?
Fixes... configuration command

To be honest, that clean function was ripped from D.JS GRAF.
Other then that seems good.


CDATA Decoding not working

struct Container: Codable {
    let value: Int
    let data: String
}

let xmlString =
"""
<container>
   <value>42</value>
   <data><![CDATA[lorem ipsum]]></data>
</container>
"""
let xmlData = xmlString.data(using: .utf8)!
let result = try decoder.decode(Container.self, from: xmlData)

result.data decoded into "".  Actually there is an error in XMLDecoder.swift:212 - Expected String but found null instead.
If we go deeper -  KeyedBox.swift:27 return elements[""].first as? SimpleBox. It will return nil because key is "#CDATA" in this case.
I've made simple code change that fixed problem in this case, but have no idea what it can break later.
.patch
diff --git a/Sources/XMLCoder/Auxiliaries/Box/KeyedBox.swift b/Sources/XMLCoder/Auxiliaries/Box/KeyedBox.swift
index 9930bfb..a3e8742 100644
--- a/Sources/XMLCoder/Auxiliaries/Box/KeyedBox.swift
+++ b/Sources/XMLCoder/Auxiliaries/Box/KeyedBox.swift
@@ -24,7 +24,7 @@ struct KeyedBox {
     }

     var value: SimpleBox? {
-        return elements[""].first as? SimpleBox
+        return elements.values.first as? SimpleBox
     }
 }

diff --git a/Sources/XMLCoder/Auxiliaries/KeyedStorage.swift b/Sources/XMLCoder/Auxiliaries/KeyedStorage.swift
index 0615ca2..a3fcb50 100644
--- a/Sources/XMLCoder/Auxiliaries/KeyedStorage.swift
+++ b/Sources/XMLCoder/Auxiliaries/KeyedStorage.swift
@@ -23,6 +23,10 @@ struct KeyedStorage<Key: Hashable & Comparable, Value> {
     var keys: [Key] {
         return buffer.map { $0.0 }
     }
+
+    var values: [Value] {
+        return buffer.map { $0.1 }
+    }

     init<S>(_ sequence: S) where S: Sequence, S.Element == (Key, Value) {
         buffer = Buffer()

Thanks for reporting it @Kirow, the fix for it will be included in the next release.

Hi @MaxDesiatov,
For when is the next release scheduled? Waiting in anticipation for this.
Thanks for a great XML coder!

Hey @kasperkronborg, this is now published in the 0.11 release and is available with SwiftPM, CocoPods and Carthage. Thanks for your feedback, I appreciate your support 🙂


chore: Merge in a fixed version of your pull request

As per @pablojim's feedback this also brings the repo up to date with origin.


Bug/Question: Footer in authorised pages

With the update of the welcome page now we have a new footer. Is it necessary to have the big footer throughout all pages including the crew ones? Can we create a smaller one for those authorised pages or do we rely on the fact that we'll fix the scrolling and won't be covering so much of the screen?

I am currently working on the dashboards. The idea is that the footer is not present in the dashboard itself, so I guess it won't be an issue.

Closing this as it will not be in the authorised pages.


🔥✨ No more availableThemes

I've been waiting to push up this PR for quite a while 😹
refs #7491

This PR completely removes the concept of "availableThemes" from both config & settings.
It replaces it instead with a concept of themes.list. Which has a nice API similar to our shiny new settings cache.
This opens the door for significant further simplification of code in the settings API/models etc.

TODO:
The last commit has a list of areas I'm still not happy with - the API for themes lib is confusing, we want to be able to hide away much more internal logic and make it easier for the themes JSON API to interact with this library.
I also need to improve test coverage around the themes lib, I have deliberately left this as I want the structure of the objects to change.
NOTE: master still has zero support for custom templates and custom error templates 😁  Will fix that after this goes in.

Done + travis has finally finished 😁


Documentation for dependency verification

Context
Adds documentation for dependency verification.

Temporarily changed base branch to reflect this PR is only about docs.

Thanks a lot for your comments @vlsi , makes a lot of sense!

@melix , does verification play together well with https://github.com/ben-manes/gradle-versions-plugin ?
In other words, the sole purpose of gradle-versions-plugin is to discover if a new version of a library or a plugin is available.
AFAIK if the verification is activated it would block gradle-versions-plugin which defeats gradle-versions-plugin :(
checksum-dependency-plugin has a property for that:

checksumIgnoreOnTask (comma separated string, default: dependencyUpdates)

I guess it would make sense to handle the use case somehow in Gradle's implementation as well.

Thanks for the comments. There are a few expectations we need to clarify:

this PR is about documenting the current implementation
the implementation may (and surely will) change in the future, we can adapt
it just has to be good enough. Look where we start: nothing. If we can provide checksum and signature verification and make people aware of its limits, the goal is reached. We don't have to pretend to be more secure than it is, but the limitations have to be understandable by any user.
we don't pretend to be more secure than whatever the industry standards are. We use BouncyCastle, with 64-bit (long( key ids internally. Even if there can be collisions, the chances that you get a collision on a this and that the signature is malicious and that the author managed to fake a checksum too is extremely unlikely. To me you reach maximum security not by putting yourself behind a single stronghold, but by combining good security practices and combining them together (checksum+signature for example).
I'm not a security expert, so I'm open to improvements in the implementation, this has to be done either through PRs or discussions with the team.
Every team will set their security standards and expectations. Not all of them will need the same level of security. If you want to go full paranoid mode, you can.

does verification play together well with ben-manes/gradle-versions-plugin

@vlsi No, it won't. At least not with the initial implementation. Remember our time is limited, we have many topics to work on, so all time we spend has to be reasonably spent. At this stage I consider this feature "good enough" for a first release. I don't think a system property to skip tasks is a good idea. Task level is too weak, you never know when a configuration will be resolved. If we want to do something (and I planned to do it but we decided to defer until we have a good, real use case), I'd be tempted to do something on the ResolutionStrategy itself, which is basically the granularity of a single dependency graph.
That said, not everything is lost: you can run with --dependency-verification lenient (or the equivalent system property) and the task would pass. I'm also in general reluctant with tools like the Gradle versions plugin which are I think nice for being aware that a new version is available, but automated upgrades are in general a bad idea. Anyway that's off topic for this conversation.

There are a few expectations we need to clarify

Just in case, all of them meet my expectations.

At least not with the initial implementation

I wanted to know if it is "implemented but undocumented" or "may be implemented later" or "will never be accepted because..."

I wanted to know if it is "implemented but undocumented" or "may be implemented later" or "will never be accepted because..."

Everything is open. In general we tend to implement the minimal set of things to have people use it, give feedback, then we improve on actual data. For example for dependency locking we will soon implement usability improvements based on user feedback. We try to avoid "guessing" what people might need. Sometimes it's obvious (I made a lot of improvements to this feature JUST by dogfooding on the Gradle build).
Also we have time (6.2 is still weeks away) so we're happy to accept PRs.

I merged this PR in order to have a first rendered version of the docs in a nightly that folks can try, but feel free to give more feedback here or create new issues at your convenience :+1:

I'm also in general reluctant with tools like the Gradle versions plugin which are I think nice for being aware that a new version is available, but automated upgrades are in general a bad idea

Note that GVP only handles awareness and points to other projects that built automation on top of it. It would be wonderful if the Gradle team could build in the plugin's features natively as a better steward for popular functionality.
Thanks @melix and @vlsi for this verification functionality!

Note that GVP only handles awareness and points to other projects that built automation on top of it.

Good so as long as it doesn't download POM files, it should be compatible. Note that I implemented #11761 today which should make it possible for the versions plugin to workaround if needed (once merged).

It would be wonderful if the Gradle team could build in the plugin's features natively as a better steward for popular functionality.

Yup, this is something we consider but given there's a good plugin for that ;) More seriously a first step will be to provide a public API for listing versions.

Good so as long as it doesn't download POM files, it should be compatible

Indirectly through the Gradle APIs. All of the heavy lifting is done by Gradle.

Configuration.resolvedConfiguration.lenientConfiguration.firstLevelModuleDependencies for resolving dependency versions, using dynamic versions for the latest.
ArtifactResolutionQuery to extract the projectUrl from the POM, if possible.


Handle image views

There is currently no mechanism for handling image views. The current plan is to have an image_views table created. Whenever a user requests an image, the following process is executed:

The users session is checked. If the image they are requesting is in their viewed.images session variable, nothing further is done, we don't log duplicate views from the same client. At least not in a short time span.
A queue job is created. This queue job checks if a view entry for this image by this user already exists in the image_views table. Create a new entry if one doesn't already exist. (Ideally if we're using MySQL consider an INSERT IGNORE query instead of two individual queries)
An entry for this image is added to the users viewed.images session variable.

Every 5 minutes or so, a task will be run to process entries on this table and update the live view counts for images. The table will be cleared every time this task is run.
Why all the complexity? Why not just increase the view count directly?
Two main reasons.

The goal is to try and build the application with concurrency and scalability in mind. If an uploaded image suddenly gets hit with a large burst of traffic, table/row locking can create a bottleneck.
Utilizing a table like this prevents users from being able to easily manipulate (artificially inflate) the view count of an image, as only one image view entry can be registered from a single user/IP address before the view task is run.

An alternative (and probably better) method of handling image_views table entries would be to skip existing checks altogether on INSERT. A GROUP BY query can be executed when image views are processed instead.


support closeOnCompletion

missed unwrapping test for previous PR

Calling closeOnCompletion etc. breaks jdk6 compatibility. We may need to use reflection for calling these methods.

Fixed the test code so that it can work in JDK6.
https://github.com/xerial/sqlite-jdbc/commit/3123878737d095cbf0478f35a9caea53268913b0
Thanks for the contribution!

right, missed that - thanks!


More examples and mentions in the docs about the glorious 'returning()' query building method

I know you're on vacation, and I hope you're enjoying it! Still loving Objection as I get deeper and deeper, so thanks again!
I noticed in the docs that you nicely mention how, if you're using Postgres, you can chain a returning(...) to an insert to "get the same result without an additional query" as insertAndFetch. I think this is a great feature that's exposed, and a great suggestion in the docs.
After wondering if it would also work for patch queries, and upon seeing these Postgres docs, I tinkered around with Objection and found that it is possible to leverage this pattern on patch, avoiding an additional query associated with patchAndFetch. This was a happy discovery, but this solution/trick is not mentioned anywhere else outside of insert contexts, and I think users would really benefit by seeing it in the docs, or perhaps even in the cookbook with a bit more slickness.
// The only way the docs suggest: this performs 2 queries, right? 1 update and 1 select?
person
  .$query()
  .patchAndFetch({first_name: 'Jennifer'})
  .then(function(updatedPerson) {
    ...
  })
  .catch(...);

// Do the same thing in a single query with Postgres. Yay!
person
  .$query()
  .patch({first_name: 'Jennifer'})
  .returning('*') // Get the data back in a single query, yay!
  .then(function(updatedPeople) {
    let updatedPerson = updatedPeople[0];
    ...
  })
  .catch(...);

// Do the same thing in a single query with Postgres, but turn it into a single row returned.

I did not try this out for update queries because I don't really use them, but I'd imagine it would work the same there as well?
Maybe even make a change to Objection that automatically "make single" the responses from patch (and perhaps update) queries that are built from instance queries starting with $query?
person
  .$query()
  .patch({first_name: 'Jennifer'})
  //.first() // No longer needed because Objection will .first() it for you since it's $query
  .returning('*') // Get the data back in a single query, yay!
  .then(function(updatedPerson) {
    // Objection has already applied .first() to the query, so single instance of person is returned
    // to promise resolution
    ...
  })
  .catch(...);

You guys are awesome!

You are right, returning should work for all insert, update and patch queries with those databases that supports it (postgres, oracledb and mssql). I agree that mentioning about it would be good addition to documentation.
About $query().patch() and $query.update() returning single element, I'm not sure if it would be confusing or good. I don't have strong opinion about it maybe slightly agreeing since I cannot think of any downsides in that solution. If one likes to do update / patch for multiple elements, they should do it through .query() anyways.

Hi @elhigu  - Though I personally think it would be nice and seems intuitive to make the suggested $query.patch().returning() → singleInstance (especially if it's properly documented), it's of course up to you guys to decide what you think is best. I think even leaving it as is, but improving the docs would lead most people to find this way to get it done.
I'm glad you agree that the additional examples of the existing capabilities would be helpful. I'm happy to make a PR for the doc changes if I can figure it out and if it would be helpful for you?

@newhouse maybe better separate these 2 issues now.

I completely agree that documenting that .returning() works for all update, patch and insert queries would be good thing and will merge documentation PR for that without hesitation.

For changing $query.patch and $query.update I would like to get input from more people, so could you open another issue for that and let's dedicate this one for the documentation update of returning.

👍

PR submitted: https://github.com/Vincit/objection.js/pull/249


chore: drop console.log from production builds

Details
This PR enables drop_console option in swc minifier.
I tried to add support for custom .swcrc but I couldn't get it work in any way. It would be amazing if people could configure that. But I also think it's ok to not have console log in production. What do you think?
Options docs: https://swc.rs/docs/usage/core#options
Compress docs: https://swc.rs/docs/configuration/minification#jscminifycompress
Related: https://github.com/PlasmoHQ/plasmo/issues/542
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I agree to license this contribution under the MIT LICENSE
[X] I checked the current PR for duplication.

Contacts

(OPTIONAL) Discord ID:

If your PR is accepted, we will award you with the Contributor role on Discord server.
To join the server, visit: https://www.plasmo.com/s/d

@mathieudutour that's a good point, that's why I added an open question to my PR. Ideally i would like to support .swcrc file to let people customize but after hours I just couldn't get it to work. I think it is because it is not supported when contents are passed as a string. Do you agree @louisgv?
It could also be a CLI option but it feels wrong. It feels that something like this should live in a dedicated config file.

Hmm @flexchar yeah I think most sane solution would be to suggest end-user to leverage env vars. Parcel prunes dead branches. Thus:
if (process.env.NODE_ENV === 'development') {
     console.log("etc")
}

Will be pruned if it's production

Oh, I didn't know this existed. I did have simple function log() that had this in the body but Parcel wasn't smart enough to remove those. This is because SWC processes each file individually so it is not aware of how imported functions behave. This is something I am generally really not fond of the way it work. It also prevents some other features. Hence the inspiration for this.
I am definitely less experienced in the context of custom parcel + swc. Do you think there is a relatively easy way to provide config file?


Retry only known-flaky tests

Is your feature request related to a problem? Please describe.
Test retries were added as a tactical fix to deal with non-deterministic test failures. However, this has allowed us to introduce new tests that are flaky. The test retries should be a tactical fix for existing tests that are flaky, but we should never rely on retries for newly added tests, or at least certainly not by default.
Describe the solution you'd like
Identify flaky tests from the last month or so using the flaky test finder script or something similar, then explicitly retry only those tests in the test retry configuration. Specifically, this could look like:
// test retry configuration
subprojects {
  apply plugin: "org.gradle.test-retry"
  tasks.withType(Test).configureEach {
    retry {
      if (BuildParams.isCi()) {
        maxRetries = 3
        maxFailures = 10
      }
      failOnPassedAfterRetry = false
      classRetry {
        includeClasses.add("org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests")
        includeClasses.add("org.opensearch.action.admin.indices.create.ShrinkIndexIT")
        includeClasses.add("org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT")
        includeClasses.add("org.opensearch.cluster.ClusterHealthIT")
        includeClasses.add("org.opensearch.cluster.allocation.AwarenessAllocationIT")
        includeClasses.add("org.opensearch.cluster.allocation.ClusterRerouteIT")
        includeClasses.add("org.opensearch.cluster.metadata.IndexGraveyardTests")
        includeClasses.add("org.opensearch.cluster.routing.MovePrimaryFirstTests")
        includeClasses.add("org.opensearch.cluster.service.MasterServiceTests")
        includeClasses.add("org.opensearch.gateway.RecoveryFromGatewayIT")
        includeClasses.add("org.opensearch.http.SearchRestCancellationIT")
        includeClasses.add("org.opensearch.index.IndexServiceTests")
        includeClasses.add("org.opensearch.index.IndexSettingsTests")
        includeClasses.add("org.opensearch.index.SegmentReplicationPressureIT")
        includeClasses.add("org.opensearch.index.reindex.BulkByScrollResponseTests")
        includeClasses.add("org.opensearch.index.shard.RemoteStoreRefreshListenerTests")
        includeClasses.add("org.opensearch.index.translog.RemoteFSTranslogTests")
        includeClasses.add("org.opensearch.indices.replication.RemoteStoreReplicationSourceTests")
        includeClasses.add("org.opensearch.indices.replication.SegmentReplicationIT")
        includeClasses.add("org.opensearch.indices.replication.SegmentReplicationRelocationIT")
        includeClasses.add("org.opensearch.indices.replication.SegmentReplicationTargetServiceTests")
        includeClasses.add("org.opensearch.monitor.fs.FsHealthServiceTests")
        includeClasses.add("org.opensearch.remotestore.CreateRemoteIndexIT")
        includeClasses.add("org.opensearch.remotestore.CreateRemoteIndexTranslogDisabledIT")
        includeClasses.add("org.opensearch.remotestore.RemoteStoreIT")
        includeClasses.add("org.opensearch.remotestore.RemoteStoreStatsIT")
        includeClasses.add("org.opensearch.remotestore.SegmentReplicationRemoteStoreIT")
        includeClasses.add("org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT")
        includeClasses.add("org.opensearch.remotestore.multipart.RemoteStoreMultipartIT")
        includeClasses.add("org.opensearch.repositories.azure.AzureBlobContainerRetriesTests")
        includeClasses.add("org.opensearch.search.ConcurrentSegmentSearchTimeoutIT")
        includeClasses.add("org.opensearch.search.SearchTimeoutIT")
        includeClasses.add("org.opensearch.search.SearchWeightedRoutingIT")
        includeClasses.add("org.opensearch.search.pit.DeletePitMultiNodeIT")
        includeClasses.add("org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT")
        includeClasses.add("org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT")
        includeClasses.add("org.opensearch.snapshots.SnapshotStatusApisIT
      }
    }
  }
}

(The above list of tests was generated by taking all unique class names reported by the flaky test finder script using these parameters: ruby flaky-test-finder.rb -s 20000 -e 20597)

I like it.

@kotwanikunal Agreed. I'm fine with reverting this change and finding another way to fix the problem of introducing new flaky tests.

@kotwanikunal totally share the concerns but we have been largely in the dark on how bad the situation is, the work @andrross did just shows the scale of the problem. I am :100: agree with @mch2 here, we need to:

ramp up the efforts to fix existing known flaky tests
intercept and address new flaky as early as possible

I guess let's hunt then 😎


group page with displaying users and documents

need to work on displaying both lists side by side
and
add "add new user" and "add new document" button/functions

thank you for your comments Fred

No problem. I forgot to say one last thing. Omri said we should be able to use a listener. So if you look at the stuff I have implemented for the docker client, I'm listening for updates/changes. I am pretty sure the code we currently have in UsersList and DocsList will only check Firestore once when the component is mounted.
https://github.com/docs-code/node-docker-client/blob/master/index.js
If I remember correctly, Omri said integrating Firestore listeners with React should be similar to React with Redux (not react-redux) in how we would subscribe in componentDidMount and then unsubscribe in componentWillUnmount.
I posted this link in Slack but I'll post it here again.
https://firebase.google.com/docs/firestore/query-data/listen


Uncaught Error: ENOENT: no such file or directory, lstat '~/AppData\Local\atom\app-1.32.2\easyMes...

[Enter steps to reproduce:]

...
...

Atom: 1.32.2 x64
Electron: 2.0.9
OS: Unknown Windows version
Thrown From: Atom Core
Stack Trace
Uncaught Error: ENOENT: no such file or directory, lstat 'C:\Users\PC\AppData\Local\atom\app-1.32.2\easyMesh-master'
At fs.js:1661

Error: ENOENT: no such file or directory, lstat 'C:\Users\PC\AppData\Local\atom\app-1.32.2\easyMesh-master'
    at Proxy.realpathSync (fs.js:1661:15)
    at Proxy.fs.realpathSync (ELECTRON_ASAR.js:336:29)
    at TreeView.t.exports.TreeView.moveEntry (~/AppData/Local/atom/app-1.32.2/resources/app/static/<embedded>:14:916506)
    at TreeView.t.exports.TreeView.onDrop (~/AppData/Local/atom/app-1.32.2/resources/app/static/<embedded>:14:922390)
    at HTMLDivElement.<anonymous> (~/AppData/Local/atom/app-1.32.2/resources/app/static/<embedded>:14:896171)

Commands
     -4:13.2.0 core:copy (span.syntax--string.syntax--quoted.syntax--other)

Non-Core Packages
autocomplete-clang 0.11.5
build 0.70.0
busy-signal 1.4.3
file-icons 2.1.26
intentions 1.1.5
language-ini 1.20.0
linter 2.2.0
linter-gcc 0.7.1
linter-ui-default 1.7.1
platformio-ide 2.2.6
platformio-ide-debugger 1.2.6
platformio-ide-terminal 2.8.4
tool-bar 1.1.11

Thanks for contributing!
We noticed that this looks like a duplicate of https://github.com/atom/tree-view/issues/1282 so you can subscribe there for updates if you'd like.

Because we treat our issues list as the Atom team's backlog, we close duplicates to focus our work and not have to touch the same chunk of code for the same reason multiple times. This is also why we may mark something as duplicate that isn't an exact duplicate but is closely related.
For information on how to use GitHub's search feature to find out if something is a duplicate before filing, see the How Can I Contribute? section of the Atom CONTRIBUTING guide.


Control Layer not displayed

Hi,
I don't understand why my control layer no longer appears.
Here is a part of my controller:
angular.extend($scope, {
        ...
        layers: {
            baselayers: {
                mapbox: {
                    name: 'Mapbox',
                    type: 'xyz',
                    url: "http://{s}.tiles.mapbox.com/v4/.../{z}/{x}/{y}.png?access_token=...",
                    maxZoom: 18,
                    attribution: 'Cartographie Mapbox, données OpenStreetMap'
                }
            },
            overlays: {
                test: {
                    name: 'test',
                    visible: true,
                    type: 'group'
                }
            }
        },
        ...
});

And a part of my html:
<leaflet layers="layers" ... ></leaflet>

Is there a new way to do it with recent updates? Thanks.

Ok, that come from the property "group" of my markers.

So we can close?

Yep


az login fails on WSL2 ubuntu

This is autogenerated. Please review and update as needed.
Describe the bug
unable to login on MSL2 ubuntu
Command Name
az login & az login --use-device-code
Errors:
The command failed with an unexpected error. Here is the traceback:
0. The ID token is not yet valid. Current epoch =<PHONE_NUMBER>.  The id_token was: {
  "aud": "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
  "iss": "https://login.microsoftonline.com/6c4b3346-5ad4-4991-b51e-bb35a33065b0/v2.0",
  "iat":<PHONE_NUMBER>,
  "nbf":<PHONE_NUMBER>,
  "exp":<PHONE_NUMBER>,
  "aio": "AWQAm/8TAAAAyU7dp5Pzxp0CNUuUsRFSWpiesCuefFuB1gZ8KHi69e+rRQnVh1OWytE1q2CW+A/XSkI8OxSaoJDSroUD8DAtVpKoMJqfOGfp0bhSGz1CyJD6xAn4joLciwYi77elCsfU",
  "name": "Michael Fortin",
  "nonce": "2f6b4a167c4461c48577ae0cf9ad5c01f6bdae056dba9bbe785c6c8ee5680c35",
  "oid": "7a69d856-298c-43ef-be61-441263fe09a0",
  "preferred_username"<EMAIL_ADDRESS>  "puid": "10032000526BDFC5",
  "rh": "0.ARwARjNLbNRakUm1Hrs1ozBlsJV3sATbjRpGu-4C-eG_e0YcADQ.",
  "sub": "Q74lNKVGIkeNkIdS-z37SpIAuBSADz_XwqjYpO6-YDE",
  "tid": "6c4b3346-5ad4-4991-b51e-bb35a33065b0",
  "uti": "twsFIkuWEEOwb94LdOknAA",
  "ver": "2.0"
}
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/profile/custom.py", line 158, in login
    use_cert_sn_issuer=use_cert_sn_issuer)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 155, in login
    user_identity = identity.login_with_auth_code(scopes=scopes, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 173, in login_with_auth_code
    success_template=success_template, error_template=error_template, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/application.py", line 1563, in acquire_token_interactive
    **kwargs))
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oidc.py", line 282, in obtain_token_by_browser
    **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oauth2.py", line 641, in obtain_token_by_browser
    receiver, redirect_uri=uri, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oauth2.py", line 679, in _obtain_token_by_browser
    flow, auth_response, scope=scope, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/application.py", line 136, in obtain_token_by_auth_code_flow
    auth_code_flow, auth_response, headers=headers, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oidc.py", line 205, in obtain_token_by_auth_code_flow
    auth_code_flow, auth_response, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oauth2.py", line 561, in obtain_token_by_auth_code_flow
    **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oauth2.py", line 732, in _obtain_token_by_authorization_code
    return self._obtain_token("authorization_code", data=data, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oidc.py", line 115, in _obtain_token
    ret = super(Client, self)._obtain_token(grant_type, *args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oauth2.py", line 795, in _obtain_token
    "response": _resp, "params": params, "data": _data,
  File "/opt/az/lib/python3.6/site-packages/msal/application.py", line 582, in <lambda>
    event, environment=authority.instance)),
  File "/opt/az/lib/python3.6/site-packages/msal/token_cache.py", line 307, in add
    super(SerializableTokenCache, self).add(event, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msal/token_cache.py", line 113, in add
    return self.__add(event, now=now)
  File "/opt/az/lib/python3.6/site-packages/msal/token_cache.py", line 153, in __add
    if id_token else {})
  File "/opt/az/lib/python3.6/site-packages/msal/oauth2cli/oidc.py", line 77, in decode_id_token
    err, _now, json.dumps(decoded, indent=2)))
RuntimeError: 0. The ID token is not yet valid. Current epoch =<PHONE_NUMBER>.  The id_token was: {
  "aud": "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
  "iss": "https://login.microsoftonline.com/6c4b3346-5ad4-4991-b51e-bb35a33065b0/v2.0",
  "iat":<PHONE_NUMBER>,
  "nbf":<PHONE_NUMBER>,
  "exp":<PHONE_NUMBER>,
  "aio": "AWQAm/8TAAAAyU7dp5Pzxp0CNUuUsRFSWpiesCuefFuB1gZ8KHi69e+rRQnVh1OWytE1q2CW+A/XSkI8OxSaoJDSroUD8DAtVpKoMJqfOGfp0bhSGz1CyJD6xAn4joLciwYi77elCsfU",
  "name": "Michael Fortin",
  "nonce": "2f6b4a167c4461c48577ae0cf9ad5c01f6bdae056dba9bbe785c6c8ee5680c35",
  "oid": "7a69d856-298c-43ef-be61-441263fe09a0",
  "preferred_username"<EMAIL_ADDRESS>  "puid": "10032000526BDFC5",
  "rh": "0.ARwARjNLbNRakUm1Hrs1ozBlsJV3sATbjRpGu-4C-eG_e0YcADQ.",
  "sub": "Q74lNKVGIkeNkIdS-z37SpIAuBSADz_XwqjYpO6-YDE",
  "tid": "6c4b3346-5ad4-4991-b51e-bb35a33065b0",
  "uti": "twsFIkuWEEOwb94LdOknAA",
  "ver": "2.0"
}

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
start wsl shell interminal
az login

Expected Behavior
to verify identity and login
Environment Summary
Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.33.0

Additional Context

@jiasli for awareness

Duplicate of #20388


docker-compose error on 1.26.2

Describe the bug
$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
Invalid top-level property "x-app". Valid top-level sections for this Compose file are: version, services, networks, volumes, and extensions starting with "x-".

You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the
 `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
services.worker.depends_on contains an invalid type, it should be an array
services.core.depends_on contains an invalid type, it should be an array

Expected behavior
A clear and concise description of what you expected to happen.
Docker information (please complete the following information):
$ docker --version
Docker version 19.03.13, build 4484c46d9d
$ docker-compose --version
docker-compose version 1.26.2, build eefe0d31

Additional context
Although docker-compose 1.26.2 is not the latest version (and upgrading to 1.27.4 does fix the issue), 1.26.2 is only a few months old so I would expect lots of people are still running it

Thanks for catching this. It is actually due to versions prior to 1.27.1 not having a more recent default schema (release notes). We will update the version number, after which, it should work with 1.22.0 and newer.


Recreate LabelStudio test fixture without vcr

The vcr package was erroring on import with the bump to urllib3==2.0.2. It was needed to create a fixture for a single test. This was disabled here. Recreate this fixture and/or test without the need for vcr.

Commenting to let other people know I'll work on this issue. I haven't used VCR before and am a bit rusty on tests and mocks, so it may take me a day or two.

Hey @Coniferish, thanks for jumping in to this one! From reviewing I believe these test instructions are largely about generating "recording" the initial response from the service as a test artifact test_unstructured/vcr_fixtures/cassettes/label_studio_upload.yaml. This is then played back and tested against using that artifact in subsequent runs.

Hey @ryannikolaidis. I'm having trouble figuring out how to run this test with a live instance of label-studio, kill label-studio, and then run the same test again. I'm wondering if (and/or why) label-studio needs to be running initially. It will be a lot easier for me to make a test with pytest mocks based on the label_studio_upload.yaml if that info can be treated as static and isn't something that needs to be updated each time this test is supposed to run.

Hey @Coniferish. If I am following correctly, I don't think this was intended to be re-generated every time the test ran, but rather these steps are a way of refreshing label_studio_upload.yaml in case at some point that response changes down the road. So when running the tests this is treated as a static artifact, but there is a nice way to refresh it outside of the test. That said, I don't think we need to be beholden to recreating the test exactly as-is, as long as we can recreate the coverage and update if/as needed.

Hey @ryannikolaidis. I'm still thinking through how to rewrite this test and just want to check in. There very well could be something obvious I'm missing/am simply not understanding because I haven't written tests for external services before but I'm wondering if I can get your thoughts.
This is what (I understand) is currently happening in the test...

We're creating an instance of the label_studio Client (which is from their sdk) and starting a fresh project.
Then we create the label_studio_data from the elements fixture (which is tested in the other tests).
We then import the data into the project we just created.
Finally, a regex pattern is used to check that there's a 201 response in the caplog to test that we can successfully post the data via the Client/project.

I've been brushing up on mocks, patching, and I still want to read up more on the vcr, responses, and pact packages and "consumer contract testing" generally. That said, I'm thinking project.import_task() needs to be patched to validate the data we're passing to it and I need to return/write to the log that it was successfully posted. How to validate the data and return the correct response depending on whether or not it is compliant with the sdk client/project is something I need to figure out and would appreciate any pointers on if you have any.
I'm also a little unsure about how this is functionally different from test_convert_to_label_studio_data, though, if that test ensures we're correctly converting the data. If test_convert_to_label_studio_data passes, then it seems like we should always be able to post to the project. I'm hoping you can help clarify what that difference should be for this test then... It feels like this test has a lot more setup for interacting with a running instance of Label Studio, but if that is going to be mocked, then it might just be a moot point to test POSTing data to it if we're verifying in another test that unstructured.staging.label_studio is correctly formatting data.

Oh, good points! So I think the real value of this test (as opposed to test_convert_to_label_studio_data) is actually in that the mocked response can be regenerated simply by deleting that yaml. Agreed that if we just one-time hardcode the mock, this provides a lot less value. Peeking at docs, responses looks like a really promising alternative and seems to be well maintained. That seems like a smart route to head down.
@yuming-long I know this was a while since you originally wrote, but any other thoughts here?

Hi @Coniferish
The intention of this test is to show that LabelStudio staging brick works with the LabelStudio Python SDK, which means that you can install the LabelStudio SDK and upload the data processed by our brick function (which is label_studio.stage_for_label_studio(elements) using the LabelStudio SDK. I think thats the main difference between the vcr test and  test_convert_to_label_studio_data, as you can see the latter one didn't actually update the data to a LabelStudio instance.
Also for mock tests, you are right that this can be tested with a mock test and that was also my first thought. I think the reason why i didn't continue with that idea is that the data you uploaded must follow the project format you created, which is not captured (or hard to) by mock test. This the mock test i wrote before (which was too loose to capture the format): https://github.com/Unstructured-IO/unstructured/blob/d7f87bf4949781bf3d2fc9480f3465ea215bb4ae/test_unstructured/staging/test_label_studio.py#L34
With that, the idea we had is to test working against a running label studio, then recording the interactions with VCR. The good thing about vcr is wouldn’t need to run label-studio in CI, but you do record the vcr fixtures (cassettes) against an actual running label-studio locally. It’s also very easy to re-record if label-studio ever changes, you just update a fixture to record mode per test .

Hey @yuming-long,
I don't have much experience with logging, so I've been confused how messages were being captured by caplog and then used for the assert. I have the test working with responses, but nothing is being logged to caplog with it. I looked at project.import_tasks and it returns a list of int's, representing the Imported task IDs or raises and error or exception. Would it be okay to just assert that the IDs are returned?
    imported_ids = project.import_tasks(label_studio_data)
    # imported_ids >>> [1, 2]

.../python3.8/site-packages/label_studio_sdk/project.py:
def import_tasks(self, tasks, preannotated_from_fields: List = None):
        """Import JSON-formatted labeling tasks. Tasks can be unlabeled or contain predictions.

        Parameters
        ----------
        tasks: list of dicts | dict | path to file
            Tasks in
            Label Studio JSON format

        preannotated_from_fields: list of strings
            Turns flat task JSON formatted like: `{"column1": value, "column2": value}` into Label Studio prediction
            data format: `{"data": {"column1"..}, "predictions": [{..."column2"}]`
            Useful when all your data is stored in tabular format with one column dedicated to model predictions.

        Returns
        -------
        list of int
            Imported task IDs

        """
        params = {'return_task_ids': '1'}
        if preannotated_from_fields:
            params['preannotated_from_fields'] = ','.join(preannotated_from_fields)
        if isinstance(tasks, (list, dict)):
            response = self.make_request(
                method='POST',
                url=f'/api/projects/{self.id}/import',
                json=tasks,
                params=params,
            )
        elif isinstance(tasks, (str, Path)):
            # try import from file
            if not os.path.isfile(tasks):
                raise LabelStudioException(f'Not found import tasks file {tasks}')
            with open(tasks, mode='rb') as f:
                response = self.make_request(
                    method='POST',
                    url=f'/api/projects/{self.id}/import',
                    files={'file': f},
                    params=params,
                )
        else:
            raise TypeError(
                f'Not supported type provided as "tasks" argument: {type(tasks)}'
            )
        return response.json()['task_ids']

Well in my opinion i think a 200 code for a success POST is easier to understand. But if that's the only option, i am okay with that, just please make sure to comment note on that. But i am just curious since youa re testing with responses, i think you will have a success 200 code right?

Hey @yuming-long and/or @ryannikolaidis
Could one of you take a look at the requests and label-studio Client packages with me or something? I'm struggling to figure out how to make sure the response objects are being called correctly for this test. I found how to access the individual responses that are being mocked and how to check if they were called and the status code, but the one we're trying to check for this test (/api/projects/.*/import) isn't being called for some reason--I'm seeing this when I run the test and enter pdb, I am checking response.call_count and it is returning 0 while all the other mock responses are showing they're called once.
This is also confusing because project.import_tasks(label_studio_data) is correctly showing that 2 tasks were added.
I'm going trough the docs and using dir(<object>) to figure out what methods to potentially use, so any advice in terms of how to approach solving this would also be appreciated--I know I might not have the best strategy for approaching some of these problems.
# @_recorder.record(file_path=RESPONSES_YAML_PATH)
@responses.activate
def test_upload_label_studio_data_with_sdk(elements):
    """
    Testing Instructions
    ====================
    1. Remove file `test_unstructured/responses/label_studio_upload.yaml` if it exists.
        It will be recreated later.
    2. Install the label-studio package by running command `pip install -U label-studio`.
    3. Run command `label-studio`, and login or set up label studio account on pop-up website.
    4. Update the `LABEL_STUDIO_URL` and `API_KEY` below. You can find your API_KEY by
        clicking into your account profile in Label Studio.
    5. Run this test once, and Responses will record the HTTP request to the yaml file.
    6. Kill the label studio instance, alternate (uncomment/comment) this test's decorators
        and uncomment line 35 (responses._add_from_file(file_path=RESPONSES_YAML_PATH)),
        and run the test again. Responses will replay the response.
    """
    responses._add_from_file(file_path=RESPONSES_YAML_PATH)
    # Define the URL where Label Studio is accessible
    LABEL_STUDIO_URL = "http://localhost:8080"
    # API_KEY is a temporary key from local install not actually valid anywhere
    # Update it if it is updated in your user account for Label Studio
    API_KEY = "7b613506d5afa062fe33c9cd825f106c718b82a0"
    # Connect to the Label Studio API and check the connection
    ls = Client(url=LABEL_STUDIO_URL, api_key=API_KEY)
    ls.check_connection()
    ls.delete_all_projects()
    # Create a sample project to classify types of texts
    project = ls.start_project(
        title="Text Type Classifications",
        label_config="""
        <View>
        <Text name="text" value="$text"/>
        <View style="box-shadow: 2px 2px 5px #999;
                       padding: 20px; margin-top: 2em;
                       border-radius: 5px;">

            <Choices name="type" toName="text"
                    choice="single" showInLine="true">
            <Choice value="Title"/>
              <Choice value="Narrative"/>
            </Choices>
        </View>
        </View>
        """,
    )
    label_studio_data = label_studio.stage_for_label_studio(elements)
    imported_ids = project.import_tasks(label_studio_data)
    pattern = re.compile(r"/api/projects/.*/import")
    if responses.registered():
        # Find the response for posting to the /imports api
        for response in responses.registered():
            if pattern.search(response.url):
                resp = response
                break
            else:
                resp = None
        import pdb; pdb.set_trace()
        assert resp.status == 201
    # If responses is not used and you are using a live instance of Label Studio (recording responses)
    else:
        assert len(imported_ids) == 2

More specifically, this is what I'm stuck on: https://stackoverflow.com/questions/76342385/how-do-you-test-a-response-thats-not-returned-by-lower-method

Hey @Coniferish! I just saw vcrpy got it's first (and immediately second) release since last August and looks like they have resolved urllib3 issues: https://github.com/kevin1024/vcrpy/releases/tag/v4.3.1. Maybe worth a peek to see if we can pull the latest and re-enable?
Another note: on the main branch we've temporarily bumped back down to urllib3==1.26.16 for other conflict reasons, but I would like to resolve this issue for urllib>=2 regardless, because I'm sure it's a matter of time before we need to bump there again.

Hey @ryannikolaidis. Just seeing this, but sounds good!
I'm unsure how to check that we can pull and use the latest version, though. Are you saying I should test it locally to see if the new version still works or should I be more detailed and look through their commits/updates (for both urllib3 v1 and v2)? I am noticing that they say there is an issue running urllib3 v2 on Python older than 3.10 and we are (or at least I am) using 3.8.15 (per Unstructure's env instructions). I haven't done much with making sure requirements are up to date and compatible for projects before, so any general guidance is appreciated.

Hey @Coniferish. Oh I was thinking simpler than that. I was thinking of just re-enabling the vcr test and seeing if it still fails. according to the release notes over there, they have resolved the original urllib3 issue that was causing this to fail (which is why we needed to disable in the first place)

Hey @ryannikolaidis
The test is passing, but I'm getting this warning:
test_unstructured/staging/test_label_studio.py ..................                   [100%]

==================================== warnings summary =====================================
../../.pyenv/versions/3.8.15/envs/unstructured/lib/python3.8/site-packages/requests/__init__.py:109
  /home/coni/.pyenv/versions/3.8.15/envs/unstructured/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.16) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== 18 passed, 1 warning in 1.87s ==============================

and I checked that my vcr version was the latest one:
pip show vcrpy
Name: vcrpy
Version: 4.3.1

sweet!

Hey @ryannikolaidis
The test is passing, but I'm getting this warning:
test_unstructured/staging/test_label_studio.py ..................                   [100%]

==================================== warnings summary =====================================
../../.pyenv/versions/3.8.15/envs/unstructured/lib/python3.8/site-packages/requests/__init__.py:109
  /home/coni/.pyenv/versions/3.8.15/envs/unstructured/lib/python3.8/site-packages/requests/__init__.py:109:
  RequestsDependencyWarning: urllib3 (1.26.16) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== 18 passed, 1 warning in 1.87s ==============================

and I checked that my vcr version was the latest one:
pip show vcrpy
Name: vcrpy
Version: 4.3.1

sweet! oh I think that's unrelated to this change.


Switching between AND and OR for facet value filters

I see now PivotViewer does a
facetsFilter = facetAfilter AND facetBfilter AND facetCfilter...
where
facetfilter = hasValueA OR hasValueB OR hasValueC etc.
would be nice if I could have a global option in the UI (like a toggle button), or if I could toggle OR/AND behaviour for hasValue BOOLOPERATION hasValueB BOOLOPERATION hasValueC ... (for the facetFilter combination I think having only AND is ok). Would probably even be adequate to set that search behaviour per facet in the .cxml and just have the filter UI show an icon on what bool operation it does for that facet filter's selected values
as an example see
http://gallery.clipflair.net/activity.html
(from http://ClipFlair.net project - same gallery in Silverlight is at http://gallery.clipflair.net/activity)
There I have "Keywords" facet. One would expect to be able to search for items that have both keywords X and Y, but now they can only search and find items that have keyword X or keyword Y. Would be nice to be able to change that
Where is that query logic located? Would it be easy to do such a thing without changing too many things in too many places? (a global toggle button or boolean flag/API would be the easiest I guess to place)

This behavior comes from the .some(...) call at PivotView.js:381. If we changed it to .every(...), it would do the other style of filtering.
I agree that having UI to change the operator per facet would be nice. I'm less keen on having it specified in the CXML just because the user might not understand why some filters work differently than others.

some more discussion I was having on this with researchers from the Trafilm project (http://gallery.trafilm.net - scroll down to the 3rd gallery there, the L3ST-instance one to which the discussion refers and see the "L3ST functions" facet)

can’t query for “character portrayal + non-stereotypical” (having "character portrayal" and "stereotype" values for "L3ST functions"), since PivotViewer  doesn’t have a way to say “not this value” (only has checkbox to select a value) at filter for a facet
A way to get a glimpse of which are non-stereotypical character portrayal ones is to first select “stereotype” at L3ST (or L3TT) functions facet in PivotViewer and then also tick the “character portrayal”. That way you see an animation where the non-stereotype character portrayal are added to the view (btw any non-character portrayal stereotypes are not removed from the view since it does an OR operation for selected values in a facet filter). Not exactly handy, but gives you a clue on which are those items. Works though if there aren’t many items I guess
Can’t offer a better workaround for that, best would be if Pivot instead of a normal checkbox had a three state checkbox going between empty, ticked and X (negation) states for a value. That would also need to be combined with an AND/OR switch for each facet filter (and probably an AND/OR switch at the top of the filters list too since it always does an AND operation to combine the facet filters). Haven’t bothered to try adding such functionality to the HTML5 PivotViewer though since it might become too technical like that, plus there would be an issue with the URL it produces for a current filter selection (it doesn’t currently contain any info on the AND/OR, it just implies an AND between facet filters and an OR between values selected for a certain facet filter)

so from the discussion quoted above, the extra suggestion related to this issue would be "instead of a normal checkbox have a three state checkbox going between empty, ticked and X (negation) states for a value" - that way one could query for negation of a value (and combined with AND/OR switch at a specific  facet filter [where OR is implied now] and at all filters level [where AND is implied now] it would allow much more flexible queries)


DNS Resolution is very slow for container names -- >10 seconds

I'm testing out your DNS Proxy Server but it appears to be very slow for docker container hostname resolution.  I can resolve Internet hostnames in an instant but container hostname resolution takes over 10 seconds before I start getting replies to my ping requests.
I'm on Ubuntu 16.04.4 LTS if that matters.
Let me know what I can provide to help track down the issue.

@ak2766 Thanks for use DPS
You take 10 seconds in which scenario? by accessing the browser, by running a ping command, nslookup?
I never got any delay in DNS resolution from the browser or nslookup from terminal, but I notice some delay in ping command testing it right now, please confirm your test source, then we can investigate that.
Regards

Hi Mageddo.
Yes, it happens when I'm sending pings (I did mention in the post above, maybe you missed it).
I've started the dns-proxy-server as follows:
docker run -h dns.mageddo --name dns-proxy-server -p 5380:5380 -v /var/run/docker.sock:/var/run/docker.sock -v /etc/resolv.conf:/etc/resolv.conf -d defreitas/dns-proxy-server

I then started a CentOS image with the hostname pingclient.example.com as follows:
docker run -it --name pingclient -h pingclient.example.com -d centos
I then created an image based on CentOS with bind-utils and net-tools and started it:
docker run -it --name pinger -h pinger.example.com -d pinger
I then pinged pingclient from the pinger with the following results:
PING pingclient.example.com (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=3 ttl=64 time=0.096 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=4 ttl=64 time=0.102 ms

--- pingclient.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 12044ms
rtt min/avg/max/mdev = 0.043/0.075/0.102/0.026 ms

As you can see, it took 12.044s to send 4 ping most of which was spent waiting for DNS to resolve the name.

Interestingly, dig returns results very quickly:
$ docker exec -i pinger dig pingclient.example.com

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> pingclient.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12692
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;pingclient.example.com.		IN	A

;; ANSWER SECTION:
pingclient.example.com.	0	IN	A	<IP_ADDRESS>

;; Query time: 0 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Sun May 13 02:08:12 UTC 2018
;; MSG SIZE  rcvd: 78

Yet, pinging immediately after still results in a long wait before ping replies start coming back:
$ docker exec -i pinger ping -c4 pingclient.example.com
PING pingclient.example.com (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=2 ttl=64 time=0.042 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=3 ttl=64 time=0.052 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=4 ttl=64 time=0.082 ms

--- pingclient.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 12043ms
rtt min/avg/max/mdev = 0.042/0.055/0.082/0.016 ms

For completeness sake, when I ping the IP Address, response is immediate:
docker exec -i pinger ping -c4 <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.127 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.109 ms

--- <IP_ADDRESS> ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.058/0.089/0.127/0.030 ms

FYI: I've just repeated the experiment with a user defined network and ping is now super responsive:
I created a user defined network as below:
docker network create dockernet
I then added --network dockernet option as I started the containers above.
This site was instrumental in my experimentation:
User Defined Bridges
Cheers,
ak.

Update:
It appears that the fast response is not always guaranteed.  Sometimes super fast other times I get that same ~10 second delay before I get my responses.

@ak2766 I think I got it.
Basically when you ping a hostname it will do a second question to DNS server asking for reverse DNS lookup eg. <IP_ADDRESS>.in-addr.arpa., as it doesn't exists, remote server returns an error code, DPS code was with a bug that returns nothing when it receives an error code from remote server then ping timeouts after 10s and keep going.
I will pre-release version 2.5.1 and keep testing.
Thank you very much for your contribution.

PR

Thank you very much for your contribution.

Thank you much more for creating a tool that makes my life really easy while getting to know Docker.

I've been running this peridically (stopping and starting all containers prior to testing):
for u in {1..1000}; do ping -f -c 100 pingclient.example.com 2>&1 | egrep transmitted; done
I haven't seen a number > 2ms.
Solid fix muchly appreciated.


Unable to Run ./aws-eks-na-cli ebpf loaded-ebpfdata On Node

We had recently upgraded the AWS CNI plugin to v1.15.4-eksbuild.1 on our 1.25 cluster, and then enabled enforcing network policies via the addon configuration.
I was walking through some of the examples (just to poke around a bit) found in the README here: https://github.com/aws/aws-network-policy-agent#network-policy-agent-cli
I went onto my managed worker node (via Systems Manager), and ran the ./aws-eks-na-cli ebpf loaded-ebpfdata command and received the following error:
2024-01-23 14:35:36.129352413 +0000 UTC m=+0.000841813 write error: can't rename log file: rename /var/log/aws-routed-eni/ebpf-sdk.log /var/log/aws-routed-eni/ebpf-sdk-2024-01-23T14-35-36.129.log: permission denied
2024-01-23 14:35:36.129486858 +0000 UTC m=+0.000976247 write error: can't rename log file: rename /var/log/aws-routed-eni/ebpf-sdk.log /var/log/aws-routed-eni/ebpf-sdk-2024-01-23T14-35-36.129.log: permission denied
2024-01-23 14:35:36.129530591 +0000 UTC m=+0.001019990 write error: can't rename log file: rename /var/log/aws-routed-eni/ebpf-sdk.log /var/log/aws-routed-eni/ebpf-sdk-2024-01-23T14-35-36.129.log: permission denied

I tried it as sudo as well, and received the following:
Failed to execute the cmd -  failed walking the bpfdirectory unable to get FD

This may be a non-issue, or a self-inflicted one, but I wanted to just reach out to make sure I'm not missing something obvious.
Thank you!

Environment:

Kubernetes version (use kubectl version): v1.25.16-eks-8cb36c9
CNI Version: v1.15.4-eksbuild.1
Network Policy Agent Version: Not Sure
OS (e.g: cat /etc/os-release): Amazon Linux 2
Kernel (e.g. uname -a): 5.10.199-190.747.amzn2.x86_64

This issue is fixed with 1.0.8-rc image. Will be cutting a final release soon.

This issue is fixed with 1.0.8-rc image. Will be cutting a final release soon.

Ah cool - thanks for the quick response.
I assume it's nothing that is causing any issues with the network policies being enforced themselves, just with running the CLI on the node?

Cool - thanks a bunch @jayanthvn.  I'm going to mark this one as closed.  Have a good one!


Worksheet Mode

The Scala Worksheet seems to be popular in both Eclipse and IntelliJ.  Users may find it useful if ScalaFiddle supported this mode as well.
For example, this would allow learners to follow along with videos for Functional Programming Principles in Scala coursera course using just a web browser.

You can use the new Scala Center Scastie for a more worksheet like experience. Give it a try!


3.7

add branch 3.7 for Laravel 5.7
please update package in packagegist

Sorry it took me so long, just been super busy.


Basic alert message related functions

critical and basic alert message functions from W3C webdriver specification.
16.1 Dismiss Alert : ˋdismiss_alert()ˋ
16.2 Accept Alert : ˋaccept_alert()ˋ
16.3 Get Alert Text : ˋget_alert_text()ˋ

Thanks for contributing to the crate! Do you think you could add some integration tests for these endpoints to /tests/webdriver_client_integration.rs? Let me know if you need help or don't have time to do them.

Actually, I've found some time so I can write the tests. Thanks for the code!


[Python docs feedback]

Problem description

Problem location

I saw a problem at the following URL:

I couldn't find the information I wanted. I expected to find it near the following URL

Other:

Project version(s) affected

Suggestions for a fix

Hi team,
I need small help from you. I am learning Docker from your documentation. Now I am learning build image from PYTHON application. I build the image using Dockerfile. In the next module(run containers), I am unable to run the image using container. I have attached the screenshot of the error here.

Team, kindly help in this.
Thanks and Regards,
Ganesh Revanth

Hi @ganeshrevanth123, thanks for sharing the issue. For support and troubleshooting, please try the forums or Slack. If you find any issues in the docs, please create an issue in this repo.

https://dockercommunity.slack.com/
https://forums.docker.com/

Please try the above resources for support. A few quick things to check/include, are:

In test the application, does the app run outside of the container?
Copy and paste the commands to ensure there is no typo. For example, check the capitalization of Flask and flask. It also helps if you include the contents of your Dockerfile, requirements, and py source file for others to troubleshoot.


enable standalone artifacts

Fixes: #231
Description
To undo some confusion around https://github.com/opendatahub-io/model-registry-bf4-kf/pull/289 and re-enable standalone artifact support on both the /model_artifacts endpoint and the go core API.
TODO:

[x] Test standalone artifacts through REST API (via Python)
[x] Test late binding of standalone artifacts to model versions (in Go and via REST)
[x] Update REST response status code

How Has This Been Tested?

Merge criteria:

All the commits have been signed-off  (To pass the DCO check)

[ ] The commits have meaningful messages; the author will squash them after approval or in case of manual merges will ask to merge with squash.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work.
[ ] Code changes follow the kubeflow contribution guidelines.

If you have UI changes

[ ] The developer has added tests or explained why testing cannot be added.
[ ] Included any necessary screenshots or gifs if it was a UI change.
[ ] Verify that UI/UX changes conform the UX guidelines for Kubeflow.

thank you for walking through the changes together @isinyaaa ,
well noted the modifications in https://github.com/kubeflow/model-registry/pull/376/commits/fdd02095dddf6b8a4b4b30bf0f665578a5b0ec33 are needed so that the POST modelversion/../artifact start to behave in upsert fashion
while none of the "base resource" are used in payload for std POST or PATCH, so the modification from that commit is compatible
nor impose any change on the goverter generated code

Before:

After:

Looks fine to me. And seems to be working the same.


Websocket connection through Proxy Server

Webex Bot version: 0.3.3
Python version:  3.9
Operating System: Windows

Is there an existing way to configure the webex_bot to access webexapis.com through a proxy server? The webexteamssdk module has this functionality but I have not seen anything similar in webex_bot:
proxy = {'https': 'http://<proxy_ip>:<proxy_port>'}
api = WebexTeamsAPI(access_token=WEBEX_TEAMS_ACCESS_TOKEN,proxies=proxy)
import os
import Token

from webex_bot.webex_bot import WebexBot

webex_token = Token.BOT_WEBEX_TEAMS_ACCESS_TOKEN

bot=WebexBot(webex_token,approved_domains=['domainhere.com'])
bot.run()

Without a proxy connection, I get the following error when I execute bot.run().
"ConnectionError: HTTPSConnectionPool(host='webexapis.com', port=443): Max retries exceeded with url: /v1/people/me (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002570BB6FB50>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))"

I believe this is not possible currently due to webex_bot using the "websockets" library which does not support a proxy.
https://websockets.readthedocs.io/en/stable/reference/limitations.html?highlight=proxy
https://github.com/aaugustin/websockets/issues/364

Correct. If you have a proxy, I think websockets is not the way you want to go.

@Sly-Turtle you can use websocket-client library instead. To use websocket-client with asyncio, you can look at wsapi.py as a reference.


Videos disappear on Android when scrolling inside a FlatList with removeClippedSubviews set to true

Videos disappear on Android when scrolling inside a FlatList with removeClippedSubviews set to true.
Environment
Environment:
OS: macOS High Sierra 10.13.3
Node: 8.11.1
Yarn: 1.5.1
npm: 5.6.0
Watchman: Not Found
Xcode: Xcode 9.3 Build version 9E145
Android Studio: 3.1 AI-173.4720617
Packages: (wanted => installed)
expo: ^26.0.0 => 26.0.0
react: 16.3.0-alpha.1 => 16.3.0-alpha.1
react-native: https://github.com/expo/react-native/archive/sdk-26.0.0.tar.gz => 0.54.2
Steps to Reproduce
I am using a FlatList to display a list of videos with removeClippedSubviews set to true. But on Android the videos start to disappear when you start scrolling around. See this snack and scroll up and down a bunch: https://snack.expo.io/@darkwata/expo-video-disappearing
Expected Behavior
Videos do not disappear.
Actual Behavior
Videos disappear on scroll.
Reproducible Demo
https://snack.expo.io/@darkwata/expo-video-disappearing

Might be relevant https://github.com/facebook/react-native/issues/3979


Add packages needed for Debian (deb) packages signing.

Description
This commit adds two new packages to be installed on Debian (and its derivatives e.g. Ubuntu) which are needed by Omnibus to sign the newly built packages. The newly added packages are binutils (to supply the GNU ar binary) and gnupg to ensure that GnuPG is installed.
Issues Resolved
https://github.com/chef/omnibus/issues/402
Check List

[x] All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
[x] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[x] All commits have been signed for the Developer Certificate of Origin. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

Signed-off-by: Krzysztof Wilczynski<EMAIL_ADDRESS>

Related to https://github.com/chef/omnibus/pull/799.

Paging @scotthain for review.
/cc @WarheadsSE

@chef-cookbooks/engineering-services

@scotthain I think we can just merge this. It's not a lot in terms of a scope for the change.


Adding Support to run tests with Security

Signed-off-by: Aditya Jindal<EMAIL_ADDRESS>Description
Runs tests with clusters having Security Plugin enabled.
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

CI in Local Fork: Here


Allow customising from address like in Thunderbird.

Checklist

[X] I have used the search function to see if someone else has already submitted the same feature request.
[X] I will describe the problem with as much detail as possible.
[X] This issue only contains a request for one single feature, not multiple (related) features.

App version
6.000
Problem you are trying to solve
Unable to customise from address to use a domain alias.
Suggested solution
Allow customising the address with a drop-down option like in Thunderbird.
Screenshots / Drawings / Technical details
No response

See #935