This updates and implements CsvExtractor to extract data for EIA forms 176, 191 and 757A. The extractors can take a subset of years from the ETL settings.
As part of this we abstracted the ExcelExtractor and CsvExtractor, along with corresponding metadata objects into GenericExtractor and GenericMetadata.
Testing
See unit tests, results from local Dagster executions.
# To-do list
- [x] Ensure docs build, unit & integration tests, and test coverage pass locally with `make pytest-coverage` (otherwise the merge queue may reject your PR)
- [x] For significant ETL changes, ensure the full ETL runs locally
- [ ] For major data coverage & analysis changes, [run data validation tests](https://catalystcoop-pudl.readthedocs.io/en/latest/dev/testing.html#data-validation)
- [x] If updating analyses or data processing functions: make sure to update or write data validation tests
- [x] Update the [release notes](../docs/release_notes.rst): reference the PR and related issues.
- [x] Review the PR yourself and call out any questions or issues you have

Thanks for wading into our extractors!
process_raw, process_renamed and process_final are functions that we tend to use when something is strange in the original data that prevents us from effectively extracting it - tons of empty columns we don't need to carry around, column names with . in them, etc. I think as much as possible preserving this functionality in the GenericExtractor would be ideal.
I'd suggest setting up a generic extraction function that is subclassed for both Excel and Csv but is composed of the following steps:

load - opens the file. Will be completely different for both data sources. I think this should probably be an @abstractmethod
process - includes simplify_columns, process_raw and process_renamed, as well as the append step. Identical for both, so this can just get moved into the GenericExtractor.
validate - includes all the column checks, also identical for both.
combine - combines all the datasets, and includes process_final_page.

If that's too big of a project, at the least I'd suggest adding self.cols_added to the GenericExtractor and pulling all the column checks into validate(self, df) in the GenericExtractor to cut down on redundancy. Then we could drop the processing steps from the CSV Extractor and revisit that question when we need to.
Some other small changes to fix problems Dagster will bark at you about (annoyingly, I can't make suggestions on files you haven't changed, so pardon the long list):

In pudl.metadata.sources, add the working_partitions to the current eia176 dictionary entry: "working_partitions": {"years": sorted(set(range(1997, 2023)))} - this is basically all the years of data we expect to have.
You'll want to make EIA 176 a settings object in pudl.settings.py - see Eia923Settings as an example of what this should look like. This should also get added to EiaSettings
In pudl.package_data.settings eia176 should get added to both the fast and full ETL to mirror the other EIA years.

Thanks for all the input. I've taken a pass at reconciling the extractor classes and I think it'll be relatively straightforward to wrap that up and see how it does in Dagster. That's how I'll spend Monday.
I'm keen on delivering something consolidated and clean, but if this drags out at all, I think it will be relatively easy to fall back to a less consolidated version.

I think this is making good progress. I think I've wrapped up what we need for unit tests. Abstracting the functionality has created more to review but I expect to wrap up soon.
Let me know if you need me to switch up the pace and/or approach. I still have some follow-ups to address from above, and then I will run some Dagster jobs locally to check behavior. I'm planning on tackling that tomorrow evening.

Looks like there's a tiny docs issue but otherwise this direction is looking great so far! I'm going to ping another infrastructurally-minded person to take a look this week and see if they have additional feedback.

I've got the DAG working locally for all years (screenshot). I thought I'd be able to preview the output in the UI but I'm not quite finding that; when I peek at the output location at least, I see eia176-looking data of some formatting/encoding.
I plan on filling out the PR description tomorrow evening, along with any comments that I think will be helpful in the formal PR. The documentation step has failed again, despite a hail-mary attempt at fixing it. I'll have to find a better way to track down the unexpected indentation it's complaining about.
Finally, I noticed our Zenodo records have multiple versions. Right now we're pointing to a concrete version (instead of the more generic pointer to the latest version). I'm curious how we handle newer version.

I still need to do the implementations for EIA 191 and 757A to close out the ticket. Then I need to go through the PR checklist.

This looks great to me! I think after the last few little changes, we're going to want to run first the fast ETL and then the full ETL + validations to make sure that this PR doesn't introduce any unexpected changes into all the other datasets. If the full ETL proves too hard on your local machine we can kick off a build on Batch, so just let me know if you want me to do that.
I also took a look through our docs and it doesn't seem like any changes are required, so you should be all good to go there.

FYI I plan on wrapping things up here sometime this week.

I was able to run ferc_to_sqlite_fast and etl_fast successfully locally. I started with a clean slate under the latest conditions, having run make install-pudl, reactivating by pudl-dev environment, blowing away preexisting SQLite DBs, and running alembic upgrade head to get the latest schema. One step did fail along the way, but I was able to patch the bad reference easily and rerun that step for a successful overall materialization (see screenshots).
I had a slightly gnarly merge conflict I was able to resolve, but I wanted to call out here for good measure. I captured the relevant changes from conflicts with https://github.com/catalyst-cooperative/pudl/pull/3272 -- namely using the calamine engine to read Excel files. I'm motivated to get this merged before another conflict (😅). I'll run the full ETL tonight and see how it does.

etl_full had a couple issues (after rerunning some steps successfully following timeouts from Zenodo):
dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "core_ferc1__yearly_depreciation_changes_sched219":
The above exception was caused by the following exception:
AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.056631                      0.058                     False                  0.045143                               0.045                               True                         0.012066                                        0.7                                    False                       0.543337                                      1.0                                   False

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "core_ferc1__yearly_operating_revenues_sched300":
The above exception was caused by the following exception:
AssertionError: Found errors while running tests on the calculations:
                                                                             error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group            table_name                                     group_value
utility_id_ferc1 core_ferc1__yearly_operating_revenues_sched300 275                 0.033333                       0.17                     False                   0.02507                                0.02                               True                              0.5                                        1.0                                    False                            0.0                                      1.0                                   False

I think it had previously been suggested some tolerance tests were too sensitive? I can look further into it.

@davidmudrauskas I'm going to kick off a full build on this branch to see if I can reproduce the calc check failures you're seeing. The code changes you made shouldn't touch the FERC data much at all, so I can look into this and see what else is going wrong.

I caught some issues running integration tests (via make pytest-coverage):
=========================================================================================== short test summary info ============================================================================================
FAILED test/integration/etl_test.py::TestCsvExtractor::test_extract_eia176 - AttributeError: module 'pudl.extract.csv' has no attribute 'get_table_file_map'
FAILED test/integration/etl_test.py::TestExcelExtractor::test_extract_eia860 - AttributeError: 'DataFrame' object has no attribute 'sheet_names'
FAILED test/integration/etl_test.py::TestExcelExtractor::test_extract_eia923 - AttributeError: 'DataFrame' object has no attribute 'sheet_names'
========================================================== 3 failed, 81 passed, 4 skipped, 6 xfailed, 2 xpassed, 1843 warnings in 15398.31s (4:16:38) ==========================================================

I was able to patch them, add a couple more for the new extractors, and run the failed and new ones successfully. I'm getting through make pytest-validate now, and once anything from that is settled I'll look for a final review and approval to unblock merging.

Looks like I still have to execute ferc_to_sqlite_full locally and then re-execute the failed steps in my etl_full execution to get things to a proper state to run validations without failure. Working through that.

@davidmudrauskas If it becomes difficult to do this locally or more trouble than it's worth, we can rerun the full build on Google Batch. No trouble at all, just let me know if you'd like me to kick off a build!

@e-belfer Yeah let's do that, thanks. I think I've run into some stale artifacts and running the full jobs/tests/validations would be a lift.

@davidmudrauskas No trouble at all, I'll kick things off now and merge them in if everything passes.

The batch build from this afternoon just passed, so I've queued this to merge.


Usage help

I will be using shellcode.bin generated with Donut, I can rename this to blob.bin but I am unsure about how to perform the XOR with default key 0x08
Thanks in advance for your help.

Hi @Simon-Davies , what I did was alter the line in the source code that performs the XOR decode. That way it doesn't have to be Xored. I think I was able to do it, but I couldn't tell if the shellcode was working or not.


Compatibility with Roots

I'm  just wondering if you have come across the Roots framework, and if so, do you have any idea on how to integrate your package into Roots?
I'm sure lots of the Roots guys would love a package like this, myself included.
I've attempted integration myself but this error is far as I have got:
The "templates" directory does not exist in xxxx./wp/wp-faker/vendor/twig/twig/lib/Twig/Loader/Filesystem.php on line 94

Nevermind! Figured it out.  Awesome package. Thank you!
By the way, faker needs to be updated in composer.json. V.1.5 has some errors.


"http_auth_ldap: Authentication timed out"

Hello!
It looks like nginx_auth_ldap module sometimes fails to detect LDAP server connection failure.
nginx version: nginx/1.8.0
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --add-module=nginx-auth-ldap-be8ff8eecb2578304d72e1f4947466ac9ba50295 --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_geoip_module --with-http_sub_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E

Module configuration:
  ldap_server ldap01 {
    url ldaps://ldap01.example.com/ou=user,dc=example,dc=com?uid?sub?(objectClass=posixAccount);
    binddn "cn=nginx,ou=system,dc=example,dc=com";
    binddn_passwd "secret";
    #group_attribute gidNumber;
    #group_attribute_is_dn on;
    require valid_user;
  }

  auth_ldap_cache_enabled on;
  auth_ldap_cache_expiration_time 10000;
  auth_ldap_cache_size 1000;

Access configuration:
location /secured {
    auth_ldap "Forbidden";
    auth_ldap_servers ldap01;
    return 204;

    location /secured/_.gif {
            empty_gif;
    }
}

How it looks from client side:
$ curl -vvv<EMAIL_ADDRESS>* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 80 (#0)
* Server auth using Basic with user 'test.98'
> GET /secured/_.gif HTTP/1.1
> Authorization: Basic dGVzdC45ODpJeVJPOFZ3bWptOW1Ca3g=
> User-Agent: curl/7.37.1
> Host: <IP_ADDRESS>
> Accept: */*
>
* Empty reply from server
* Connection #0 to host <IP_ADDRESS> left intact
curl: (52) Empty reply from server

How it looks in debug log:
log_err.txt
How it looks after nginx restart:
log_ok.txt

I'm having the same timout issue. Did you find a solution?

Actually not :(

I somehow fixed it and got it working, after undergoing rigorous trial and error. Of course, by the time it started working, I did not remember my original configuration.
Have you tried using non-ssl?

i met this problem too...

My setup works correctly on first start up,  but if I come back an hour later, nginx is completely hung up.

For me, I had to change things around a bit:

URL should only contain dc info. IE: ldaps://ipa.example.nl/dc=example,dc=nl?uid?sub?(objectClass=person)

give binddn a user id: binddn "uid=admin,cn=users,cn=accounts,dc=example,dc=nl";

I run FreeIPA, and the key to finding this info for me, was running ldapsearch -x uid=admin on the FreeIPA/LDAP server.

Hello
I meet same issue with nginx 1.13.6.
http_auth_ldap: Authentication timed out
the 1st time after setup, nginx and ldap works well.
but after several hours, error log show ldap error.
the browser can pop up the login window, but not content show in browser.
anyone has a solution?
Thanks.

I have same trouble with timeout.
After start, nginx and ldap works well, but some times later error log show ldap error.

@wu-chapb says that you may change the port from 389 to 3268 in issues/113.
I cant workaround with that cause my LDAP server is not AD.


Update lower bound for estimator to also include RC0

Updating the version for estimator lower bound for 2.9 release

For future, please don't use "update " title/commits. See https://cbea.ms/git-commit/


Consolidate Bucket and Name Definitions

Create a top-level helloRetail.yml for cross-project reference management (and use throughout)
Declare the deployment bucket in helloRetail.yml
Use references to other serverless.yml's for service names
Use references to other serverless.yml's for export names

WARNING: dependent on external PR (will add link after creation)

PR is https://github.com/serverless/serverless/pull/3208

Please make sure that this consumes a version of SLS which supports the "self-reference" feature. Thanks!

Good point @gwsii!  Commit updated.  Had expected to hold off this merge until the sls merge but this makes sense, assuming that merge is accepted.  Off to smooth the skids on that!

Okay looks good to me. Thanks, Erik.


Migrate MarkerArrayDisplay

This PR migrates the MarkerArrayDisplay to ros2/rviz.
Closes #104.
In this context we noticed that both the MarkerDisplay and the MarkerArrayDisplay are able to display MarkerArray messages, the feature being kind of hidden in the first case.
@wjwwood: Any idea why the MarkerDisplay is able to additionally display MarkerArray messages? It feels like the feature does not provide any value since the MarkerArrayDisplay exists as well.

CI:

Linux
Linux-aarch64
macOS
Windows

CI after rebase:

Linux
Linux-aarch64
macOS
Windows

@wjwwood We added the last commit (7ba84ff) only the make the tests pass. This commit resolves an undetected merge error from last night and will be published as a pr of its own. The last commit will cause a conflict once the other pr is merged. Simply remove the last commit from this pr before merging will work.


Publish 7.3-20211002230125+0000

Publish 7.3-20211002230125+0000

@bot-gradle test and merge

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test and merge

OK, I've already triggered a build for you.


io.quarkus:quarkus-qute-parent:pom:2.11.0.Final was not found in https://repository.jboss.org/nexus/content/groups/public/

Describe the bug
io.quarkus:quarkus-qute-parent:pom:2.11.0.Final was not found in https://repository.jboss.org/nexus/content/groups/public/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of jboss-public-repository-group has elapsed or updates are forced

Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
GraalVM version (if different from Java)
No response
Kogito version or git rev (or at least Quarkus version if you are using Kogito via Quarkus platform BOM)
No response
Build tool (ie. output of mvnw --version or gradlew --version)
No response
Additional information
No response

@xunfeng1980 well if the artifact is on Central, it should download it, even if Nexus is not up to date ...
Did you try to remove your local repository and retry ? sometimes the issue is Maven and if any issue downloading the artifact, it does not retry ...
I that is not the case, could you please share a reproducer ?

remove local repository and download form Central is ok，but why download form JBoos，I find some config on pom.xml
<!-- distributionManagement section -->
  <distributionManagement>
    <repository>
      <id>jboss-releases-repository</id>
      <name>JBoss Releases Repository</name>
      <url>https://repository.jboss.org/nexus/service/local/staging/deploy/maven2/</url>
    </repository>
    <snapshotRepository>
      <id>jboss-snapshots-repository</id>
      <name>JBoss Snapshot Repository</name>
      <url>https://repository.jboss.org/nexus/content/repositories/snapshots/</url>
    </snapshotRepository>
  </distributionManagement>

  <repositories>
    <!-- Bootstrap repository to locate the parent pom when the parent pom
      has not been build locally. -->
    <repository>
      <id>jboss-public-repository-group</id>
      <name>JBoss Public Repository Group</name>
      <url>https://repository.jboss.org/nexus/content/groups/public/</url>
      <layout>default</layout>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>never</updatePolicy>
      </releases>
      <snapshots>
        <enabled>true</enabled>
        <updatePolicy>daily</updatePolicy>
      </snapshots>
    </repository>
  </repositories>

remove local repository and download form Central is ok，but why download form JBoos，I found some configuration in pom.xml

  <!-- distributionManagement section -->
  <distributionManagement>
    <repository>
      <id>jboss-releases-repository</id>
      <name>JBoss Releases Repository</name>
      <url>https://repository.jboss.org/nexus/service/local/staging/deploy/maven2/</url>
    </repository>
    <snapshotRepository>
      <id>jboss-snapshots-repository</id>
      <name>JBoss Snapshot Repository</name>
      <url>https://repository.jboss.org/nexus/content/repositories/snapshots/</url>
    </snapshotRepository>
  </distributionManagement>

  <repositories>
    <!-- Bootstrap repository to locate the parent pom when the parent pom
      has not been build locally. -->
    <repository>
      <id>jboss-public-repository-group</id>
      <name>JBoss Public Repository Group</name>
      <url>https://repository.jboss.org/nexus/content/groups/public/</url>
      <layout>default</layout>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>never</updatePolicy>
      </releases>
      <snapshots>
        <enabled>true</enabled>
        <updatePolicy>daily</updatePolicy>
      </snapshots>
    </repository>
  </repositories>

JBoss is used for our snapshot artifacts. that is why there is this configuration in repositories.
distributionManagement is mandatory as this is where we are releasing the artifacts


Journey OST

Journey OST 🦇
🎼 Music Composer:  Austin Wintory
👍🏻 Best for:  High Concentration Works 🔥
🔗 Links: Youtube

That's an Awesome playlist.

Please submit a pull request. 😊


[branch-2.9][fix][deploy]: fix the pid occupied check when use pulsar-daemon start or stop process.

Motivation
Cherry-pick #14701

@Nicklee007  PTAL :-)


Fixing responses for update management swagger

This fix will allow SDK to through proper exception so we can see correct error in PS Cmdlets.

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[x] The title of the PR is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[x] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[x] I have read the contribution guidelines.
[x] My spec meets the review criteria:

[x] The spec conforms to the Swagger 2.0 specification.
[x] The spec follows the guidelines described in the Swagger checklist document.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-libraries-for-java

Automation for azure-sdk-for-python

Automation for azure-sdk-for-node

Automation for azure-sdk-for-go

@anuchandy Thanks!, yes, this is the exact behavior we want.


Code flow in angular and refreshtoken

Hi,
I read the documentation to configure the client oidc, here:
https://manfredsteyer.github.io/angular-oauth2-oidc/docs/additional-documentation/code-flow-+-pcke.html
I use the code flow, not implicit flow.
but I have some problem after the user authenticate in the identity server and he is redirect to the angular client.  In the console it show me the error "Bad request: 400. invalid_grant"
At the moment all the client is protected. (if the user is not authenticated he is redirect to the login page which is on a separate application)
When the app Angular start it executes this code (oauthService is of type OAuthService) :
        this.oauthService.configure(authConfig);
        this.oauthService.tokenValidationHandler = new JwksValidationHandler();
        this.oauthService.loadDiscoveryDocumentAndTryLogin().then(() => {
            if (this.oauthService.hasValidAccessToken()) {
                this.oauthService.loadUserProfile().then((t) => {
//code
                });
            } else {
                this.oauthService.initCodeFlow();
            }
        }).catch((err) => {
//enter here, when I return after authentication
            console.log(err);
        });
        this.oauthService.setupAutomaticSilentRefresh();

authConfig is defined as:
export const authConfig: AuthConfig = {

    issuer: environment.issuer,
    sessionChecksEnabled: true,
    redirectUri: window.location.origin + '/',
    clientId: 'clientIdDefinedInServer',
    responseType: 'code',
    scope: 'openid profile email offline_access roles impersonation ScopeApiCustom',
    showDebugInformation: !environment.production,
};

In identity server the client has the offline_access set to true.
What is my error? I must use this.oauthService.initCodeFlow(); in another way?

If you get an invalid_grant message, then your server is not set up to do Code Flow with PKCE from the origin you're coming from (I think/suppose/guess).
The relevant RFC suggests this explanation for when servers should respond in that way:

invalid_grant
The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.

Your usage of this library at first sight seems okay, so I'd look for a solution in reconfiguring the server, possibly asking a question on StackOverflow with a full repro.

Hi,
Thanks for the reply. The server can use Code flow with PCKE (it is an identity server 4) and, of course, the client is set to use it (server side and client side).
I try to obtain more information about the problem:

If I manually refresh the page  the user obtain his claim abd the access token
I put (it's just a try, I'm not very skilled about authentication) this.oauthService.initCodeFlow(); inside the catch. It print the error, redirect to the identity server, redirect to the client and the user is authenticated, but I also see token and scope required by the client in the address bar (see the attachment, where you can see also, obfuscated, the email of the user).

Sorry for my english :-)
I doubt it's a server problem, I guess I made a mistake in the configuration of the angular client

Your ...AndTryLogin() call should lead to this code that handles code flow info in the URL hash fragment, and I presumed the invalid_grant comes in response to the code being exchanged for a token?
You could try to compare your approach with my attempt at Code flow, it talks with https://demo.identityserver.io/ which is also IdentityServer4.
If all this fails I recommend creating a minimal repro for us (e.g. on StackBlitz, talking to the demo identityserver) - without a way to reproduce your scenario it's a bit of a guessing game for us.

Hi,
sorry for the lack of update.
My approach is based on your attempt :)
Tomorrow I can try another time with another approach.

Hi, a co-worker report me this problem in the same project:

angular-oauth2-oidc.js:1354 Error performing password flow TypeError: Cannot read property 'toLowerCase' of null

That's strange, I do not use password flow (I post my auth.config.ts ). I am not very skilled about OAuth, I hope i do not misunderstand something important.

Hi, a co-worker report me this problem in the same project:

angular-oauth2-oidc.js:1354 Error performing password flow TypeError: Cannot read property 'toLowerCase' of null

That's strange, I do not use password flow (I post my auth.config.ts ). I am not very skilled about OAuth, I hope i do not misunderstand something important.

I have the same problem, Even though I use code flow, I get password flow error

Same happens to me

@urdarinda It may be due to Jwks Tokenvalidator, Since you using
     this.oauthService.tokenValidationHandler = new JwksValidationHandler();
You need specify your token format or skip JwksValidatorHandler

We have the same setup for our project and after some of the same errors you guys mentioned, the refresh token is working for us.
The only things we needed were defining the offline_access scope and using the setupAutomaticSilentRefresh method.
Importend to note is propably the order, in which the methods are called:
this.oauthService.configure( environment.authentication );
this.oauthService.setupAutomaticSilentRefresh();
this.oauthService.tokenValidationHandler = new JwksValidationHandler();

Thx all for the extra help! I can confirm that previous comment suggests an order of calls that I also recommend. Could also try v9 to see if the merged PRs solved your issues.
If the issue persists let us know, here or in a fresh issue with a fresh (minimal) repro if you can!


Fixes #2681 - Circumvent cache if not yet preloaded

I am not completely happy with this patch and still have to test the migration but it should demonstrate the problem.

I tested changed it to using an Option. If you like the design, I will fix the tests.

:+1: for adding missing test cases!


millstone producing wrong item counts, including vanilla recipes

Describe the Bug
As written already on discord, if a millstone has a chanced output and its count is higher than 1, the actual produced output ignores the count and always returns only 1.
create-fabric-0.5.1-c-build.1160+mc1.19.2
Reproduction Steps
As above, tested with vanilla data\create\recipes\milling\charcoal.json
Expected Result
Item count not being ignored.
Screenshots and Videos

Crash Report or Log
No response
Operating System
Windows 10
Mod Version
0.5.1f
Minecraft Version
1.19.2
Other Mods
No response
Additional Context
No response

Looking more into this, it looks like the count is being counted, but as a random range.
With a count of 4 I get mixed results of 1-4. As if it takes count as a range.
I still don't think this is the way it's meant to work, even if at least it's not truly game breaking.

Seems to be intended, I was pointed to this:
https://github.com/Creators-of-Create/Create/blob/d66170c8fed5d02ad46f44fd0e66685be88a3170/src/main/java/com/simibubi/create/content/processing/recipe/ProcessingOutput.java#L50-L60
The conclusion is that the counts are indeed variable, but they should average to the same ratio, and so over time the output ends up balanced.


Added test for abstract broker.

Signed-off-by: Pavel Kirilin<EMAIL_ADDRESS>

Codecov Report

Merging #52 (903a6fa) into release/0.1.2 (a1ef1b0) will increase coverage by 0.84%.
The diff coverage is 100.00%.

@@                Coverage Diff                @@
##           release/0.1.2      #52      +/-   ##
=================================================
+ Coverage          59.21%   60.05%   +0.84%
=================================================
  Files                 51       52       +1
  Lines               1319     1337      +18
=================================================
+ Hits                 781      803      +22
+ Misses               538      534       -4

Impacted Files
Coverage Δ

taskiq/abc/broker.py
77.90% <ø> (+2.90%)
:arrow_up:

taskiq/abc/tests/test_broker.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Implements design for city navigation

This commit adds LPBlue logo for city navigation.
https://github.com/lonelyplanet/bane/pull/80
https://github.com/lonelyplanet/destinations-next/pull/161

:+1:


[MTSRE-1170] refactor: overhaul of kubectl-package

Summary
Refactoring kubectl-package command structure in preparation of future feature development.

Waiting on #281 for rebase


Make lead_organisations optional

We're getting rid of lead and supporting organisations.
This commit is to make sure that they're optional in our schemas.
Another piece of work to completely get rid of them will come as soon as publishing applications will have been updated to stop publishing those types of organisation.
ticket: https://trello.com/c/us3MI1n9/602-fix-organisation-tagging

:+1:


Update error text for parsing run ID

cc @ElDeveloper

Looks good, thanks! Can you add a test for that exception? Otherwise looks good to go. Thanks so much for addressing so promptly! 😃

A test was added in #65 to assert parsing of an iSeq run ID, is there a different test that should be added here?

Yes, I was thinking about a test that checks for the correct exception error message depending on the malformed input.

Thanks so much!


Issues with Component Rework: List & List-Nav

List & List-Nav:
The migration guide suggests:

                💀
                Skeleton

The css applied to list nav and the Skeleton Showcase omit the
 element though, meaning you currently have to add
.list-nav li {
	@apply flex items-center space-x-4 whitespace-nowrap;
	/* Padding */
	@apply px-4 py-3;
	/* Rounded */
	@apply rounded;
	/* Hover */
	@apply hover:bg-primary-500/10;

}

Yourself if you want to use
-wrapped anchors in navigation.
After giving it more of a try, the
 should probably just be removed from the documentation

@Sarenor You may have missed it, but I included some text under the Navigation List usage text that explain how we settled on the structure. It's due to accessibility (a11y) concerns:

Note the use of the ul tag, which is required to meet accessibility guidelines.

Per this linked page the guidelines show the following (NOTE: i've shorted this to one li/anchor):

Home

    …

And then for Nav lists they show this:

        …

So I've interpreted that as meaning they recommend the following structure for Nav lists:
nav > ul > li > a
Likewise, it's been a couple weeks, but I believe that Axe - the tool we use to test a11y - was also giving us an warning if we didn't use this structure.
It's very verbose, I agree, I'd prefer to drop the extra wrapping element if we could, but for now this appears to be the recommendation for a11y compliance.
If you can find info that states otherwise please share! I'd love to have a definitive answer on this!

I've updated the text slightly to this:

While verbose, we do recommend you use all tags showng below as meets accessibility guidelines.

Also, unrelated, but I've included instruction for how we suggest handling TW element nav list highlight states. This aligns with how we handle this for our sidebar lists:

You're right it works with

I apparently still had an issue with my CSS imports when I tested that because it looked absolutely horrible.
Still might want the skeleton showcase to also use  around the anchor tags then though.
Thanks for the response and sorry for needlessly taking your time!

Sorry I'm not sure what you're referring to with your bullet point. Where would that note go?


support secretbox

It would be cool.

Most definitely.

@benjaminapetersen this will likely be copy pasting https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/secretbox/secretbox.go followed by wiring it up as an option.


Not working

Not working now. Will be fixed soon...
TODO:

[ ] Fix tool
[ ] Rewrite code

Is there a way to donate?

Is there a way to donate?

yes there is
https://github.com/Airkek/Youtube-Viewers/blob/master/Donate.md

Keep the great job up, also does it increase video views or livestream viewrs like 20-30 people are watching you, I am confused lemme know

What happened ??? not increase live viewer please help

What happened ??? not increase live viewer please help

You need to wait for the author to update this bot please don't bug him with such replies, make sure to star and fork to support the author or u can donate him

thank u so much for working on the problem soon its fixed and up and going i will be donating again :)

Airkek , you're saying about the url of the youutbe api right?, i tried to get it from the youtube watchtime via F12 of the brownser to compare but i still couldn't, it ends up calling a url of the api different from what it should be
msshawon works with chromedriver and totally different, airkek sends me where you look for the link, other programs cosneguiram get it with rbtools and my tube

Airkek , you're saying about the url of the youutbe api right?, i tried to get it from the youtube watchtime via F12 of the brownser to compare but i still couldn't, it ends up calling a url of the api different from what it should be
msshawon works with chromedriver and totally different, airkek sends me where you look for the link, other programs cosneguiram get it with rbtools and my tube

i gonna buy rbtools and get watchtime url from it

That mean , problem is sloved soon ?

That mean , problem is sloved soon ?

yes

Great buddy !!

I will continue to work with yt heartbeat url

Is it work ?

it should

Ok , let's see buddy !!

I am waiting for you. Thank you very much.

Airkek , When will you give the good news that the guys is now working😅😅

I have a proposal. That is, can you create a group for those who donate to you and please share for that group?

I have a proposal. That is, can you create a group for those who donate to you and please share for that group?

I will be donating again soon as it's back up and running again 🙂

I will be donating again soon as it's back up and running again 🙂

Hihi. I think that would be more worthy for the author, right?

I will be donating again soon as it's back up and running again 🙂

Hihi. I think that would be more worthy for the author, right?

👍👍

I see bots they make now include Cookies

oh shit, rbtools working through selenium chromedriver ;c

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

Did you buy it already? Is running good?

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

Did you buy it already? Is running good?

I had one of the old model, but it was a trial and it worked practically the same, a problem that formatted the pc and I still had no experience with dissamble, this one on the site is promising everything including updating, it even has a trial version for a lower price.

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

Did you buy it already? Is running good?

I had one of the old model, but it was a trial and it worked practically the same, a problem that formatted the pc and I still had no experience with dissamble, this one on the site is promising everything including updating, it even has a trial version for a lower price.

Thanks but I'll wait till this is up and running that to much work lol

yes i'm waiting for airkek, but i also want to help with programming, i'm trying to fix url, and maybe this program could help airkek to find url faster, since it works with sockets and not youtube api, problem that i can get here in brazil purchase

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

Did you buy it already? Is running good?

I had one of the old model, but it was a trial and it worked practically the same, a problem that formatted the pc and I still had no experience with dissamble, this one on the site is promising everything including updating, it even has a trial version for a lower price.

Thanks but I'll wait till this is up and running that to much work lol

yes i'm waiting for airkek, but i also want to help with programming, i'm trying to fix url, and maybe this program could help airkek to find url faster, since it works with sockets and not youtube api, problem that i can get here in brazil purchase

yes i'm waiting for airkek, but i also want to help with programming, i'm trying to fix url, and maybe this program could help airkek to find url faster, since it works with sockets and not youtube api, problem that i can get here in brazil purchase

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

Did you buy it already? Is running good?

I had one of the old model, but it was a trial and it worked practically the same, a problem that formatted the pc and I still had no experience with dissamble, this one on the site is promising everything including updating, it even has a trial version for a lower price.

Thanks but I'll wait till this is up and running that to much work lol

yes i'm waiting for airkek, but i also want to help with programming, i'm trying to fix url, and maybe this program could help airkek to find url faster, since it works with sockets and not youtube api, problem that i can get here in brazil purchase

Oh okay yea more help the better

I just need to know how the url heartbeat is built, I don't know the youtube api, I need an example at least, because in the c# code, I saw the url, but I can't pull the watchtime url from the original youtube, it changes some parameters and even why the bot stopped

I just need to know how the url heartbeat is built, I don't know the youtube api, I need an example at least, because in the c# code, I saw the url, but I can't pull the watchtime url from the original youtube, it changes some parameters and even why the bot stopped

Well hoping the author replies soon with good news 😊

I also agree I donated to airkek during the time the viewbot stop working to thank him when the viewbott stopped working  it means a lot to me that program and unlike these other people who say they will donate to you once you make the program work they there lieing Low lives Air kek they will not donate an they tease an say they would donate but airkek is not a dumbass so shut up with that ill donate thinking its gonna make airkek work any faster with your fake lies an  promises of donations I believe it’s their fault that made the program stop working because they used it so often so much from what I understand Air kek has a twitch view bot but it’s private I feel like if you donate to airkek on his PayPal and leave your email with a message maybe  he will find it in his heart to share/offer a view of of around 10$ American lets be fair an pay in pay pal it or bitcoins and if you can’t afford to do this then you shouldn’t be bugging airkek to make him work faster for free you know who you are and maybe with the kindness airkek will offer send us a program for YouTube or  for Twitch  let me know what's up air  kek I got bit coins to donate I got PayPal $ I'm down an also I just miss using your program and unlike these other people I will donate to you before during and after thank you let me know air 💨 kek

I also agree I donated to airkek during the time the viewbot stop working to thank him when the viewbott stopped working it means a lot to me that program and unlike these other people who say they will donate to you once you make the program work they there lieing Low lives Air kek they will not donate an they tease an say they would donate but airkek is not a dumbass so shut up with that ill donate thinking its gonna make airkek work any faster with your fake lies an promises of donations I believe it’s their fault that made the program stop working because they used it so often so much from what I understand Air kek has a twitch view bot but it’s private I feel like if you donate to airkek on his PayPal and leave your email with a message maybe he will find it in his heart to share/offer a view of of around 10$ American lets be fair an pay in pay pal it or bitcoins and if you can’t afford to do this then you shouldn’t be bugging airkek to make him work faster for free you know who you are and maybe with the kindness airkek will offer send us a program for YouTube or for Twitch let me know what's up air kek I got bit coins to donate I got PayPal $ I'm down an also I just miss using your program and unlike these other people I will donate to you before during and after thank you let me know air 💨 kek

I completely agree, and as I said I'm trying to help with the code, because I know how hard it is, that is, if I can find it for sure I would play for airkek, just to make it easier for him

oh shit, rbtools working through selenium chromedriver ;c

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

This is scam. Developers runs away with crypto delivering same Airkek bot with new gui. Avoid payments

oh shit, rbtools working through selenium chromedriver ;c

I understand, this is the same as mshawon, I tried to buy this one, but my region does not allow https://software.dbtechlabs.com/youtube-view-bot.html
yt heartbeat url, where can I look, or better what should this url look like?

This is scam. Developers runs away with crypto delivering same Airkek bot with new gui. Avoid payments

I dont think the author is scamming us lol

I just bought it https://software.dbtechlabs.com/youtube-view-bot.html -- and it runs pretty bad. There are viewers but not like = the code here.

I just bought it https://software.dbtechlabs.com/youtube-view-bot.html -- and it runs pretty bad. There are viewers but not like = the code here.

Well let's hope the author gets this one fix soon

How is the rebuilding coming 😊

Do you think the new rewrite will be released today? 😊🤞

Do you think the new rewrite will be released today? 😊🤞

no

Do you think the new rewrite will be released today? 😊🤞

no

Okay 😊

When rewrite sill be released ?? Can u please update about it , any progress ?

-this actually works: https://anonfiles.com/t1H1pc3bu1/YT_BOT_PAID_zip
https://www.youtube.com/watch?v=Ce9hr19L9JI

-this actually works: https://anonfiles.com/t1H1pc3bu1/YT_BOT_PAID_zip
https://www.youtube.com/watch?v=Ce9hr19L9JI

Is this real?

@bangbang113 I think so for me it worked fine.

-this actually works: https://anonfiles.com/t1H1pc3bu1/YT_BOT_PAID_zip
https://www.youtube.com/watch?v=Ce9hr19L9JI

It worked. But do not share the source code.

-this actually works: https://anonfiles.com/t1H1pc3bu1/YT_BOT_PAID_zip
https://www.youtube.com/watch?v=Ce9hr19L9JI

It worked for me too. Is there any website for good free proxies or cheap stable proxies? I haven't been able to find them

https://proxyscrape.com/

do u think the new relese will be out today?

-this actually works: https://anonfiles.com/t1H1pc3bu1/YT_BOT_PAID_zip
https://www.youtube.com/watch?v=Ce9hr19L9JI

Where do you see it? It is really great. Source View suggested and available in real time a lot


Build on OpenSearch 1.0.0

opensearch-security pull request intake form
Please provide as much details as possible to get feedback/acceptance on your PR quickly

Category: (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Github Issue # or road-map entry, if available:

Description of changes:

Why these changes are required?

What is the old behavior before changes and new behavior after changes? (Please add any example/logs/screen-shot if available)

Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)

TO-DOs, if any: (Please describe pending items and provide Github issues# for each of them)

Is it backport from main branch? (If yes, please add backport PR # and commits #)

By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1304 (a861b62) into main (1083d99) will increase coverage by 0.01%.
The diff coverage is n/a.

:exclamation: Current head a861b62 differs from pull request most recent head 2fe070d. Consider uploading reports for the commit 2fe070d to get more accurate results

@@             Coverage Diff              @@
##               main    #1304      +/-   ##
============================================
+ Coverage     64.73%   64.75%   +0.01%
+ Complexity     3195     3194       -1
============================================
  Files           247      247
  Lines         17230    17230
  Branches       3045     3045
============================================
+ Hits          11154    11157       +3
+ Misses         4526     4523       -3
  Partials       1550     1550

Impacted Files
Coverage Δ

...ecurity/configuration/ConfigurationRepository.java
73.07% <0.00%> (-2.20%)
:arrow_down:

...security/configuration/DlsFlsFilterLeafReader.java
60.47% <0.00%> (+0.70%)
:arrow_up:

...nsearch/security/dlic/rest/api/AuditApiAction.java
68.08% <0.00%> (+4.25%)
:arrow_up:

...ecurity/configuration/StaticResourceException.java
25.00% <0.00%> (+25.00%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1083d99...2fe070d. Read the comment docs.


Child FFD deltas not calculated with the same point set as parent.

Previously known issue written here for documentation.
The Problem
When working with a child FFD embed in a parent FFD, a sub set of the points embedded in the parent will also be embed in the child.  When calculating the additional movement of embed points due to the modification of the child FFD, the child FFD will first update the points embed inside of itself according to how the control points of the child were moved when the parent deformed. However, updating the points according to how the control points of the child were moved will not produce a point set identical to those embed in the parent! So when the child calculates deltas (additional position modifications) and they are added to the modifications caused by the motion of the parent FFD,  the deltas will not have the intended geometric effect. For example in cases where the parent FFD does shape modifications and the child rotates a subset of these points (think morphing LE/TE), the rotation will skew the shape modifications. An example of this is shown in the picture below.

When it's an Issue

when combining parent shape variables with child shape deformation that isn't uniform for each point (child shape variables and rotation will cause issues, but translation will not).
Especially apparent when using large child FFD modifications

Work Around

use a linear spline surface for the Child FFD  (works well for rotations, but not for child shape variables)

long term fix

change the data structure for child FFDs so that it references the same point set as the parent (same object in memory) but only works with a sub set of these points (the ones inside the child FFD). This way the points that the child FFD uses to calculate its additional modifications will always be the same as the points warped by the parent deformation.

@joanibal can you post the global design variable function you used in the child?
While writing the tutorial I wanted to understand why global variable funcs that use += operations work properly. When self.update() is called, the FFD coefficients are set back to their original (baseline) locations and then the global DVs are computed in order of their addition, then sectionlocal and local variables. So you don't accumulate changes across iterations via += operations.
On the other hand, child FFDs do some weird stuff that I don't totally understand, but I don't think it's doing the same reset. So if you used += or *= operations in your child global dv funcs, it could lead to some weird behavior.
        if not self.isChild:
            self.FFD.coef = self.origFFDCoef.copy()
            self._setInitialValues()
        else:
            # Update all coef
            self.FFD._updateVolumeCoef()

            # Evaluate starting pointset
            Xstart = self.FFD.getAttachedPoints(ptSetName)

            if self.complex:
                # Now we have to propagate the complex part through Xstart
                tempCoef = self.FFD.coef.copy().astype('D')
                Xstart = Xstart.astype('D')
                imag_part = numpy.imag(tempCoef)
                imag_j = 1j

                dPtdCoef = self.FFD.embededVolumes[ptSetName].dPtdCoef
                if dPtdCoef is not None:
                    for ii in range(3):
                        Xstart[:, ii] += imag_j*dPtdCoef.dot(imag_part[:, ii])

Here are the functions I used with child FFDs

def twist_flap(val, geo):
    geo.rot_z['flap_axis'].coef[:] = val[:] - 29.5

def translate_x_flap(val, geo):
    C = geo.extractCoef('flap_axis')
    C[:, 0] += val[:] + 0.15421028
    geo.restoreCoef(C, 'flap_axis')

def translate_y_flap(val, geo):
    C = geo.extractCoef('flap_axis')
    C[:, 1] += val[:] - 0.0816069
    geo.restoreCoef(C, 'flap_axis')

def twist_slat(val, geo):
    geo.rot_z['slat_axis'].coef[:] = val[:] + 29.0

def translate_x_slat(val, geo):
    C = geo.extractCoef('slat_axis')
    C[:, 0] += val[:] - .0911552
    geo.restoreCoef(C, 'slat_axis')

def translate_y_slat(val, geo):
    C = geo.extractCoef('slat_axis')
    C[:, 1] += val[:] - 0.0735696
    geo.restoreCoef(C, 'slat_axis')


TN-3077 2FA interrupting staff account initiation.

Solve problems with new staff user's coming in on invite link, verifying name & dob, promoting auth details into invited account, etc., by suspending 2FA during this bootstrap process.
NB - this means the initial new staff user interaction WILL require them to validate (name & dob) but will NOT require them 2FA.  It would be a significant change to alter that and so, I hope this is acceptable.

@pbugni  can you please remind me what the lifetime is of invite URLs and 2FA codes?

...
NB - this means the initial new staff user interaction WILL require them to validate (name & dob) but will NOT require them 2FA. It would be a significant change to alter that and so, I hope this is acceptable.

@mcjustin
For the access tokens (url_tokens) we include in emails, they're good for 30 days.
For the 2FA 6 digit one time use token, 30 minutes

@pbugni  Thanks for the details.

Do passwords reset URLs use the same 30 day expiry?
Based on https://github.com/uwcirg/truenth-portal/pull/4085, I think the pw reset flow includes 2FA before they can proceed post-reset, is that correct?

@mcjustin the reset-password functionality is built into flask-user.  The expiration on those defaults to 2 days (can be overridden w/ config:
lib/python3.7/site-packages/flask_user/settings.py:    um.reset_password_expiration   = sd('USER_RESET_PASSWORD_EXPIRATION',  2*24*3600)   # 2 days

The password reset flow includes the challenge (first, last, DOB), but does NOT include 2FA.

@pbugni  please merge, since this is a blocking error for new staff.
I'm going to create another story for the longer term, to consider requiring a re-login anytime the user sets their pw (aka registers), as we do after resets.  Flow w/ screenshots here. This, so we have 2FA, since these invite emails can be out in the wild for 30 days.


fix: select cmd by type

We should run cmd only for specific server type.
Run on DomainController:
Before:

After:

fixes #65

lgtm


Placing classes in their own files

Due to the large amount of code in a single file, the library can be difficult to read and understand for most developers. This is not all the refactoring work that needs to be done, but it is a good starting point.

@RobertoPrevato hi!
Can you review this PR? I think it's a good improvement that will help attract new contributors

Hi,
Sorry for taking so long to reply here. In the last months I've been focused on blacksheep, in my already limited time for open source projects.
I agree there is the need to simplify this library, and I was planning to improve it (as a side note, I also have an idea that should improve its performance considerably). But I won´t merge this pull request because having one class definition per Python module is not a coding style I want to adopt. I would instead group classes and functions in modules by purpose.
I welcome contributions and I hope to not disappoint anybody. I invite to first ask maintainers' opinion before working on a general overhaul of a library like this one. 🙏🏼


"HCI_USB" sample doesn't compile with "nucleo_wb55rg" board

Description
Using "west" to compile the example "hci_usb" (located in samples/bluetooth) fails with the "nucleo_wb55rg". However, it compiles with the "nrf52840_pca10056" board for example.
To Reproduce
Steps to reproduce the behavior:

Open terminal
Go to the "zephyr" project path.
Execute zephyr_env.sh to create ZEPHYR_TOOLCHAIN_VARIANT, ZEPHYR_BASE and ZEPHYR_SDK_INSTALL_DIR variables.
west build -p auto -b nucleo_wb55rg samples/bluetooth/hci_usb
Compilation fails

Environment:

OS: Ubuntu 18.04 LTS
Zephyr library version 2.1.99

Console output
-- west build: making build dir /home/hfernandez/Projects/zephyrproject/zephyr/build pristine
-- west build: build configuration:
source directory: /home/hfernandez/Projects/zephyrproject/zephyr/samples/bluetooth/hci_usb
build directory: /home/hfernandez/Projects/zephyrproject/zephyr/build
BOARD: nucleo_wb55rg (origin: command line)
-- west build: generating a build system
Zephyr version: 2.1.99
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.4")
-- Selected BOARD nucleo_wb55rg
-- Found west: /home/hfernandez/.local/bin/west (found suitable version "0.6.3", minimum required is "0.6.0")
-- Loading /home/hfernandez/Projects/zephyrproject/zephyr/boards/arm/nucleo_wb55rg/nucleo_wb55rg.dts as base
Devicetree configuration written to /home/hfernandez/Projects/zephyrproject/zephyr/build/zephyr/include/generated/generated_dts_board.conf
Parsing Kconfig tree in /home/hfernandez/Projects/zephyrproject/zephyr/samples/bluetooth/hci_usb/Kconfig
Loaded configuration '/home/hfernandez/Projects/zephyrproject/zephyr/boards/arm/nucleo_wb55rg/nucleo_wb55rg_defconfig'
Merged configuration '/home/hfernandez/Projects/zephyrproject/zephyr/samples/bluetooth/hci_usb/prj.conf'
Configuration saved to '/home/hfernandez/Projects/zephyrproject/zephyr/build/zephyr/.config'
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc
-- Cache files will be written to: /home/hfernandez/.cache/zephyr
CMake Warning at ../../../subsys/usb/CMakeLists.txt:27 (message):
CONFIG_USB_DEVICE_VID has default value 0x2FE3.
This value is only for testing and MUST be configured for USB products.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hfernandez/Projects/zephyrproject/zephyr/build
-- west build: building application
[1/148] Preparing syscall dependency handling
[63/148] Building C object zephyr/CMakeFiles/z...phyr.dir/drivers/usb/device/usb_dc_stm32.c.obj
FAILED: zephyr/CMakeFiles/zephyr.dir/drivers/usb/device/usb_dc_stm32.c.obj
ccache /opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc -DBUILD_VERSION=zephyr-v2.1.0-210-g5f629c892993 -DCORE_CM4 -DHSE_VALUE=32000000 -DKERNEL -DSTM32WB55xx -DUSE_FULL_LL_DRIVER -DUSE_HAL_DRIVER -D_FORTIFY_SOURCE=2 -D__PROGRAM_START -D__ZEPHYR__=1 -I../include -I../include/drivers -Izephyr/include/generated -I../soc/arm/st_stm32/stm32wb -I../drivers -I../ext/hal/cmsis/Core/Include -I../subsys/usb -I../subsys/bluetooth -I/home/hfernandez/Projects/zephyrproject/modules/hal/stm32/stm32cube/stm32wbxx/soc -I/home/hfernandez/Projects/zephyrproject/modules/hal/stm32/stm32cube/stm32wbxx/drivers/include -I/home/hfernandez/Projects/zephyrproject/modules/hal/stm32/stm32cube/stm32wbxx/drivers/include/Legacy -I/home/hfernandez/Projects/zephyrproject/modules/hal/stm32/lib/stm32wb/hci -isystem ../lib/libc/minimal/include -isystem /opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/8.3.0/include -isystem /opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/8.3.0/include-fixed -Os -imacros/home/hfernandez/Projects/zephyrproject/zephyr/build/zephyr/include/generated/autoconf.h -ffreestanding -fno-common -g -mthumb -mcpu=cortex-m4 -imacros/home/hfernandez/Projects/zephyrproject/zephyr/include/toolchain/zephyr_stdint.h -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wno-main -Wno-pointer-sign -Wpointer-arith -Wno-unused-but-set-variable -Werror=implicit-int -fno-asynchronous-unwind-tables -fno-pie -fno-pic -fno-strict-overflow -fno-reorder-functions -fno-defer-pop -fmacro-prefix-map=/home/hfernandez/Projects/zephyrproject/zephyr/samples/bluetooth/hci_usb=CMAKE_SOURCE_DIR -fmacro-prefix-map=/home/hfernandez/Projects/zephyrproject/zephyr=ZEPHYR_BASE -ffunction-sections -fdata-sections -mabi=aapcs -march=armv7e-m -std=c99 -nostdinc -MD -MT zephyr/CMakeFiles/zephyr.dir/drivers/usb/device/usb_dc_stm32.c.obj -MF zephyr/CMakeFiles/zephyr.dir/drivers/usb/device/usb_dc_stm32.c.obj.d -o zephyr/CMakeFiles/zephyr.dir/drivers/usb/device/usb_dc_stm32.c.obj   -c /home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:153:44: error: 'DT_USB_NUM_BIDIR_ENDPOINTS' undeclared here (not in a function)
struct usb_dc_stm32_ep_state out_ep_state[DT_USB_NUM_BIDIR_ENDPOINTS];
^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c: In function 'usb_dc_stm32_clock_enable':
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:199:10: error: 'DT_USB_CLOCK_BUS' undeclared (first use in this function); did you mean 'DT_ADC_1_CLOCK_BUS'?
.bus = DT_USB_CLOCK_BUS,
^~~~~~~~~~~~~~~~
DT_ADC_1_CLOCK_BUS
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:199:10: note: each undeclared identifier is reported only once for each function it appears in
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:200:10: error: 'DT_USB_CLOCK_BITS' undeclared (first use in this function); did you mean 'DT_SPI_2_CLOCK_BITS'?
.enr = DT_USB_CLOCK_BITS,
^~~~~~~~~~~~~~~~~
DT_SPI_2_CLOCK_BITS
In file included from ../include/arch/arm/irq.h:18,
from ../include/arch/arm/arch.h:27,
from ../include/arch/cpu.h:17,
from ../include/kernel_includes.h:34,
from ../include/kernel.h:17,
from ../include/device.h:11,
from ../include/drivers/usb/usb_dc.h:20,
from ../include/usb/usb_device.h:39,
from /home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:49:
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c: In function 'usb_dc_stm32_init':
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:389:14: error: 'DT_USB_IRQ' undeclared (first use in this function); did you mean 'TSC_IRQn'?
IRQ_CONNECT(DT_USB_IRQ, DT_USB_IRQ_PRI,
^~~~~~~~~~
../include/sw_isr_table.h:70:5: note: in definition of macro 'Z_ISR_DECLARE'
{irq, flags, &func, (void *)param}
^~~
../include/irq.h:51:2: note: in expansion of macro 'ARCH_IRQ_CONNECT'
ARCH_IRQ_CONNECT(irq_p, priority_p, isr_p, isr_param_p, flags_p)
^~~~~~~~~~~~~~~~
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:389:2: note: in expansion of macro 'IRQ_CONNECT'
IRQ_CONNECT(DT_USB_IRQ, DT_USB_IRQ_PRI,
^~~~~~~~~~~
In file included from ../include/arch/arm/arch.h:27,
from ../include/arch/cpu.h:17,
from ../include/kernel_includes.h:34,
from ../include/kernel.h:17,
from ../include/device.h:11,
from ../include/drivers/usb/usb_dc.h:20,
from ../include/usb/usb_device.h:39,
from /home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:49:
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:389:26: error: 'DT_USB_IRQ_PRI' undeclared (first use in this function); did you mean 'DT_SPI_2_IRQ_PRI'?
IRQ_CONNECT(DT_USB_IRQ, DT_USB_IRQ_PRI,
^~~~~~~~~~~~~~
../include/arch/arm/irq.h:82:32: note: in definition of macro 'ARCH_IRQ_CONNECT'
z_arm_irq_priority_set(irq_p, priority_p, flags_p);
^~~~~~~~~~
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:389:2: note: in expansion of macro 'IRQ_CONNECT'
IRQ_CONNECT(DT_USB_IRQ, DT_USB_IRQ_PRI,
^~~~~~~~~~~
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c: In function 'usb_dc_attach':
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:433:34: error: 'LL_APB1_GRP1_PERIPH_PWR' undeclared (first use in this function); did you mean 'LL_APB1_GRP1_PERIPH_LCD'?
if (LL_APB1_GRP1_IsEnabledClock(LL_APB1_GRP1_PERIPH_PWR)) {
^~~~~~~~~~~~~~~~~~~~~~~
LL_APB1_GRP1_PERIPH_LCD
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c: In function 'usb_dc_ep_configure':
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:556:6: error: 'DT_USB_RAM_SIZE' undeclared (first use in this function); did you mean 'DT_SRAM_SIZE'?
if (DT_USB_RAM_SIZE <=
^~~~~~~~~~~~~~~
DT_SRAM_SIZE
In file included from ../include/arch/arm/irq.h:17,
from ../include/arch/arm/arch.h:27,
from ../include/arch/cpu.h:17,
from ../include/kernel_includes.h:34,
from ../include/kernel.h:17,
from ../include/device.h:11,
from ../include/drivers/usb/usb_dc.h:20,
from ../include/usb/usb_device.h:39,
from /home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:49:
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c: In function 'usb_dc_ep_write':
/home/hfernandez/Projects/zephyrproject/zephyr/drivers/usb/device/usb_dc_stm32.c:722:15: error: 'DT_USB_IRQ' undeclared (first use in this function); did you mean 'TSC_IRQn'?
irq_disable(DT_USB_IRQ);
^~~~~~~~~~
../include/irq.h:267:43: note: in definition of macro 'irq_disable'
#define irq_disable(irq) arch_irq_disable(irq)
^~~
[68/148] Building C object zephyr/arch/arch/ar...iles/arch__arm__core__cortex_m.dir/fault.c.obj
ninja: build stopped: subcommand failed.
ERROR: command exited with status 1: /usr/bin/cmake --build /home/hfernandez/Projects/zephyrproject/zephyr/build

USB is not supported on nucleo_wb55rg

@Hekkfern, indeed usb is not yet supported on nucleo_wb55rg.
Though, USB IP on STM32WB is similar to other STM32 USB IP supported on zephyr.
So if you're interested, you can find below an example on how USB could enabled on STM32WB:
https://github.com/zephyrproject-rtos/zephyr/pull/18806/commits/83585fe034fd307b43b23c91be2a8ab8effa89e1
Then, you can check USB related changes in following commit to enable support on nucleo_wb55rg:
https://github.com/zephyrproject-rtos/zephyr/pull/18806/commits/590621e644691644b09d6fd77407b7864cf126fd


Remove invalid less selector for less4j

time-grid.less has a dangling child selector (.rbc-row > {) which is technically illegal less syntax. The normal js-based less compiler everyone uses will ignore the child selector and emit css anyway, but alternative less compilers (like the java-based less4j) have stricter syntax requirements and error-out when they encounter this construct.
Patch removes the dangling child selector.

I don't know what it should be, but what it actually is compiled to today (by less compilers that accept this without error) is .rbc-row { not .rbc-row > * {. See this example.

learn some new about LESS all the time :P thanks!


New version: MSYS2.MSYS2 version 20230526

[X] Have you signed the Contributor License Agreement?
[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[X] Does your manifest conform to the 1.4 schema?

Microsoft Reviewers: Open in CodeFlow

/azp run


target URL content is not stable even if it's stable

nishant57@v2202312205092251320:~/BB/test$ ghauri -r req.txt --dbs --confirm --timeout 10 --threads 40 --level 3 --flush-session --fresh-queries --proxy "http://0T34H1Wb0zzxyz4mdc8xyzxyzq8P9:cRSxyzxyz3iCC1D_country-nz@proxy.digiproxy.cc:8082" -v 5

  ________.__                        .__  {1.3}
 /  _____/|  |__ _____   __ _________|__|
/   \  ___|  |  \\__  \ |  |  \_  __ \  |
\    \_\  \   Y  \/ __ \|  |  /|  | \/  |
 \______  /___|  (____  /____/ |__|  |__|
        \/     \/     \/         https://github.com/r0oth3x49
                                 An advanced SQL injection detection & exploitation tool.

[*] starting @ 14:49:20 /2024-01-02/

[14:49:20] [INFO] parsing HTTP request from 'req.txt'
[14:49:20] [DEBUG] {'GET': [], 'POST': [<Parameter('username')>, <Parameter('password')>, <Parameter('keepLogin')>, <Parameter('recaptcha')>, <Parameter('useRecaptcha')>], 'COOKIE': [], 'HEADER': []}
[14:49:20] [INFO] flushing session file
JSON data found in POST body. Do you want to process it? [Y/n/q] n

[14:49:23] [INFO] testing connection to the target URL
[14:49:23] [TRAFFIC_OUT] HTTP request [#1]:
POST /api/account/authenticate HTTP/1.1
Host: test.example.co.nz
Content-Type: application/json;charset=utf-8

{"username":"myemailhere@gmail.com","password":"Password@12345","keepLogin":true,"recaptcha":"","useRecaptcha":"false"}

[14:49:26] [TRAFFIC_IN] HTTP response [#1]:
HTTP/1.1 200 OK
Date: Tue, 02 Jan 2024 13:49:26 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Cache-Control: no-cache; no-store
Set-Cookie: token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXlsb2FkIjoiaDAza2JGY3VqSWFOZHhMZU90YUtXNTdJOWYxL3hETkWURYMEtvSmErWG5Hd09MQmZHQVBmaUFIZVNhTUxSVEc0cVpHWHo1QmlxY2JiTmVrODUrblBTUFA0RTJRcVJxbEVpYnlFN29JRmtWZXpJVnJta1U2Z0l0b29RZVFQRm4rTmRzWTRvekc1d3dtR0Fjd05QaG1vemNrKzcyaFp4NVhWZXQxQVdKekJqbytTQmdsSytKLy9FYVpnUnJzIiwianRpIjoiYWE4MzYzNjItNTI2ZC00OTNkLWI0YzctY2EwZWI3NzFjZmFhIiwia2VlcF9sb2dpbiI6InRydWUiLCJleHAiOjE3MTI3NjA1NjYsImlzcyI6Im1lbWJlci5zdGFyYnVja3MuY28ubnoiLCJhdWQiOiZXIuc3RhcmJ1Y2tzLmNvLm56In0.2VUHA6YI_squtjE53nZw1XEdq4N5EzV51EGnszNNMuk; expires=Wed, 03 Jan 2024 01:49:26 GMT; path=/; secure; samesite=lax
Content-Security-Policy: default-src 'self' https://www.google-analytics.com https://stats.g.doubleclick.net https://www.google.com ; script-src-elem 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; script-src 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; style-src-elem 'self' https://fonts.googleapis.com 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' https://fonts.gstatic.com; frame-src 'self' https://www.google.com https://recaptcha.net
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, AUTH_APP, AUTH_TIMESTAMP, AUTH_KEY
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=16070400; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Referrer-Policy: no-referrer
URI: https://test.example.co.nz/api/account/authenticate

[14:49:26] [INFO] testing if the target URL content is stable
[14:49:27] [TRAFFIC_OUT] HTTP request [#2]:
POST /api/account/authenticate HTTP/1.1
Host: test.example.co.nz
Content-Type: application/json;charset=utf-8

{"username":"myemailhere@gmail.com","password":"Password@12345","keepLogin":true,"recaptcha":"","useRecaptcha":"false"}

[14:49:30] [TRAFFIC_IN] HTTP response [#2]:
HTTP/1.1 200 OK
Date: Tue, 02 Jan 2024 13:49:29 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Cache-Control: no-cache; no-store
Set-Cookie: token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXlsb2FkIjoiaDAza2JGY3VqSWFOZHhMZU90YUtXNTdJOWYxL3hETk5qQ1NWWURYMEtvSmE09MQmZHQVBmaUFIZVNhTUxSVEc0cVpHWHo1QmlxY2JiTmVrODUrblBTUFA0RTJRcVJxbEVpYnlFN29JRmtWZXpJVnJta1U2Z0l0b29RZVFQRm4rTmRzWTRvekc1d3dtR0Fjd05QaG1vemVhWZXQxQVdKekJqbytTQmdsSytKLy9FYVpnUnJzIiwianRpIjoiMmE0YWY1ZDAtNTViZC00MWE1LTg1NzAtM2RjYjc5NmQyMjYwIiwia2VlcF9sb2dpbiI6InRydWUiLCJleHAiOjE3MTI3NjA1NjksImlzcyI6Im1lbWJlci5zdGFyYnVja3MuY28ubnoiLCJhdWQiOiJtZW1iZXIuc3RhcmJ1Y2tzLmNvLm56In0.ueSsSpNAZrjnOpEItDEwGFpOuANxFQMQqGW2TbDkkXA; expires=Wed, 03 Jan 2024 01:49:29 GMT; path=/; secure; samesite=lax
Content-Security-Policy: default-src 'self' https://www.google-analytics.com https://stats.g.doubleclick.net https://www.google.com ; script-src-elem 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; script-src 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; style-src-elem 'self' https://fonts.googleapis.com 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' https://fonts.gstatic.com; frame-src 'self' https://www.google.com https://recaptcha.net
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, AUTH_APP, AUTH_TIMESTAMP, AUTH_KEY
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=16070400; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Referrer-Policy: no-referrer
URI: https://test.example.co.nz/api/account/authenticate

[14:49:30] [DEBUG] r1: 737, r2: 737
[14:49:30] [WARNING] target URL content is not stable (i.e. content differs). Ghauri will base the page comparison on a textual content, Switching to 'text-only'
[14:49:30] [TRAFFIC_OUT] HTTP request [#3]:
POST /api/account/authenticate HTTP/1.1
Host: test.example.co.nz
Content-Type: application/json;charset=utf-8

{"username":"myemailhere@gmail.com'\",..))","password":"Password@12345","keepLogin":true,"recaptcha":"","useRecaptcha":"false"}

[14:49:32] [TRAFFIC_IN] HTTP response [#3]:
HTTP/1.1 401 Unauthorized
Date: Tue, 02 Jan 2024 13:49:32 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Cache-Control: no-cache; no-store
Content-Security-Policy: default-src 'self' https://www.google-analytics.com https://stats.g.doubleclick.net https://www.google.com ; script-src-elem 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; script-src 'self' https://www.googletagmanager.com https://www.google-analytics.com https://google-analytics.com https://connect.facebook.net https://recaptcha.net https://www.gstatic.com 'unsafe-inline'; style-src-elem 'self' https://fonts.googleapis.com 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' https://fonts.gstatic.com; frame-src 'self' https://www.google.com https://recaptcha.net
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, AUTH_APP, AUTH_TIMESTAMP, AUTH_KEY
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=16070400; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Referrer-Policy: no-referrer
URI: https://test.example.co.nz/api/account/authenticate

[14:49:32] [WARNING] It seems the session got expired, update the session and re-run

[*] ending @ 14:49:32 /2024-01-02/

nishant57@v2202312205092251320:~/BB/test$

@r0oth3x49 You can see in the 1st and 2nd requests, the response is same. Also ghauri states it too:
[14:49:30] [DEBUG] r1: 737, r2: 737
But then ghauri says: [WARNING] target URL content is not stable (i.e. content differs)
And in the third request it changes the whole request to:
POST /api/account/authenticate HTTP/1.1
Host: test.example.co.nz
Content-Type: application/json;charset=utf-8

{"username":"myemailhere@gmail.com'\",..))","password":"Password@12345","keepLogin":true,"recaptcha":"","useRecaptcha":"false"}

Which ofcourse is incorrect and thus the response is:
HTTP/1.1 401 Unauthorized

The issue is that even if the response is same ghauri says the content is not stable.
Another suggestion:
Idk how it verifies if the response is same, maybe via response length. But I think the better way is by doing some string similarity check. The response size/length can be different, like if some random token is added in the response that the same response will be a bit different second time. So I think string similarity check with 90-95% similarity ratio will be good.
Desktop (please complete the following information):

OS: Debian
Ghauri version which is causing the error: 1.3 (latest)

@r0oth3x49 ping

it is doing that by similarity check. not the length/size. and i would need a target to reproduce this and fix. if you are okay sharing you can share that on my email<EMAIL_ADDRESS>otherwise i can't reproduce and fix.


fix error in html comment

the end tag of a comment is published, so just removing what I think should have been commented out

@ivorb : Thanks for your contribution! The author, @diberry, has been notified to review your proposed change.

@ivorb  Thank you for submitting this fix. #sign-off


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-k8s-29-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 4290

Questions ?
Please refer to the Backport tool documentation


Implemented Patiencesort

Fixes #5
Implemented Patience Sorting algorithm
Implemented according to the description of https://en.wikipedia.org/wiki/Patience_sorting and https://en.wikipedia.org/wiki/Merge_algorithm#K-way_merging

Thank you! Apologies for taking a year before merging. :)


luna-shell sometimes fails to compile on Windows

When I run stack build luna-shell on luna/luna-core repo for a first time, I get:
luna-shell-1.2: configure (lib + exe)
Configuring luna-shell-1.2...
luna-shell-1.2: build (lib + exe)
Preprocessing library for luna-shell-1.2..
Building library for luna-shell-1.2..
[1 of 5] Compiling Luna.Shell.CWD   ( src\Luna\Shell\CWD.hs, .stack-work\dist\7d103d30\build\Luna\Shell\CWD.o )
[2 of 5] Compiling Luna.Shell.Interpret ( src\Luna\Shell\Interpret.hs, .stack-work\dist\7d103d30\build\Luna\Shell\Interpret.o )
[3 of 5] Compiling Luna.Shell.Command ( src\Luna\Shell\Command.hs, .stack-work\dist\7d103d30\build\Luna\Shell\Command.o )
ghc.EXE: internal error: IMAGE_REL_AMD64_ADDR32[NB]: High bits are set in 244506561 for ghczmprim_GHCziTypes_ZMZN_closure
    (GHC version 8.4.2 for x86_64_unknown_mingw32)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
Completed 2 action(s).

--  While building package luna-shell-1.2 using:
      C:\s\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_<IP_ADDRESS>_ghc-8.4.2.exe --builddir=.stack-work\dist\7d103d30 build lib:luna-shell exe:luna --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
    Process exited with code: ExitFailure 3

If I run then command again, the build succeeds.
I guess it might be related with TH memory usage — https://ghc.haskell.org/trac/ghc/ticket/13112 — as suggested in similar issue: https://github.com/haskell-tools/haskell-tools/issues/277

@mwu-tow could you please link it (describe the connection) in related GHC trac issue? This is definitely not our bug and we could not do much with it.

Does it always happen in Luna.Shell.Command?

Could you try whether using -fexternal-interpreter GHC option helps and/or upgrading GHC version to 8.4.3? You'll probably need to bump LTS version in stack.yaml

@iamrecursion It happens iff Luna.Shell.Command is built after Luna.Shell.Interpret by the same GHC process. I'd expect the issue lies with Interpret, as it takes ~10GB to compile. I'm not sure why it didn't come out in #231, perhaps I missed it.

It doesn’t surprise me that Interpreter results in high memory usage. It pulls in a lot of the code that uses type families and will be incurring the same penalties as the rest of the codebase.
I don;’t know if there are ways to reduce it, but we’ll definitely look into it as part of the optimisation work.

@iamrecursion Another thing that puzzles me is why sequentially compiling several modules causes the issue but compiling them one by one succeeds. It looks as if there was a severe memory leak in GHC.
@mikusp The GHC option -fexternal-interpreter does help. GHC 8.4.3 doesn't help. (but took more than two hours and one hard machine reset before failing…)

I don’t know, honestly! I think it’s a GHC problem not ours.

This is a GHC issue, not ours. Closing.


What to do when different models have different numbers of non-null pairs with the obs

@btang1 encountered this in his #238 situation.
This is not something we usually expect to happen with models, since model fields are usually non-null at all time/space and we usually configure things such that multiple models being compared span the same period. But it's not impossible, for example:

model's time starts later than the others, so when pairing is done, the model value is set to null for those times before
search radius setting is too small and prevents pairing to model value, causing model value to be set to null in the pair
the model data does have null at certain times, perhaps from a post-processing to mask unrealistic values, or something done in a MONETIO reader, in a way that's different from the other models

Ideas:

check pairs, print warning message if different non-null counts among different obs-model pairs
drop where obs + all model values are not all non-null
annotate counts in plots

We are planning to start with the warning message. And then add the option to drop.

I can start to work on drop obs( option 2).

I can start to work on drop obs( option 2).

Can you start with the warning message? Even when dropping we probably want to print a message with the info of what the differences are / what is being dropped.

Sure! I will work on warning now! Will update the result with you!
On Feb 6, 2024, at 3:21 PM, Zachary Moon @.***> wrote:
I can start to work on drop obs( option 2).
Can you start with the warning message? Even when dropping we probably want to print a message with the info of what the differences are / what is being dropped.
—
Reply to this email directly, view it on GitHubhttps://github.com/NOAA-CSL/MELODIES-MONET/issues/245#issuecomment-1930688047, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMGOO6ZERW7J5AQZV5ZSCTDYSKGDDAVCNFSM6AAAAABC4QBKFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZQGY4DQMBUG4.
You are receiving this because you were mentioned.Message ID: @.***>

I checked all AQM runs.
if model output start at 13pm, 'time' variable start at 12pm, 'time_iso' is 13pm. NO PROBLEM!
if model output start at 1am, 'time' variable start at 12pm, 'time_iso' is 1 meaning 13pm. That is the problem.
This happened in all model runs output at 1am. and could be some thing happen when output the model results. not post-processing step.

@zmoon I have a branch to add a warning, and I test it works.
code start from:   https://github.com/btang1/MELODIES-MONET/blob/PairOBS/melodies_monet/driver.py#L1052
code end at: https://github.com/btang1/MELODIES-MONET/blob/PairOBS/melodies_monet/driver.py#L1062
also changed code at : https://github.com/btang1/MELODIES-MONET/blob/PairOBS/melodies_monet/driver.py#L925


Is there an API way to renaming tabs?

Hello!
I was unable to find a way to rename query tabs via the API. Is that possible? If not, can that be a feature request?
Thanks!

Sure, the tabs object now has a rename and close functions (example: yasgui.current().rename('new title');)

Thank you, Laurens!


Test branch

Added test files for app test. Compare review and merge. Thanks

Okay... Test branch reviewed and approved for merge.
Thanks and well done.


find solution for logging out to the console only in dev mode

something like https://www.npmjs.com/package/debug perhaps?
or https://www.npmjs.com/package/better-log

https://github.com/rollup/rollup-plugin-strip does the trick 👌

fixed in 91441155a8a2ba93e2c2a88b45f4c81df80d52a6


Class Query executed does not exist

Not sure this has been reported before, but we're using some new profiling tools and are seeing this beeing reported as a 'problem'.
It's not a critical one as it's captured by a try/catch. But I'm wondering why it's even trying to lookup something like 'query executed'?

Message:
Class Query executed does not exist
Stacktrace:
ReflectionClass.__construct
/srv/www/vhosts/deployer/master/releases/123/vendor/bugsnag/bugsnag/src/Client.php:205
Bugsnag\Client.leaveBreadcrumb
/srv/www/vhosts/deployer/master/releases/123/vendor/bugsnag/bugsnag-laravel/src/BugsnagServiceProvider.php:87
Bugsnag\BugsnagLaravel\BugsnagServiceProvider.Bugsnag\BugsnagLaravel{closure}
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php:349
Illuminate\Events\Dispatcher.Illuminate\Events{closure}
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php:200
Illuminate\Events\Dispatcher.dispatch
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Database/Connection.php:825
Illuminate\Database\Connection.event
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Database/Connection.php:682
Illuminate\Database\Connection.logQuery
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Database/Connection.php:635
Illuminate\Database\Connection.run
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Database/Connection.php:333
Illuminate\Database\Connection.select
/srv/www/vhosts/deployer/master/releases/123/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:1705

Hi @danijelk,
The Laravel notifier registers a listener for queries by default to create breadcrumbs from these queries, making it easier to track down root causes of issues.  This behaviour can be disabled by setting the environment variable BUGSNAG_QUERY to false in the application configuration.
As for the reason the exception is being caught, the code in leaveBreadcrumb attempts to get a more legible class name from a given class name.  As the given name "Query executed" isn't a class name, the ReflectionClass constructor throws an exception which we suppress.  This shouldn't have an impact on the operation of Bugsnag.

Great just what I was hoping for. Will close the issue, thanks for the detailed explanation.


LocalTileLayer does not appear on map

Hello !
First of all thanks for the work, this library sounds amazing and I can't wait to try it.
However, I still have difficulties to run the introduction notebooks from my machine (Ubuntu 22).
When lauching introduction.ipynb as a jupyter notebook, I have this JavaScript error at block 7 when layer is added on the map :
Failed to create view for 'UrlView' from module 'ipyurl' with model 'UrlModel' from module 'ipyurl'
ReferenceError: process is not defined
    at e.makeSettings (http://localhost:8888/nbextensions/ipyurl/index.js?v=20230426160827:2:69011)
    at e.makeSettings (http://localhost:8888/nbextensions/ipyurl/index.js?v=20230426160827:2:67987)
    at o.get_url (http://localhost:8888/nbextensions/ipyurl/index.js?v=20230426160827:2:404124)
    at o.render (http://localhost:8888/nbextensions/ipyurl/index.js?v=20230426160827:2:403836)
    at http://localhost:8888/nbextensions/jupyter-js-widgets/extension.js?v=20230426160827:2:747630

My environment was created from scratch using /binder/environment.yml with python 3.10, notably my ipyurl version is 0.1.2 and my xarray-leaflet version is 0.1.16
Here are my jupyter versions :

IPython          : 8.12.0
ipykernel        : 6.22.0
ipywidgets       : 8.0.6
jupyter_client   : 8.2.0
jupyter_core     : 5.3.0
jupyter_server   : 1.24.0
jupyterlab       : 3.6.3
nbclient         : 0.7.3
nbconvert        : 7.3.1
nbformat         : 5.8.0
notebook         : 6.5.4
qtconsole        : not installed
traitlets        : 5.9.0

Many thanks for any help !

Hi @agirnow, thanks for the kind works!
What version of ipyurl do you have? Try installing v0.1.3, which requires ipywidgets>=7,<8.

Thanks for your reply !
I have updated ipywidgets to 7.6.0 and ipyurl to 0.1.3, and it removed the JavaScript error I had before.
However, I still do not have anything that displays on the map, executing block 7 just relocates the map to the zone of interest without displaying anything...
Do you have any idea about what can cause this ?
Many thanks,
Alexandre

Any idea @davidbrochart ?

Any error reported in the browser console?

Yes, here they are

Could you provide the output of pip list, or the equivalent if it's a conda environment?

env.txt
Here you will find the complete list of my conda environment.

Thanks, it seems that I am making progress !
Now I have a colorbar that displays but no data and map, and a collection of errors like this :
[E 16:07:11.517 NotebookApp] Error unpacking user from cookie: Expecting value: line 1 column 1 (char 0)
[W 16:07:11.517 NotebookApp] Clearing invalid/expired login cookie username-localhost-8888
[I 16:07:11.517 NotebookApp] 302 GET /xarray_leaflet/tmp/xarray_leaflet_iao94655/2/3/1.png (<IP_ADDRESS>) 1.140000ms

Sure !
env2.txt

@davidbrochart the weekend brought advice, i no longer have issues with the introduction notebooks, everything is working fine ! Many thanks for your help :)


Support for table-valued functions

First off, thanks for a great piece of software.
SQLite now supports Table-Valued functions.  Is there any possibility of the JDBC driver getting an interface for these in the same vein as Function or Function.Aggregate.

A table-valued function is used like a function but is implemented by a virtual table.
So the driver must support at least read-only virtual table (sqlite3_module, sqlite3_vtab and sqlite3_vtab_cursor).

Thanks!
A virtual table would be sufficient for my purposes as well. It seems like the jdbc driver can make use of vtables, but only those defined by an extension.

Indeed, the driver can load/use C extension containing virtual table implementation(s) but you can't implement virtual table in Java.
If you want to use an existing table-valued function, you can already do that with:
http://sqlite.org/lang_corefunc.html#load_extension
SELECT load_extension('path to the dll');
SELECT value FROM table_valued_func();

Agreed this would be pretty useful. I have a case where a query needs to extract the bounds of a polygon, minx,miny,maxx,maxy, for r-tree insertion, and it's round tripping 4 times back to Java to call 4 separate functions, where one round trip would be sufficient... but to do that, I understand table valued functions would be required.


Unable to login: -1501 error code

I'd love to get this to work as I'm trying to setup a halloween prop, but every time I try the login method fails with:
Exception: Error Code: -1501, Invalid Request or Credentials
My P100 is quite new, with firmware 1.2.10 which I think may be newer than most people are successfully running with.  This may be the problem.  I've tried applying the cookie parsing PR which made no difference.
I am quite sure that my credentials are entered correctly.
Just wondering if anyone has any suggestions, or has come across this issue (perhaps with the later firmware versions).
Thanks
Chris

Hi, This should be fixed in the latest version (0.1.0) :)


Question: How can I remove cocktail from collection?

Hi 👋🏻
I added some coctails to one of my collections. But now I want to remove one of them from collection. But I can't find how to do it correctly from WebUI.
Can you help me?

When you click on a cocktail you will have a list of collections that it belongs to and some additional options, on the right side under the "Ingredient spotlight"

@karlomikus  Thanks a lot


some issue in award in setting page

Describe the bug

images -> image in form and cropper
image is not showing in cropper
when we clear name it clears all the data
Screenshots

https://www.berrycast.com/conversations/e44a4d08-275c-5f49-b21b-533742a484a8


Use real Vertx server for Cucumber tests

The Cucumber tests should verify that both the client and server are working as expected. At the moment, they're using a stub server implementation. Since I want to add a test for checking the behaviour of out-of-date versions of gvm and the implementation of that feature requires code changes on the server, we really need to be running against the server code.
I guess the issue here is the dependency on MongoDB, but I wonder whether it's possible to run the server app with a dummy mongo-persister busmod that we can initialise with duff data. Or we can have the Cuke tests require a running MongoDB instance, but that's probably a non-starter.

We have since cut over the acceptance tests to use wiremock instead. This was done because acceptance tests should test the behaviour of the CLI, not be an end-to-end suite.
Having said that, I plan to write some end-to-end multi-platform smoke tests using spock soon.


FreeOTP migration script

Hello,
congratulations on such a nice OTP app. It has everything I want in an app, so I migrated my codes from FreeOTP to andOTP. I wrote a short script to do this.
To run the script, get the tokens.xml file from the FreeOTP directory on the phone, run the script with the filename as a parameter (./freeotp_migrate.py tokens.xml), and the script will output an andOTP-compatible plaintext JSON file. Just restore that and you're done.
Here's the script, released under the MIT license:
#!/usr/bin/env python3

import base64
import json
import sys
import xml.etree.ElementTree

def read_tokens(filename):
    e = xml.etree.ElementTree.parse(filename).getroot()
    for item in e.findall("string"):
        if item.get("name") == "tokenOrder":
            continue
        token = json.loads(item.text)
        token["secret"] = base64.b32encode(
            bytes(x & 0xff for x in token["secret"])
        ).decode("utf8")

        issuer = token.get("issuerAlt") or \
            token.get("issuerExt") or \
            token.get("issuerInt")
        label = token.get("label") or token.get("labelAlt")

        if label and issuer:
            token_label = "%s - %s" % (issuer, label)
        else:
            token_label = label or issuer

        yield {
            "algorithm": token["algo"],
            "secret": token["secret"],
            "digits": token["digits"],
            "type": token["type"],
            "period": token["period"],
            "label": token_label,
        }

def main():
    if len(sys.argv) != 2:
        print("Usage: ./freeotp_migrate.py <filename>")
        sys.exit(1)

    # Dump the tokens.
    output = json.dumps(
        list(read_tokens(sys.argv[1])),
        sort_keys=True,
        indent=2,
        separators=(',', ': ')
    )

    print(output)

if __name__ == "__main__":
    main()

If you want, I can put it on a page on my site and you can link to it, for convenience.

@skorokithakis Yes, it would be nice if you could upload the script somewhere so I can link to it easily.
If you do I will put it on the Migration page in the wiki.

Just adding the command for those that need to know the location of tokens.xml. I was searching for a while:
adb pull /data/data/org.fedorahosted.freeotp/shared_prefs/tokens.xml tokens.xml
Thanks for creating this great script!

I have posted the script (and a writeup with instructions) here:
https://www.stavros.io/tips/migrate-freeotp-to-andotp/
Thanks!

Thank you! I added it to the wiki 👍