Don't Merge! PR real time test for source branch full build

Docs Build status updates of commit 2eee624:
:clock10: Pending: waiting for processors (28 builds ahead of you)
:warning: Docs Build is busy, currently there are 28 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit 2eee624:
:clock10: Full build: the average full build time is 5 min(s) 48 sec(s), this is based on the last 30 build(s) for this repository.

Docs Build status updates of commit 2eee624:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Missing toolchain when building for aarch64-unknown-linux-gnu

When building for aarch64-unknown-linux-gnu on github action, the job fails with because the toolchain is not installed: https://github.com/tursodatabase/libsql/actions/runs/8897008194
I'm not sure if this is a cargo dist issue, but it's not clear to me from the workflow file how rust is installed on the runner, so I don't know how to fix the toolchain issue

nvm, bumping cargo dist solved it :) thanks


Update build

Update to Ruby 2.7.x.
Bump Gemfile vendor locking.

Signed-off-by: Ben Kochie<EMAIL_ADDRESS>

That probably does not pass docker build.

https://github.com/nanoc/nanoc/issues/1501 -> I'd recommend to stay at 2.6

Ok, I rolled it back to 2.6.x

Ruby 2.7 is now usable as of Nanoc 4.11.19. The underlying issue (https://github.com/nanoc/nanoc/issues/1501) is resolved.


low severity npm audit vulnerability in devkit/build-angular due to webpack-dev-server  > yargs depedency

🐞 Bug report
Is this a regression?
No
Description
npm audit reports a low severity "Prototype Pollution" vulnerability in yargs-parser, which is a dependency of @angular-devkit/build-angular via webpack-dev-server. This has been fixed in webpack-dev-server 3.11.0, so the dependency just needs to be updated to that version.
🔥 Exception or Error

                       === npm audit security report ===

                                 Manual Review
             Some vulnerabilities require your attention to resolve

          Visit https://go.npm.me/audit-guide for additional guidance

  Low             Prototype Pollution

  Package         yargs-parser

  Patched in      >=13.1.2 <14.0.0 || >=15.0.1 <16.0.0 || >=18.1.2

  Dependency of   @angular-devkit/build-angular [dev]

  Path            @angular-devkit/build-angular > webpack-dev-server > yargs >
                  yargs-parser

  More info       https://npmjs.com/advisories/1500

🌍 Your Environment

Angular CLI: 8.3.26
Node: 12.13.0
OS: win32 x64
Angular: 8.2.14
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router

Package                                    Version
--------------------------------------------------------------------
@angular-devkit/architect                  0.803.26
@angular-devkit/build-angular              0.803.26
@angular-devkit/build-optimizer            0.803.26
@angular-devkit/build-webpack              0.803.26
@angular-devkit/core                       8.3.26
@angular-devkit/schematics                 8.3.26
@angular/cdk                               8.2.3
@angular/cli                               8.3.26
@angular/material                          8.2.3
@angular/material-moment-adapter           8.2.3
@ngtools/webpack                           8.3.26
@nguniversal/module-map-ngfactory-loader   v8.2.6
@schematics/angular                        8.3.26
@schematics/update                         0.803.26
rxjs                                       6.5.5
typescript                                 3.5.3
webpack                                    4.39.2

Duplicate of https://github.com/angular/angular-cli/issues/17642


A way to save and load the output of upstream model for speedup

Feature request
Is your feature request related to a problem? Please describe.
I am using this toolkit to run several experiments where the upstream is frozen, by modifying the downstream each time. I find that the training is quite slow for some upstream models. In terms of computational complexity, the downstream models are significantly lighter than the upstream. If there is a way to save and load the upstream outputs for each file, it will make the training significantly faster.
Describe the solution you'd like
Adding this feature is usually straightforward if there is only one kind of feature to store (For example, the last hidden layer). We just need to store the output of the featurizer in the first epoch to a cache path for each audio file in the dataset, and load them in the subsequent epochs, without having to compute them on the fly. But when we want the featurizer to be trainable (when we are using the weighted sum approach), we will need to store all the hidden layers of the upstream, and handling all that is quite complicated, because the output of upstream is a dictionary, whose size varies with batch size, and entries vary with the upstream model. This can be handled in the data loader itself, or separately inside the training loop.
I honestly feel it is worth spending some time adding this feature, as it will save a lot of compute resources and time.

Hi @shreyas2206 ,
Sure! I totally agree with you. This would be an extremely useful feature. However, I might not have enough time to add this function and test the backward compatibility for all the tasks. Any volunteers interested in implementing this function are highly welcome. We care about acknowledging contributions and will give credit for major features in the main README log.
Many thanks for the proposal!
Sincerely,
Leo

Hi Shreyas,
Cool! This sounds super interesting.
I believe this requires a substantial amount of SSD storage to load all the hidden states efficiently, but I think that the SSD requirement may not be a significant bottleneck nowadays (it was back in 2021 in our lab).
I would be happy to help implement this feature.
If you encounter any problems, please don't hesitate to post here so we can discuss.
Thanks!
Sincerely,
Leo


Fix unix on windows install failure

For the 'unix on windows' systems, gvm install was running
src/make.bash instead of src/make.bat. This fixes this
issue

please resolve conflicts

Conflict resolved


AWS credentials cache key needs to include the region

Report
Hello, I have two deployments using ScaledObjects based on SQS queues in different regions (say, eu-central-1 and us-east-1) and I want to authenticate to AWS using pod identity. The first ScaledObject authenticates correctly (one running on eu-central-1), and then the AWS config (with region included) is cached in the config cache. The second ScaledObject fails to start, because the operator tries to connect to a queue in another region (us-east-1), but the cached config includes the region from the first queue (eu-central-1).
If I understand the code correctly, the getCacheKey function here: https://github.com/kedacore/keda/blob/85d4dca17f9e2e58bdc91f046e6dbe8e6235e78f/pkg/scalers/aws/aws_config_cache.go#L71 needs to include region in the returned string so that the configs are cached per region.
Expected Behavior
Both ScaledObjects report as Ready
Actual Behavior
Only the first ScaledObject is ready, the second one never authenticates successfully.
Steps to Reproduce the Problem
1.Create two queues in different region
2.Create ScaledObjects for them using pod identity as auth mechanism
Logs from KEDA operator
{"level":"error","ts":"2024-09-03T14:22:34Z","logger":"scale_handler","msg":"error getting metric for trigger","scaledObject.Namespace":"**REDACTED**","scaledObject.Name":"**REDACTED**","trigger":"awsSqsQueueScaler","error":"operation error SQS: GetQueueAttributes, https response error StatusCode: 400, RequestID: d511bcd7-6b16-5a65-bb0d-e38676fca9a3, AWS.SimpleQueueService.NonExistentQueue: The specified queue does not exist or you do not have access to it.","stacktrace":"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics\n\t/workspace/pkg/scaling/scale_handler.go:553\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics\n\t/workspace/pkg/metricsservice/server.go:48\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler.func1\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:106\ngithub.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).Start.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func6\n\t/workspace/vendor/github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/server.go:22\ngithub.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler\n\t/workspace/pkg/metricsservice/api/metrics_grpc.pb.go:108\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/workspace/vendor/google.golang.org/grpc/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/workspace/vendor/google.golang.org/grpc/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/workspace/vendor/google.golang.org/grpc/server.go:1019"}

KEDA Version
2.15.0
Kubernetes Version
1.30
Platform
Amazon Web Services
Scaler Details
AWS SQS
Anything else?
No response

Hello,
Interesting and you're probably right. WDYT @ThaSami ?

I believe I am seeing this issue as well. Definitely paying attention to that PR.

@JorTurFer any chance for a review and, hopefully, merge of the attached PR?

@JorTurFer bumping for review on the PR

Can this be re-opened until the next release occurs?


[User Request] Declaratively invoke setters (and possibly functions) simultaneously

As a maintainer of a repository of kpt package, I'd like to ensure that all package setters have been invoked (ideally pre-commit) so that the underlying manifests are rendered with the desired configuration in version control. This would at least help prevent inconsistencies that may arise when people directly modify the values in a Kptfile, while forgetting to propagate those changes by making the corresponding calls to kpt cfg set
As far as I can tell, it is currently only possible to invoke setters imperatively. It seems the use case I've outlined above would be easier if it were possible to invoke setters declaratively, similar to how we can invoke functions declaratively. Something akin to:
# effectively kpt cfg set . ${SETTER} for all of the package's setters
kpt cfg set .

More broadly, it would be nice to have a command to "ensure" that a Kptfile and its settings and functions are reflected in the underlying manifests, effectively: kpt cfg set . && kpt fn run .. Then it'd be simple to run something like find . -name Kptfile -exec kpt cfg ensure {} from the root of the repository.
Thoughts? I'd appreciate any additional ideas/approaches to address this

kpt cfg set . is one of our short term goals and thanks for bringing this up.
Ensure is an interesting idea but combining set and fn should be something which should be thought about in terms of feasibility. Question arises, why not include pkg and live? so that ensure gets/updates, sets, validates and applies package. Can you elaborate on your end to end use case to give us more insights?

#1107 is related.

@mikebz This is more related to declarative setters which is being tracked here. https://github.com/GoogleContainerTools/kpt/issues/1051
Marking this as duplicate.


The buttons in the modal don't work

Clicking the "X" in the top right corner works fine, but clicking either the close or the save button doesn't do anything.

Yeah this was a very basic sample to give a starting point for developers to use.  I will get it updated this weekend and make sure all of the samples are at least demonstrating points we want to show off.  thanks!

This should be fixed now actually since the modal has been extracted.


Replace annotations:2.0.1 with jsr305:3.0.0.

The latter is new and smaller.
Alternatively, consider marking the existing dependency provided or optional (that's safe).

@ronmamo, any thoughts on this?
(Just went over my list of open PRs and found this one. I rebased the branch.)

The change looks good :+1:

cool


Don't clearfix if flexbox is enabled. Fixes #18849

See #18849.
Added:
@if (not $enable-flex) {
  @include clearfix;
}

to .card-header, .card-block and .card-footer

modal has the same problem, would you mind adding this fix to that as well?


[transpileDeclaration API] What to do about "has or is using private name" diagnostics

🔍 Search Terms
isolated declarations transpile transpiledeclaration diagnostic private name
✅ Viability Checklist

[X] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[X] This wouldn't change the runtime behavior of existing JavaScript code
[X] This could be implemented without emitting different JS based on the types of the expressions
[X] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[X] This isn't a request to add a new utility type: https://github.com/microsoft/TypeScript/wiki/No-New-Utility-Types
[X] This feature would agree with the rest of our Design Goals: https://github.com/Microsoft/TypeScript/wiki/TypeScript-Design-Goals

⭐ Suggestion
Either make ts.transpileDeclaration not report "has or is using private name" diagnostics, or provide a method to easily check whether a diagnostic is related to private names.
📃 Motivating Example
Usage of ts.transpileDeclaration for any project with isolatedDeclarations
💻 Use Cases
ts.transpileDeclaration will report usage of global types such as Pick, Omit, Promise, or project-specific ones, as using private names. These are not really actionable, so I wonder if these diagnostics should just not be reported at all. What I've been doing is to just filter them out. The structured way of doing that is checking the diagnostic code, but there are so many codes to check that I decided to take a more unstructured approach by just checking diagnostic.messageText.includes('private name'). So at the very least, if these diagnostics will continue to be reported, it would be nice if there was some boolean method that says whether or not a given diagnostic is related to private names.

I think this is just an unfortunate conflation of behaviors in our normal declaration emit errors. Emitting "X is using private name Y" when Y actually just doesn't resolve at all seems wrong - it's not that it's there but inaccessible from here, it's that it's not there at all. In all of these cases, the checker should be reporting a Name Y cannot be found error or the like (if its' run), so I don't think declaration emit should be doubling up on that with its' own confusing error. We probably want to trim "X is using private name Y" errors to just resolvable privates and leave the unresolved names errors to the checker pass.


Improve Composer autoloader including.

If we want to require Slim Console globally, require __DIR__ . '/../vendor/autoload.php'; won't work, because in this case the Composer autoloader has a different path.

@l0gicgate  in my practice one of these files always existed. I can add additional check, if you consider it necessary.

I mean, files always existed, if composer install was executed :smile:

I just want to get this right. Are there any instances where autoload.php is not present but our bin file could exist? That else block makes the assumption that it always would.

Are there any instances where autoload.php is not present but our bin file could exist?

I can’t imagine a situation where this can happen, except when the vendor directory has been deleted.

That else block makes the assumption that it always would

As far as I remember, there are only those two paths where autoload.php can be placed.

Alright we can always add onto this condition later if an issue arises.


feat(bridge-ui-v2): navigation support to swap-ui from bridge-ui

This PR adds a navigation bar for swap from the Bridge UI, enabling seamless navigation between two apps. Notably, while a redirect from the Swap UI to the Bridge UI already exists, this update addresses the lack of a reciprocal link and ensures improved user experience and navigation consistency.
reference on swap-ui

reference on bridge-ui

@arularmstrong Thanks, very good idea.  Please fix the linting issues then we can merge it.
Run pnpm svelte:check && pnpm lint:fix to see all the issues

Sure, fixing the lint issues.

make sure to add PUBLIC_DEFAULT_SWAP_URL=https://swap.katla.taiko.xyz/ over the env.

@arularmstrong approved. But please note, that the changes won't be visible before our next bridge release.


Add tickmarks in Fig 28 , 29 (and prob others) like in Figure 34

Fig 28 currently forces 2018 to be added on, but it overlaps with 2020.
Much clearer (and more standard) to have tickmarks like in Figure 34. I don't mind doing it (at some point). Tickmarks make it easy enough to check what the final plotted year corresponds to.

I'm good with the style of individual years getting unlabeled tick marks as proposed if somebody has time and interest to go through and make that change.
However, just reporting here that for the assessment histories plot, there's a switch to specifically check for whether the end year is too close to the next multiple of 10, and leave off the label if it is:
  # add x-axis mark for final year (without label if too close to nearest decade mark)
  axis(1, at=max(years), label=ifelse(max(years)%%10 < 2, "", max(years)))
``

Good to know - maybe we just use that elsewhere to make the final year show up. Thanks.

Ironically, I posted that comment about the conditional label on the tick mark while editing the function before actually looking at the updated graph. The ifelse(max(years)%%10 < 2, "", max(years)) only works if the end year is just AFTER the interval of 10, not just before. I fixed it to go both directions, but found that adding an unlabeled tick mark at 2018 just looks strange.
Instead, I think it looks best to just add extra short ticks for all years (adding the extra ticks up to 2018 looks strange because of a gap in 2019 before the bigger mark at 2020)
The command I used was
axis(1, at=1950:2030, labels=rep("", length(1950:2030)), tcl = -0.3)

Completed in commit b07f52690836065b6aac4cc7bb76729b5fbb496e.
Figures 30 and 31. I put the Unfished equilibrium text a little below the axis tick labels and added the year (1964) to the text. Let me know if anyone doesn't like it and we can remove that or do something else.

The unfished equil is a little unclear (and it's a bit confusing to mention 1964).
I have a feeling it's tricky to force the x-axis to start ticks at 1966, yes? Then the unfished would be more clearly separate. Hmmm... don't want this to take time (a quick back-up option is to revert the commit (I think Chris said he did it all in one on purpose), and just force the xlim for now to stop at 2019 (xaxs="i"). Sorry for not checking this earlier.

Looking at Chris's commit b07f52690836065b6aac4cc7bb76729b5fbb496e, it's done in a clear and easy to understand way, so it should be quick to edit to start ticks in 1966. I can do that this evening.

Change completed in commit 8e2a62b. It requires source(all.r) because model-setup.r now contains controls over the big and little ticks. @andrew-edwards if this isn't what you were looking for, feel free to modify or suggest changes as you wish.
# tick marks for time series plot
big.ticks <- seq(1970, end.yr+4, 5)
little.ticks <- start.yr:max(big.ticks)

Perfect, thanks!


infratribe defies to be edited

I'm trying to synonymize infratribe Scaphurae (genus group in old OSF) with tribe Scaphurini (stat. nov. - published June 11th), but Edit taxon name is sort of inert. Not possible to establish objective synonym relationship nor add missing type genus Scaphura.
Unintelligible error:

The first isn't true and it doesn't let the second be resolved:

What does this mean?

This seems to be the correct relationship for Scaphurae Westwood, 1838 with Scaphurini:

... but the error keeps appearing ...

Scaphurini, which I guess is the subject-taxon-name, is valid.

please use "family group name form" relationship. TW uses synonymy in strinct ICZN sense.

How do I then add citations for the original form Scaphurae (used in altogether four sources)? Only for Subsequent name forms can be added citations.

Citation on protonym is always added to the original form. Citation on subsequent forms will be added to those.

Thank you, I think I understood how it works. Now it looks like this:

When I now change the infratribes ending in -ae to tribes ending in -ini, add the original form with -ae below, and the subsequent form with -ini, the display looks strange:

Aegimiini as synonym of itself. Should I have added Aegimiini as new name and then establish the relationship of Aegimiae to Aegimiini separately? But in the Scaphurini example above which I reconstructed from scratch occurs the same.

Look at the timeline, everything looks correct:

The classification tree shows you that there are two protonyms.

Thank you.
The paper published today proposes a few infratribes ending in -iti (to accommadate groups below subtribes). TW doesn't like this.

ICZN does not regulate the endings of infratribes. I do not remember where the initial idea of -ad came from. I think, I will just remove the restriction.


[Enhancement]: Add back the setting for DoNotRedirect in the tabsettings

Is there an existing issue for this?

[X] I have searched the existing issues

Description of problem
My issue is described in this forum post:
https://dnncommunity.org/forums/using-dnn/optimization/issue-with-friendly-url/#12705
My url's look something like:
http://www.website.com/page/subpage/parameter/123/nice-friendly-name
And this worked on some DNN instances, and not on some other instances.
On some instances of DNN whenever I went to this url, I got a 301 redirect which removed the last part of the url, so I ended up on:
http://www.website.com/page/subpage/parameter/123
For SEO and nice-and-friendly url's in communication I think this is a BAD thing...
I noticed there is an entry in the tabsettings which influences this behaviour, but was at some time removed (I think) from the UI/tabsettings page.
So whenever I added an entry in the tabsettings for this tabid, with the SettingName 'DoNotRedirect' and the value of 'True', DNN would not perform the 301 redirect, and keep my nice and friendly url.
So I think it would be a good idea to add this option back into the page/tab settings...
Description of solution
Add the option back into the tab/page settings for the 'DoNotRedirect' setting.
Description of alternatives considered

Anything else?

Do you be plan to contribute code for this enhancement?

[ ] Yes

Would you be interested in sponsoring this enhancement?

[ ] Yes

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Also: https://dnncommunity.org/forums/using-dnn/extensions/redirects-acting-as-rewrites/


Introduce terminalRows annotation

Introduces terminalRows to select the number of rows for this cell.
In a future PR, we can consider syncing with the existing terminal, as well as edge validation cases.

I don't think itbworked

I don't think itbworked

Currently in the process of releasing this feature. It will work in v1.7.3, which should hit distribution in the next hour, if that's what you mean. Did I miss understand?


establish working definition of group

These will be the working definitions and notations that we will use for the entirety of the project. Chapter 1 should build up to these definitions, possibly using definitions of their own, while chapters 2 and onward should use these definitions in future theorems.
The namespace in which these definitions are located is meant to be temporary, as it will be moved into a chapter once the structure of the project becomes more concrete.
Once we agree, let's merge this PR so that it's only one commit, sqaushing it if necessary.

Now depends on #4.


Feature request: IP binding

I'd love to see support for Nginx servers binding to specific IPs for servers that have more than one IP. I know this is super simple to do after the general configuration files have been generated, but it may prove useful for people that may not be sure how to do it.

Thank you for the feedback, @imnjs!


Add support for nunjuncks

Hey,
Would it be possible to add support for njk / nunjuncks files?
https://mozilla.github.io/nunjucks/
Zeno.

I guess, the same icon that's in use for mustache should work.


apache 2.5 support not working

Hi,
i using apache version:
Server version: Apache/2.5.0-dev (Unix)
Server built:   Feb 16 2016 12:14:24
when i compile mod_pagespeed and link module _ap24 server not start.
because
httpd: Syntax error on line 504 of /usr/local/apache2/conf/httpd.conf: Syntax error on line 14 of /usr/local/apache2/conf/pagespeed.conf: API module structure 'pagespeed_module' in file /usr/local/apache2/modules/mod_pagespeed_ap24.so is garbled - expected signature 41503235 but saw 41503234 - perhaps this is not an Apache module DSO, or was compiled for a different Apache version?
httpd: abnormal exit 1

Hi...  As the message says, modules built for Apache 2.4 won't work on 2.5; you'd need to build from source to be able to work with it.

I built modules from source (apache2 and mod_pagespeed).
make AR.host=pwd/build/wrappers/ar.sh AR.target=pwd/build/wrappers/ar.sh BUILDTYPE=Release
NO_SUDO=1 APXS_BIN=/usr/local/apache2/bin/apxs ./install_apxs.sh

Weird. Passing in V=1 to make might give some idea --- are the compilation commands getting -I lines matching where the 2.5 headers are?
(#define MODULE_MAGIC_COOKIE 0x41503234UL /* "AP24" */)


Particles re-rendering on state change of parent component

I currently have a wrapping particles container rendered in a separate component as the parent component I am trying to render the particles container in. When the parent component state changes, the particles component re-renders on state change of the parent. I'm not passing props to the child particles container so why would it re-render? My particles.json file is in my assets folder and is not being updated. Any suggestions?
import settings from '../../assets/particles.json';

const ParticlesContainer = () => (
  <Container>
    <Particles
      params={settings}
    />
  </Container>
);```

This configuration should be normal?
I mean, separating Particle Component to another component that doesn't change its state seems like a workaroud, not a fix at all.
Is this the final solution?

How to get it working though?
I created a ParticleComponent which doesn't take in any props and doesn't have any state.
I want to use this in a Page which has its own state, but whenever the state changes, the particles jump.
The code is something like:
// Page1
render() {
    return (
        <React.Fragment>
            <ParticleContainer/>

{this.state.appName}

                // rest of the JSX which is dependent on state

        </React.Fragment>
    );
}

I had this issue with beta version 2.4.0. On certain state changes at the parent level, the particles would re-render and flicker rapidly. I had also tried breaking it out into a separate component along with other tweaks.
Upgrading to version 2.7.0 resolved my issue.


Improve Snapshot type so that it respects builtin objects

Summary
This PR improves Snapshot type so that it does not affect builtin object, like Date.
Examples for old and new behaviours can be found in TS playground: for TS 3.5.1 and for TS 4.9.5. As you can see from the OldSnapshot example, it iterates through every member of Date, Map and Set object and converts these objects to readonly POJOs for some reason.
The solution might not be perfect though, so I would suggest adding some tests for typings (or maybe you already have and I just won't be able to find any, so please let me know).
Also, I wouldn't be able to run tests, because this project is hardly relying on Yarn, so will see if the tests are passing.
And lastly, I would like to ask if you can expose both Options (for useSnapshot hook) and Snapshot<T> types, so that people might implement extensions upon this hook and rely on public API.
Check List

[x] yarn run prettier for formatting code and docs

Thanks for your suggestions!

so I would suggest adding some tests for typings

Would you be able to help on it?

this project is hardly relying on Yarn

That's true, but I think you can run normal tests with npm test. (CI tests are really complicated.)

if you can expose both Options (for useSnapshot hook) and Snapshot types,

As you might guess, the first one can be extracted fairly easily, and the second one is intentional.

people might implement extensions upon this hook and rely on public API.

If they rely on internal types, I'd suggest to pin the valtio version and use it. We may really change them in the future without notice.

Would you be able to help on it?

Yup, I think I can write bunch of tests for Snapshot type using something like tsd. But I am have no idea how to get this done for patches for TS 3.4

That's true, but I think you can run normal tests with npm test. (CI tests are really complicated.)

Nope, it doesn't work and I have to install Yarn. And maybe I will, it's just that I have to do this for such small thing. Not a big problem, I think. But having scripts running using npm might be more convenient, as it is the default (and I personally could've run tests via pnpm).

As you might guess, the first one can be extracted fairly easily, and the second one is intentional.

And so I did, yes, but it would be just much more handful to have useSnapshot Options to be part of the public API.

If they rely on internal types, I'd suggest to pin the valtio version and use it. We may really change them in the future without notice.

Alright then :)

Would you be able to help on it?

Yup, I think I can write bunch of tests for Snapshot type using something like tsd. But I am have no idea how to get this done for patches for TS 3.4

We can simply drop such tests for very old TS. Please make it a new test file.

That's true, but I think you can run normal tests with npm test. (CI tests are really complicated.)

Nope, it doesn't work and I have to install Yarn. And maybe I will, it's just that I have to do this for such small thing. Not a big problem, I think. But having scripts running using npm might be more convenient, as it is the default (and I personally could've run tests via pnpm).

Oops, my bad. My recent change requires yarn for test.
Yeah, we may consider improving those build/test scripts in the future and I hope to make it work with npm too.

Added some tests for Snapshot type, ended up using ts-expect. They will be run through regular tsc --noEmit command.

Feels like we never reach an ideal solution. false positives are better than false negatives.

Can't agree with you more.
So I've added SnapshotIgonre union type with the list of objects that Snapshot will not turn into readonly. Please let me know if I've missed something.

Out of curiosity: Why GitHub does not let me to request review from both of you at the same time? x)

Wellp, looks like I need yarn after all :)

Wait, which yarn version do I need? Is it 1.x?

yes, it's classic yarn.

Okay, I have no idea why tests for old ts versions are failing. It doesn't seem like I've changed anything significant since yesterday, and yesterday's builds were passing the tests.
Also, please add contribution guidelines of some sort, just to clarify how to develop for valtio.

Feels like we never reach an ideal solution.

Fwiw @dai-shi if the "gah classes" gives you too much grief, I think a potential compromise is requiring users to opt-in via a base class, like:
// in valtio
abstract class BaseStore {
  [hiddenSymbol]: ...
}

// in apps, opt-in to "treat my class as a store"
class User extends BaseStore {}

I know this sort of "forced base class" is a trigger for OO purists, but it would let both the runtime / canProxy do an target instanceof BaseStore and mapped type / Snapshot<T> do an target extends BaseStore and know 100% for sure the user wants their data to be proxified/snapshotted/etc.
I.e. I believe it'd let us remove the "disallow list" of WeakMap+WeakSet+other special things we think of, and move to just "is it a basestore (or the vanilla object literal)"?
I dunno, personally I'd like to avoid that, and try to keep just "vanilla classes" working out-of-the-box. But just mentioning it as a potential middle ground for someday.

Fixed Snapshot type tests for TS 3.7. It appears that import type was introduced in 3.8.

Are you hesitant renaming the new file to tests/typings/snapshot.test.ts?

No, but this will require configuration for jest, I think. It should be ignored by it.

Didn't catch this one. For some reason VSCode decided not to show ESLint errors for me :D

No, but this will require configuration for jest, I think. It should be ignored by it.

Oh, I wasn't aware that. Which part is bad? We have tests/snapshot.test.ts, so the typings subdir could be something, but not that one...

Which part is bad?

The part where jest will likely fail trying to run through this file :)
These test files should not be executed in runtime. They only meant for tsc typeschecks.
But technically we could just ignore the whole typings directory for jest, I guess. Doesn't really matter where these tests placed, or how they named. They're just regular TypeScript files.
I can add configuration for jest and rename the file as you ask. If you insist :)

The part where jest will likely fail trying to run through this file :)

Ah, I thought the other way around.
Well, we run those type only tests with jest, so, it should be fine. Does it cause CI errors?

Well, we run those type only tests with jest, so, it should be fine. Does it cause CI errors?

If jest will ignore test file without any test in it, then it should be fine. I'm not familiar with jest that much, but, for instance, AVA will fail in that case. So I just assumed it might have same behaviour, because it seem reasonable.

Google says it will fail. So in that case we need to either ignore these files (as I suggested - the whole tests/typings directory), or add a noop test, or add todo test.

This is how it's done in jotai.
https://github.com/pmndrs/jotai/blob/main/tests/vanilla/types.test.tsx
So, it's a noop test?

It does seem like it. I don't see any use of jest's expect function. But they look like they're regular jest tests. I can rewrite these test like this, if you want.

Not sure though why do use have Component at the end of each test.

I think some of eslint rules complain otherwise.

I can rewrite these tests like this, if you want.

Yeah, please try it for now. We'll revisit later with other repos. Or maybe with #646.
I may consider including typing tests in tests/snapshot.test.ts for now, as it works with ts 3.7.5. That makes all test files flat.

I may consider including typing tests in tests/snapshot.test.ts for now, as it works with ts 3.7.5. That makes all test files flat.

I've merged typings tests file into tests/snapshot.test.ts.


⚠ Bofrost.be has degraded performance

In 8b7f6c0, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1322 ms

Resolved: Bofrost.be performance has improved in c25a4be.


[improve][offload]PIP-211: Introduce Offload threshold, P1

PIP: https://github.com/apache/pulsar/issues/18004
Motivation
See https://github.com/apache/pulsar/issues/18004
Modifications

Move TimeWindow.java, WindowWrap.java from pulsar-broker to pulsar-common
Add OffloadLedgerHandle.java
Pass OffloadLedgerHandle's instance to LedgerOffloader instead of LedgerHandle.

Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[x] doc-required
[ ] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

Origin PR: https://github.com/apache/pulsar/pull/17485

I think that AsyncTokenBucket should be used for rate limiting instead of implementing a separate solution. It was introduced in PIP-322.

@lhotari
TimeWindow is not introduced by this PR, it exists a long time. Just reuse it here. If you think AsyncTokenBucket is better, I can fix this.

@lhotari I will remove AsyncTokenBucket related class from pulsar-broker to pulsar-common, is it reasonable?

Codecov Report
Attention: Patch coverage is 83.05085% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 73.71%. Comparing base (bbc6224) to head (ada5b82).
Report is 107 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22385      +/-   ##
============================================
+ Coverage     73.57%   73.71%   +0.13%
+ Complexity    32624    32404     -220
============================================
  Files          1877     1886       +9
  Lines        139502   139926     +424
  Branches      15299    15326      +27
============================================
+ Hits         102638   103141     +503
+ Misses        28908    28810      -98
- Partials       7956     7975      +19

Flag
Coverage Δ

inttests
27.03% <11.86%> (+2.45%)
:arrow_up:

systests
24.48% <52.54%> (+0.16%)
:arrow_up:

unittests
72.98% <79.66%> (+0.14%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

...apache/bookkeeper/mledger/ManagedLedgerConfig.java
96.38% <100.00%> (+0.08%)
:arrow_up:

...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java
80.95% <100.00%> (+0.29%)
:arrow_up:

...org/apache/pulsar/broker/ServiceConfiguration.java
99.39% <100.00%> (+<0.01%)
:arrow_up:

...n/java/org/apache/pulsar/broker/PulsarService.java
82.62% <ø> (+0.25%)
:arrow_up:

...ker/resourcegroup/ResourceGroupPublishLimiter.java
86.95% <ø> (ø)

...rg/apache/pulsar/broker/service/BrokerService.java
80.96% <100.00%> (+0.18%)
:arrow_up:

.../pulsar/broker/service/PublishRateLimiterImpl.java
86.15% <ø> (ø)

...broker/service/persistent/DispatchRateLimiter.java
78.63% <ø> (ø)

...roker/service/persistent/SubscribeRateLimiter.java
54.54% <ø> (ø)

...ava/org/apache/pulsar/broker/stats/TimeWindow.java
66.66% <ø> (ø)

... and 8 more

... and 181 files with indirect coverage changes


irc: fix timer memleak

This is a gotcha called out in the time.After docs. time.After will
leak the underlying channel if nothing ever receives on it.

Ah, this makes sense and could be the cause of some mem leaks I've been seeing. Tried it out and it looks fine, have committed it as 2e8a98a925b322ad81f716bcb46c185e07a8dc93 to master. I'm gonna do some more testing around memory leaks and all (StackImpact is super useful for this)


[Gally]: master <- dev

Automatically created by Git-Ally

:tada: This PR is included in version 3.6.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[Global] Content Import

it would be great to have a way to get from sitemap.xml or some other list of URLs that encompass something like the results from a google search for site:theblog.adobe.com and turn those URLs into a set of .md files...
i think in this particular case a selector of section#content would probably be used to get to the content and then it looks a straight forward html to md conversion.
from there i think it would be great to find a batch way to convert folder structure into gdocs or .docx

The import is straight forward.
This blog post https://theblog.adobe.com/new-data-governance-capabilities-in-adobe-experience-platform-help-brands-manage-data-better/
would easily give this markdown:
https://github.com/kptdobe/helix-htmlimporter/blob/master/content/output/b1.md
To not overload the md, there is some processing to do on the html (with JSOM and jquery it is super simple!): https://github.com/kptdobe/helix-htmlimporter/blob/master/index.js#L18-L31
For now I do a random transformation to isolate the author and the topics at the end in own sections, I know it will not be the final output but this illustrates how easy it is.
We'll need to separately extract the author info, they are most likely coming from somewhere else and aggregated server side. Will have to do something similar in a special section at the end.
@rofe Once we know the required structure for a blog md file, I will adjust the processing.
All of this assuming the blog posts share the same dom. Which is already not the case for old ones but we can exclude those I assume.

Great! Looking at #3 and #4 I think we should strip the avatar and links from author and topics and keep them plain text. Link and avatar to the author's page would be added client-sided, as well as the links to products and topics.

Yes. While importing blog posts, I can also create the author and topic pages populated with content I find in each page (first one wins).
Working on it now.

Worked on the "real" download: https://github.com/kptdobe/helix-htmlimporter/tree/master/content/output/posts
For the selected live urls, downloads and converts everything necessary (html + imgs).
Now working on extracting author and topics.

I think blog pages now look like the expected final result (+/- some details). See examples in:
https://github.com/kptdobe/helix-htmlimporter/blob/master/content/output/posts/
Still TODO (on Monday):

create author pages during import
create topics pages during import
handle the "products in the page" if needed

Ok, was a good thing to test in onedrive... Multiple issues with storing .md files in onedrive:

helix-pages does only render .docx files - we would need to implement a re-try (check first if .docx available, otherwise try .md.
you can read a .md file stored in onedrive but you cannot edit it nor export it to something else. Pure read-only.

This is blocking us for the import. The only elegant solution would be to create a .docx file instead of a .md.

How about just storing imported (i.e. static/past) posts directly as .md in the repo, and only render "new" posts from onedrive/.docx?

Good point. We can start like this.
I just populated the corresponding author pages. See https://github.com/kptdobe/helix-htmlimporter/tree/master/content/output/authors

regarding older blog posts... here is one i found from 2008.
https://theblog.adobe.com/conversion-success-events/
i think that ideally, no matter if new or old the md result should be the same...

We are lucky, the DOM is almost the same, the importer works as is: https://github.com/kptdobe/helix-htmlimporter/blob/master/content/output/posts/conversion-success-events.md

Ok, was a good thing to test in onedrive... Multiple issues with storing .md files in onedrive:

helix-pages does only render .docx files - we would need to implement a re-try (check first if .docx available, otherwise try .md.

this is relative simple to implement... this can be a feature for word2md

you can read a .md file stored in onedrive but you cannot edit it nor export it to something else. Pure read-only.

well, you can replace it...

This is blocking us for the import. The only elegant solution would be to create a .docx file instead of a .md.

@davidnuescheler wdyt? should be convert the html back to docx?

Having the md files as static read-only documents in Onedrive does not bring much value (even if you could edit then via the onedrive folder on your desktop with your favorite md editor). I import the authors (bio), topics and soon the products, those will be "living" parts.
The HTML to MD convertion is easy and looks like a reverse pipeline (with a pre.js to interact with the DOM parts you may want to remove / patch). I looked at a HAST / MDAST to docx converter but it does not exist.
This would be cool :)

Having the md files as static read-only documents in Onedrive does not bring much value (even if you could edit then via the onedrive folder on your desktop with your favorite md editor).

well, the impoted old posts are not modified anymore. hence could be considered as archived

I import the authors (bio), topics and soon the products, those will be "living" parts.

this could be a one-by-one bases. If an author wants to edit his bio, he can copy-base the md in a word doc. that is probably easier.
the topics are a manually crafted "sitemap". those don't need to by dynamic.

The HTML to MD convertion is easy and looks like a reverse pipeline (with a pre.js to interact with the DOM parts you may want to remove / patch). I looked at a HAST / MDAST to docx converter but it does not exist.

there are a ton of docx generation libraries. eg https://www.npmjs.com/package/docx looks simple enough. writing to mdast2docx should be too difficult.

Yes, that's basically the reverse operation of helix-word2md.
As a tmp step, I committed the current project state into a branch. Here is a test blog entry (still some styling work to be done:) https://alexonedrive--theblog--davidnuescheler.hlx.page/content/posts/four-paintings-that-show-what-you-can-do-in-adobe-fresco.html
Here is the corresponding md file: https://github.com/davidnuescheler/theblog/blob/alexonedrive/content/posts/four-paintings-that-show-what-you-can-do-in-adobe-fresco.md
I added https://github.com/adobe/helix-pages/issues/125 to be able to mix docx and md files together (currently not possible).

adobe/helix-pages#125 is blocking the work here since we cannot have .md and .docx in the same project.

Latest status: md can be imported, dumped into OneDrive / Google Docs and now rendered from there.
Next issue: the images are imported together and must be rendered from OneDrive / Google Docs. See https://github.com/adobe/helix-word2md/issues/109.
Workaround: I changed the importer to B64 encode and inline the images in the md file. This works but makes the md unreadable...

For the demo, we could also just dump the .mds in the GH repo if that helps. We'd just have to find a way to index them.
cc @dominique-pfister

Workaround: I changed the importer to B64 encode and inline the images in the md file. This works but makes the md unreadable...

can we change the importer to upload the images to azure instead?

I should be able to do that. Where do I find the AZURE_BLOB_SAS and AZURE_BLOB_URI params ?

Where do I find the AZURE_BLOB_SAS and AZURE_BLOB_URI params ?

they are already params on the runtime helix-private package....

Well, the importer is for now a local cli. I would need to pass the params manually. I think the only thing I need is to be able to decrypt the secrets in project-helix. If you could add me as an admin there...

@tripodsan solution works like a charm. I can start feeding the legacy folder in OneDrive with existing blog posts as md files.

Legacy rendered from md: https://theblog--davidnuescheler.hlx.page/legacy/posts/creating-adobe-experience-platform-pipeline-with-kafka.html
Missing the products (again...).


Attached Image showing Face even though it doesn't contain Any with more than 90% confidence

Please make sure that this is a solution issue.
System information (Please provide as much relevant information as possible)

Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
MediaPipe version:
Bazel version:
Solution (e.g. FaceMesh, Pose, Holistic):
Programming Language and version ( e.g. C++, Python, Java):

Describe the expected behavior:
Standalone code you may have used to try to get what you need :
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
Other info / Complete Logs :
Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached:

Hi @surapuramakhil ,
Could you please provide steps to reproduce this issue.

FaceDetectionOptions faceDetectionOptions = FaceDetectionOptions.builder()
            .setStaticImageMode(true).setModelSelection(1).setMinDetectionConfidence(.9f).build();

Now send Image as BITMAP to face detector

Hi @surapuramakhil ,
Could you please elaborate this issue to understand better on this.


Issue with k8s.io/docs/tutorials/kubernetes-basics/expose-interactive/

This is a...

[ ] Feature Request
[x] Bug Report

Problem:
The description is confusing at the following places:

We have now a running Service called kubernetes-bootcamp. Here we see that the Service received a unique cluster-IP, an internal port and an external-IP (the IP of the Node). The reality is that external-IP is <none>
curl host01:$NODE_PORT. Where does "host01" come from? It is neither my current host, nor the endpoint of kubernetes-bootcamp (according to kubectl describe). I can figure that it has the same ip as service kubernetes (kubectl describe deployment/kubernetes), but why?

Proposed Solution:
Better explanation to eliminate the confusion.
Page to Update:
http://kubernetes.io/...

Thanks for the feedback on this. We've updated how the setup works to make the hostname clearer.
/close


css and asset cleanup: There is a lof of stuff the website does not use since we massively forked the gitbooks jekyll template - the things that we don’t use shall better be cleaned up / removed from the repo

Closing; dupe of https://github.com/vmware-tanzu/secrets-manager/issues/106


Fix mesh surface upgrade tool error.

Fixes #100232
Passes false to the restart_editor method (we don't want to return to the project list).  This fixes this error:
ERROR: Error calling deferred method: 'EditorNode::EditorNode::restart_editor': Method expected 1 arguments, but called with 0.
   at: _call_function (core/object/message_queue.cpp:222)

Thank you for your contribution, however this is a duplicate of:

https://github.com/godotengine/godot/pull/99695

Closing this PR as there's already an open PR solving this, but thank you nonetheless


fix: provide analytics data for graphql plugin

Description

Currently all the required data is already being sent. This PR introduces a minor change to rename the plugin name along the same lines as the REST API plugin since the current plugin name could be a bit misleading.

Fixes #12394
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Manual
Automated test for this seems not possible at the moment.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Closing since some extra changes seem to have crept in. Will open a new PR.


Add additional responses

See https://fastapi.tiangolo.com/advanced/additional-responses/
Existing tests were updated to match OpenAPI schema more.

@koxudaxi I also need this feature...what is the status on this PR?

I'm sorry. I'm very busy🙇
I will release it as soon as possible by this weekend.

@rominf @koxudaxi
I've been testing this PR out and there are two things I've noticed so far.

When an operation has additional_responses and you are building the function there isn't a return type annotation. Does it make sense to use the Union type in these instances?

def {{operation.function_name}}({{operation.snake_case_arguments}}){% if operation.additional_responses %} -> Union[{{operation.response}},{{operation.additional_responses.values()|map(attribute='model')|unique|join(', ')}}] {% else %} -> {{operation.response}}{% endif %}:

I'm seeing some funny behavior with the root (/) endpoint. The operation.snake_case_path gets rewritten as a long auto generated string instead of just /

'/#-datamodel-code-generator-#-root-#-special-#'

@LongBeachHXC I've fixed (1) and also made my code cleaner. As for (2), I believe it's unrelated to this PR.

@rominf I'm noticing the following in my use case. I have an endpoint with many responses but the same response object.
responses={
        '403': {'model': ErrorContainer},
        '422': {'model': ErrorContainer},
        '500': {'model': ErrorContainer},
    }

) -> Union[User, ErrorContainer, ErrorContainer, ErrorContainer]:

@LongBeachHXC Fixed.

Applied reformat.

I don't know what is this, and how to fix it :-(

@rominf
Thank you for fixing the problem.
We just met the error.
I guess the error is the same.
https://stackoverflow.com/a/69913764
I will create a PR to fix the Black problem.

@rominf
I just released the new version 0.3.1
Thank you very much.
https://github.com/koxudaxi/fastapi-code-generator/releases/tag/0.3.1


Feature request: Allow chosing a branch

The ability to chose a branch of the repo other than the main one

Would a syntax like https://tokei.rs/b1/github/XAMPPRocky/tokei_rs@branch_name be suitable? @branch_name would be optional.

Thank you for your issue!  Is @ the conventional syntax for addressing a specific branch? If it is then it seems fine, otherwise I'd prefer a query parameter.

I don't think it is conventional in git, but I've seen @ being used for pip install. I think query parameter is better actually.


[Bug]: start-build-end.sh fails with go.mod file not found

What happened?
I cloned the repository, ran start-build-env.sh and after a while it fails with
=> ERROR [7/8] RUN go get github.com/linkedin/goavro/v2                                                                                0.1s

[7/8] RUN go get github.com/linkedin/goavro/v2:
0.134 go: go.mod file not found in current directory or any parent directory.
0.134 	'go get' is no longer supported outside a module.
0.134 	To build and install a command, use 'go install' with a version,
0.134 	like 'go install example.com/cmd@latest'
0.134 	For more information, see https://golang.org/doc/go-get-install-deprecation
0.134 	or run 'go help get' or 'go help install'.

Dockerfile:10
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[X] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue

I try to do this and it works:
first, comment 'RUN go get github.com/linkedin/goavro/v2' in start-build-env.sh
second, run start-build-env.sh
now, if there dont have other questions you will in [Beam Build Env]:~/beam
and then do this:
cd $GOPATH && go get github.com/linkedin/goavro/v2 && cd -

it will download the package: github.com/linkedin/goavro/v2 in ./pkg/mod/github

I try to add this to start-build-env.sh's Dockerfile content but it's not working:
...
ENV GOPATH ${DOCKER_HOME_DIR}/beam/sdks/go/examples/.gogradle/project_gopath
# switch to GOPATH to run go get command
WORKDIR ${DOCKER_HOME_DIR}/beam/sdks/go/examples/.gogradle/project_gopath/src
COPY . .
# This next command still runs as root causing the ~/.cache/go-build to be owned by root
RUN go get github.com/linkedin/goavro/v2
...


s5p-compress "some chunks keys are not dimensions on this object"

Hello, apologies for raising a new issue, however I have come across a new error when attempting the s5p-compress script and wondered if you had any ideas of how to over come it. I have tried everything I can think of to get around it, and I seem to get the same error no matter what product or band name I use.
Any ideas would be greatly appreciated. Thank you again for all the help previously.

python s5p-compress.py L2_data/L2__NO2___/S5P_OFFL_L2__NO2____20210310T102908_20210310T121038_17650_01_010400_20210312T035825.nc nitrogendioxide_tropospheric_column

/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rasterio/init.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py:761: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
warnings.warn(str(rio_warning.message), type(rio_warning.message))
Traceback (most recent call last):
File "s5p-compress.py", line 184, in
export_dir=EXPORT_DIR,
File "s5p-compress.py", line 38, in main
DS = rioxarray.open_rasterio(netcdf_file, chunks={"time": chunk_size})
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 774, in open_rasterio
mask_and_scale=mask_and_scale,
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 552, in load_subdatasets
default_name=subdataset.split(":")[-1].lstrip("/").replace("/", ""),
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 865, in open_rasterio
result = _prepare_dask(result, riods, filename, chunks)
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/rioxarray/_io.py", line 603, in _prepare_dask
return result.chunk(chunks, name_prefix=name_prefix, token=token)
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/xarray/core/dataarray.py", line 1058, in chunk
chunks, name_prefix=name_prefix, token=token, lock=lock
File "/home/ab/bw43776/anaconda3/envs/Sentinel/lib/python3.7/site-packages/xarray/core/dataset.py", line 1916, in chunk
"some chunks keys are not dimensions on this " "object: %s" % bad_dims
ValueError: some chunks keys are not dimensions on this object: {'time'}

Dear Beth Wilkins,
I think the problem comes from the band name.
Please use this band name: tropospheric_NO2_column_number_density
I hope it helps,
Note: by the way, can we organize a meeting for a short discussion !
cheers

Thank you so much for your reply @HichemOmr.
I also thought the problem was the band name as I was completely unsure what else it could be, so I ran the following script to display the names of the variables in the .nc files:
import netCDF4 as nc4
from netCDF4 import Dataset
import numpy as np
s5p_file = 'L2_data/L2__O3____/S5P_OFFL_L2__O3_____20210310T102908_20210310T121038_17650_01_020104_20210312T035820.nc'
fh = Dataset(s5p_file, mode='r')
print(fh.groups['PRODUCT'].variables.keys())
Which gave the following return:
python variables.py
odict_keys(['scanline', 'ground_pixel', 'time', 'corner', 'polynomial_exponents', 'intensity_offset_polynomial_exponents', 'layer', 'vertices', 'latitude', 'longitude', 'delta_time', 'time_utc', 'qa_value', 'nitrogendioxide_tropospheric_column', 'nitrogendioxide_tropospheric_column_precision', 'nitrogendioxide_tropospheric_column_precision_kernel', 'averaging_kernel', 'air_mass_factor_troposphere', 'air_mass_factor_total', 'tm5_tropopause_layer_index', 'tm5_constant_a', 'tm5_constant_b'])
I have tried the band name you suggested and added some of the optional functions but unfortunately got the same issue of:
anaconda3/envs/xarray/lib/python3.7/site-packages/xarray/core/dataset.py", line 1916, in chunk
"some chunks keys are not dimensions on this " "object: %s" % bad_dims
ValueError: some chunks keys are not dimensions on this object: {'time'}
I have checked that my conda environment meets the requirements.txt file (as I thought maybe the version of xarray was wrong or something similar), and have tried multiple band names and product types and always get this same weird issue. I have also tried it on a colleague's machine and they have had the same issue.  Apologies if this is not an issue directly related the the s5p-tools package, I was just wondering if maybe you knew a way around it. Thank you, Beth.

Launched 4 processes
Traceback (most recent call last):
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 199, in _acquire_with_cache_info
file = self.cache[self.key]
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\lru_cache.py", line 53, in getitem
value = self.cache[key]
KeyError: [<class 'netCDF4.netCDF4.Dataset'>, ('C:\Users\pawar\s5p-tools-master\L2_data\L2__O3\S5P_NRTI_L2__O3_____20210928T081425_20210928T081925_20514_02_020201_20210928T090545.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 401, in
main(
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 221, in main
attributes = {
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 223, in
"time_coverage_start": xr.open_dataset(filename).attrs[
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\api.py", line 497, in open_dataset
backend_ds = backend.open_dataset(
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 551, in open_dataset
store = NetCDF4DataStore.open(
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 380, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 328, in init
self.format = self.ds.data_model
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4_.py", line 389, in ds
return self.acquire()
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\netCDF4.py", line 383, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "C:\Users\pawar.conda\envs\project1\lib\contextlib.py", line 119, in enter
return next(self.gen)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "C:\Users\pawar.conda\envs\project1\lib\site-packages\xarray\backends\file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self.args, **kwargs)
File "src\netCDF4_netCDF4.pyx", line 2307, in netCDF4.netCDF4.Dataset.init
File "src\netCDF4_netCDF4.pyx", line 1925, in netCDF4.netCDF4.ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'C:\Users\pawar\s5p-tools-master\L2_data\L2__O3\S5P_NRTI_L2__O3_____20210928T081425_20210928T081925_20514_02_020201_20210928T090545.nc'
(project1) C:\Users\pawar\s5p-tools-master>Python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)
(project1) C:\Users\pawar\s5p-tools-master>Python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)
(project1) C:\Users\pawar\s5p-tools-master>python s5p-request.py L2__O3____
Traceback (most recent call last):
File "C:\Users\pawar\s5p-tools-master\s5p-request.py", line 15, in
from s5p_tools import (
ImportError: cannot import name 'bounding_box' from 's5p_tools' (unknown location)
Will You please guide me on the above errors


Bug fix

When trying to get subscribers on a page that doesn't have them, it gave an error.

breaks PHP5 compatibility

@mzaharov thank you for contribution


"[youtube] jNOEntzvJ48: Requested format is not available. Use --list-formats for a list of available formats"

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
Germany
Provide a description that is worded well enough to be understood
[info] Available formats for jNOEntzvJ48:
ID  EXT   RESOLUTION FPS CH │  FILESIZE  TBR PROTO │ VCODEC       VBR ACODEC     ASR MORE INFO
──────────────────────────────────────────────────────────────────────────────────────────────────────────
sb2 mhtml 48x27        0    │                mhtml │ images                          storyboard
sb1 mhtml 60x45        0    │                mhtml │ images                          storyboard
sb0 mhtml 120x90       0    │                mhtml │ images                          storyboard
160 mp4   192x144     25    │   5.59MiB  27k https │ avc1.4D400B  27k video only     144p, IOS, mp4_dash
134 mp4   480x360     25    │  11.76MiB  56k https │ avc1.4D4015  56k video only     360p, IOS, mp4_dash
18  mp4   480x360     25  2 │  32.15MiB 154k https │ avc1.42001E      mp4a.40.2  44k 360p, IOS
243 webm  480x360     25    │  19.73MiB  95k https │ vp9          95k video only     360p, MWEB, webm_dash
135 mp4   640x480     25    │  23.58MiB 113k https │ avc1.4D401E 113k video only     480p, IOS, mp4_dash
I only need the audio, but this video is special. Only the video source exists, I guess --extract-audio needs to be more intelligent; or does a special command line switch to handle this case already exist?
This case can be handled by:

automatically download the video
save it on disk
extract the audio
only keep the audio

I'm surprised --extract-audio does not handle this case, I think it should.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] $ yt-dlp -f bestaudio --extract-audio -vU -- https://www.youtube.com/watch?v=jNOEntzvJ48
[debug] Command-line config: ['-f', 'bestaudio', '--extract-audio', '-vU', '--', 'https://www.youtube.com/watch?v=jNOEntzvJ48']
[debug] User config "/home/ju/.config/yt-dlp/config": ['--write-description', '--write-info-json', '--add-metadata', '--embed-subs', '--ignore-errors', '--no-overwrites', '-o', '%(channel)s/%(title).150B/%(fulltitle).200Bs. – %(id)s.%(ext)s', '--concurrent-fragments', '1', '--compat-options', 'multistreams', '--compat-options', 'no-youtube-channel-redirect', '--compat-options', 'no-clean-infojson', '--cookies=/home/ju/.config/yt-dlp/cookies.txt', '--extractor-args', 'youtubetab:skip=webpage,authcheck', '--lazy', '--download-archive', '/home/ju/.config/yt-dlp/download-index.txt']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds [6fef82402] (zip)
[debug] Compatibility options: multistreams, no-youtube-channel-redirect, no-clean-infojson
[debug] Python 3.12.7 (CPython x86_64 64bit) - Linux-6.11.0-8-generic-x86_64-with-glibc2.40 (OpenSSL 3.3.1 4 Jun 2024, glibc 2.40)
[debug] exe versions: ffmpeg 7.1 (setts), ffprobe 7.1
[debug] Optional libraries: brotli-1.1.0, certifi-2024.08.30, mutagen-1.47.0, requests-2.32.3, secretstorage-3.3.3, sqlite3-3.46.1, urllib3-2.0.7
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests
[debug] Loaded 1837 extractors
[debug] Loading archive file '/home/ju/.config/yt-dlp/download-index.txt'
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds)
[youtube] Extracting URL: https://www.youtube.com/watch?v=jNOEntzvJ48
[youtube] jNOEntzvJ48: Downloading webpage
[youtube] jNOEntzvJ48: Downloading ios player API JSON
[youtube] jNOEntzvJ48: Downloading mweb player API JSON
[debug] Loading youtube-nsig.62ccfae7 from cache
[debug] [youtube] Decrypted nsig ODz4UyiaJcSiW_t3M3B => d0-It-eyAeR13A
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec, channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
ERROR: [youtube] jNOEntzvJ48: Requested format is not available. Use --list-formats for a list of available formats
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1624, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1780, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1839, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2973, in process_video_result
    raise ExtractorError(
yt_dlp.utils.ExtractorError: [youtube] jNOEntzvJ48: Requested format is not available. Use --list-formats for a list of available formats

--extract-audio is exactly the command you need. Just don't pass a -f option.


[Analyzer] Weird typing issue

This tracker is for issues related to:

Analyzer

The element type 'DateTime?' can't be assigned to the list type 'Object'.
dart [list_element_type_not_assignable]
(https://dart.dev/diagnostics/list_element_type_not_assignable)

Dart SDK version: 2.18.2 (stable) (Tue Sep 27 13:24:11 2022 +0200) on "macos_arm64"
I came across this weird typing glitch. I realise I've not provided much information of worth here so if there is anything else I can provide let me know (although this is not appearing in a a minimal reproducible example - it's in large production app). There are no other analzyer errors in the project at time of getting this.

Looks like profileLastLoaded has static type DateTime?, which is not a subtype of Object, which makes the type error correct.
If you know for sure that profileLastLoaded is non-null then you can use profileLastLoaded! whose type is DateTime (but if it is null after all then it will throw). Otherwise you could use an if-element to provide this list element conditionally, similar to this example:
int? whatever() => 4;

void main() {
  int? maybeInt = whatever();
  List<int> xs = [1, 2, 3, if (maybeInt != null) maybeInt, 5, 6];
}

I'll close this issue because the given information seems to be consistent with the assumption that it's all working as intended. Please create a new issue if there is more information available to change this situation.

Apologies @eernstg -I had assumed null was of type Object (it makes sense that it isn't). I will need to use List<Object?>
Thanks

Apologies

No need for that, thanks for contributing when you saw something unexpected!


[SPARK-31850][SQL]Prevent DetermineTableStats from computing stats multiple times for same table

What changes were proposed in this pull request?

Changes to avoid stats computation for same table multiple times

Repro steps
spark.sql("set spark.sql.hive.convertMetastoreParquet=false")
spark.sql("create table c(id INT, name STRING) STORED AS PARQUET")
val df = spark.sql("select count(id) id from c group by name order by id " )
df.queryExecution.analyzed

Note:

There is no log line in DetermineTableStats to indicate that stats compute happened. Need to add a log line or use a debugger

The above can be repro-ed with first query on a created table.

Why are the changes needed?
Stats computation might be an expensive operation especially for a large table
Once stats are computed for a table, it can be re-used.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit tests added

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@dongjoon-hyun @cloud-fan @gatorsmile Can anyone please help reviewing this change. Thanks

Tagging few more committers from the file's git history for review: @HeartSaVioR @holdenk @maropu
Thank you.

Even if https://github.com/apache/spark/pull/28662 fixed, we still need this fix? At least, it seems the issue of the query above will go away if #28662 merged.

@maropu in the above comment are referring to chnages in https://github.com/apache/spark/pull/28686 ?
Even if #28686 is fixed, the issue will still be there when the flag to convert Parquet/Orc Hive table to datasource tables is disabled.

Ur, that's a wrong number. As you said, I meant #28686.

You disable that option in your usecase? I thought most users turn on the flag to convert it to a datasrouce table. Anyway, I just want to know a priority to fix this issue.

@maropu I my example I took the case of parquet as data format. This can happen with formats other than parquet/orc(like JSON, CSV etc)

Can you explain why DetermineTableStats will calculate the statistics multiple times?

  at org.apache.spark.sql.hive.DetermineTableStats.hiveTableWithStats(HiveStrategies.scala:121)
  at org.apache.spark.sql.hive.DetermineTableStats$$anonfun$apply$2.applyOrElse(HiveStrategies.scala:150)
  at org.apache.spark.sql.hive.DetermineTableStats$$anonfun$apply$2.applyOrElse(HiveStrategies.scala:147)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$2(AnalysisHelper.scala:108)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$870.808816071.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:108)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$869.1113025977.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$4(AnalysisHelper.scala:113)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$872.1354725727.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$1144.1492742163.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:113)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$869.1113025977.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$4(AnalysisHelper.scala:113)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$872.1354725727.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$1144.1492742163.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:397)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:350)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDown$1(AnalysisHelper.scala:113)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$869.1113025977.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown(AnalysisHelper.scala:106)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsDown$(AnalysisHelper.scala:104)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsDown(LogicalPlan.scala:29)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators(AnalysisHelper.scala:73)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperators$(AnalysisHelper.scala:72)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:29)
  at org.apache.spark.sql.hive.DetermineTableStats.apply(HiveStrategies.scala:147)
  at org.apache.spark.sql.hive.DetermineTableStats.apply(HiveStrategies.scala:114)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:149)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor$$Lambda$863.668742490.apply(Unknown Source:-1)
  at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)
  at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)
  at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:49)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:146)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:138)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor$$Lambda$862.773865813.apply(Unknown Source:-1)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:138)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:176)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$20.applyOrElse(Analyzer.scala:2139)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$20.applyOrElse(Analyzer.scala:2116)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$3(AnalysisHelper.scala:90)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$866.1662235713.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$1(AnalysisHelper.scala:90)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$Lambda$864.152426436.apply(Unknown Source:-1)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:86)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp$(AnalysisHelper.scala:84)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2116)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2115)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:149)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor$$Lambda$863.668742490.apply(Unknown Source:-1)
  at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
  at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
  at scala.collection.immutable.List.foldLeft(List.scala:89)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:146)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:138)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor$$Lambda$862.773865813.apply(Unknown Source:-1)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:138)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:176)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:170)

In the above trace we can see DetermineTableStats got triggered as part of executing ResolveAggregateFunctions.
ResolveAggregateFunctions is part of a batch which has FixedPoint as its execution strategy, hence the computation can happen any number of times, based on when Fixed point is reached.
This is not specific to ResolveAggregateFunctions, any analyzer rule which invokes org.apache.spark.sql.catalyst.analysis.Analyzer#executeSameContext  will face this issue.
In best case , DetermineTableStats rule will run atleast thrice as part of analysis, twice as part of ResolveAggregateFunctions(assuming fixed point is reached in the first two attempts ) and once as past of postHocResolutionRules#DetermineTableStats

Once you finish query analysis of a dataframe, the analyzed plan is kept as QueryExecution.analyzed. Why accessing it will cause re-calculation?
In the description, i had written the code to trigger the analysis phase. At the end of analysis phase, DetermineTableStats would have run multiple times, which could slow down query perf.

I see, it is because calling executeSameContext to analyze one logical plan during analysis. I'd say that the reproducing steps in the description is confusing because it doesn't show any evidence about multiple runs of DetermineTableStats. I think it is better to describe the issue clearly.
Btw, as we already calculate statistics and save into HiveTableRelation in DetermineTableStats. To prevent redundant calculation, is it much easier and simpler to just add new condition like:
class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
    case relation: HiveTableRelation
      if DDLUtils.isHiveTable(relation.tableMeta) && relation.tableMeta.stats.isEmpty  && relation.tableStats.isEmpty =>
      hiveTableWithStats(relation)
    ...
}

The above condition is already present.
But we return a copy of relation(code: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala#L137) with the updated Table Stats at the end of the method

When ResolvedAggregateFunction rule runs again(to achieve Fixed point), it will not be aware of the updated relation. executeWithSameContext with rerun the Stats collection as part of DetermineTableStats rule.
When the DetermineTableStats rule actually runs as part of Analysis phase, it will not be aware of the updated relation
@viirya

@viirya @maropu Can you please help review this PR

@viirya @maropu Can you please help review this PR

https://github.com/apache/spark/pull/28686 should handle most cases. Closing this PR.


doc: update admin-api docs

NOTE: Please read the Contributing.md guidelines before submitting your patch:
https://github.com/apache/incubator-apisix/blob/master/Contributing.md#how-to-add-a-new-feature-or-change-an-existing-one
Summary
SUMMARY_HERE
Full changelog

[Implement ...]
[Add related tests]
...

Issues resolved
Fix #XXX

@scaat many thx, merged


Usage of static logger instance in public static methods at CertificateUtil.java

The public static method use static logger. But GoodLoggerCheck won't allow static logger.
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/identity/azure-identity/src/main/java/com/azure/identity/implementation/util/CertificateUtil.java#L27

Please update this issue with a fully-qualified reference to the bad file.

Closing as the guideline is changing around static loggers. https://github.com/Azure/azure-sdk/pull/1155


uri_template: supporting operator<<

Commit Message: uri_template: supporting operator<<
Additional Description:
This minor change adds pretty-print support for thr uri_template ParsedSegment.
Risk Level: low
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: N/A

/retest

/retest

Assigning Yanjun as code-owner
/assign @yanjunxiang-google

LGTM

Assigning Alyssa as code-owner and senior-maintainer.
/assign @alyssawilk


Make running yarn dev with no errors

📝 Description
Fix multiple errors occurred after running yarn dev for multiple PF components (however this is not reproducible always):
<ComponentName> cannot be used as a JSX component
Fix the errors by adding missing "@types/react-dom" package, also adding "@patternfly/react-icons" and "@types/react-dom" to the resolutions in the package.json file, as some of our packages use those packages as their dependencies, and that lead to the mess as yarn does not do de-duplication, and so those packages were installed more times, which lead to typescript being confused. And so also "react": ["./node_modules/react", "./node_modules/@types/react"] had to be added to tsconfig.json file, too, to tell it explicitly, which package and types to use.
Also remove the duplication of CheckupsStorageForm.tsx.
TODO:

[x] fix broken unit tests

🎥 Screenshots
Before:
Terminal after executing yarn dev:

Listing the dependencies for a specific package:
yarn list --pattern @types/react
yarn list -- pattern @patternfly/react-icons

After:

/hold

/hold cancel

/retest

/retest

No errors with this code on my PC @hstastna

@avivtur @metalice @pcbailey @upalatucci @vojtechszocs please review

/lgtm


Hide all tooltips

Hi @florent37
I am using this on RecyclerView, When it's scrolled i need to hide tooltips which are sticked to window.
I can use tooltip.close();
but i show lot of tooltips, so is there any method to hide all tooltips at once?

And also the tooltips are visible when we scroll or go to other fragment.

i encountered the same problem.


Paymaster on metamask using usdc to paid fees

🐛 Bug Report
📝 Description
When i swap in syncswap used usdc to pay fees, after sign window stay open and i don't have a message to tell me if transaction are good and who much it's cost
🔄 Reproduction Steps
I use metamask 7.14.0 on android 14
Steps to reproduce the behaviour
Open syncdwap.xyz connect wallet and swap eth to usdc or other crypto, sign transaction, window open don't say if is good stay open
🤔 Expected Behavior
sign tx, tx cost xxx usdc, validation and window closed when tx finish
😯 Current Behavior
no cost appear, window stay open, tx good but no message send
🖥 Environment
Smartphone on Android 14 up to date

This one is not displayed, and the same goes for imtoken

Here are some suggestions for resolving or further investigating the issue:
Clear Cache and Try Again:
Clear the cache of your browser or Metamask app to ensure there are no cached data causing the issue. Then, try the swap process again.
Update Metamask:
Make sure you are using the latest version of Metamask. If not, update to the latest version from the Google Play Store.
Check Gas Settings:
Ensure that your gas settings are appropriate for the transaction. Sometimes, setting a higher gas fee might result in a quicker confirmation.
Network Confirmation:
After signing the transaction, check the transaction status directly on the Ethereum blockchain explorer (e.g., Etherscan). This will confirm if the transaction was successfully processed and provide details about the transaction cost.
Contact Support:
Reach out to the support team of SyncSwap or Metamask, explaining the issue and providing details about your device, browser, and any relevant settings. They may be able to provide specific guidance or updates related to the bug.
Alternative Browser:
Try using an alternative mobile browser to see if the issue persists. Some browsers may have different compatibility with certain web applications.
Community Forums:
Check if there are any discussions or known issues related to SyncSwap on community forums, social media, or the project's GitHub repository. Other users may have encountered similar problems and could provide insights or solutions.
Operating System Update:
Ensure that your Android device is running the latest available version. Some issues may be resolved with operating system updates.
If the problem persists after trying these steps, it would be advisable to report the bug to the SyncSwap development team or Metamask support, providing them with the details you've mentioned in your bug report. They can investigate the issue further and potentially release a fix in future updates.

thanks
------------------ Original ------------------
From: jrchatruc @.>
Date: Thu,Feb 22,2024 4:44 PM
To: matter-labs/zksync-era @.>
Cc: a529179741 @.>, Comment @.>
Subject: Re: [matter-labs/zksync-era] Paymaster on metamask using usdc to paidfees (Issue #1168)

Here are some suggestions for resolving or further investigating the issue:
Clear Cache and Try Again:
Clear the cache of your browser or Metamask app to ensure there are no cached data causing the issue. Then, try the swap process again. Update Metamask:
Make sure you are using the latest version of Metamask. If not, update to the latest version from the Google Play Store. Check Gas Settings:
Ensure that your gas settings are appropriate for the transaction. Sometimes, setting a higher gas fee might result in a quicker confirmation. Network Confirmation:
After signing the transaction, check the transaction status directly on the Ethereum blockchain explorer (e.g., Etherscan). This will confirm if the transaction was successfully processed and provide details about the transaction cost. Contact Support:
Reach out to the support team of SyncSwap or Metamask, explaining the issue and providing details about your device, browser, and any relevant settings. They may be able to provide specific guidance or updates related to the bug. Alternative Browser:
Try using an alternative mobile browser to see if the issue persists. Some browsers may have different compatibility with certain web applications. Community Forums:
Check if there are any discussions or known issues related to SyncSwap on community forums, social media, or the project's GitHub repository. Other users may have encountered similar problems and could provide insights or solutions. Operating System Update:
Ensure that your Android device is running the latest available version. Some issues may be resolved with operating system updates. If the problem persists after trying these steps, it would be advisable to report the bug to the SyncSwap development team or Metamask support, providing them with the details you've mentioned in your bug report. They can investigate the issue further and potentially release a fix in future updates.

Thank you for this answer, i have last version of metamask and android. I think it is syncswap web site problem. Do you try it on a computer ?


support attachment subfolders

The attachments might be created in subfolders by plugins such as https://github.com/trganda/obsidian-attachment-management

Thank you for the PR, i think i fixed the problem feel free to test the master branch i am going to publish the next version in few days if it works as it should


[Java] Specify JPMS module names

Is your feature request related to a problem? Please describe.
When using fury in a modular (JPMS) application, the maven-compiler-plugin warns that
[WARNING] ****************************************************************************************************************************************************
[WARNING] * Required filename-based automodules detected: [fury-core-0.5.0-SNAPSHOT.jar]. Please don't publish this project to a public artifact repository! *
[WARNING] ****************************************************************************************************************************************************

Describe the solution you'd like
With this feature request I propose to claim and configure module names to get rid of this warning.
I basically see two feasible approaches to achieve this goal:

The least invasive approach would be to use the Automatic-Module-Name entries in the MANIFEST.MFs of the respective .jar files. This does not affect the build process in any way and can be configured relatively easy via configurations in the maven-jar-plugin (or the maven-shade-plugin if you use it).
To fully support named modules, you would have to provide a full module-info.java descriptor. Since this is a Java9+ feature, you would have to extend the build process to either use multiple compile phases (one for module-info.java on Java9+ and one for the remaining code on Java8 to maintain backwards compatibility) or build a multi-release JAR. This would make sense if you want to support tools such as jlink or jpackage in the future. However, this would also require your dependencies to be fully modularized. So if that is not the case, it is probably not worth the effort for now.

Additional context
I could lend a hand and provide an initial PR for the first solution. The only thing worth discussing beforehand is which module names you want to claim. You could either go with the direct translation of the maven coordinates, e.g. org.apache.fury.fury_core, org.apache.fury.fury_format, etc. or go with more idiomatic names such as org.apache.fury.core, org.apache.fury.format, etc.

Hi @mtf90 , thanks for bringing up this. I think we can go with first solution first. Some dependency such as arrow used by fury-format hasn't released support for JPMS. It's supported in main branch: https://github.com/apache/arrow/blob/main/java/memory/memory-core/src/main/java/module-info.java, but not released yet.
And org.apache.fury.core, org.apache.fury.format looks good to me. Would you like to create a pr for this?
In the long run, I think Fury should support JPMS fully. One thing I'm not sure whether it affect java8 users. Fury will support java8 for a long time, the support for JPMS should not break the support for java8

Dear @chaokunyang, I created a PR. Feel free to change the module names for the test modules. I went with a hierarchical naming convention since you seem to prefer the org.apache.fury.core approach.
Maybe I can share some experience for adding full JPMS support in the future since I'm in a similar situation with @LearnLib at the moment (which is why I encountered the missing module names of Fury in the first place).
We went with the two-compilation-phase approach which

compiles everything to Java 9 in order to use the compiler's built-in checks for sound module-info.java definitions and then
re-compile everything but the module-info.java file to Java 8.

This allows us to keep the module-info.java file alongside the actual code and profit off of the IDE's support for modules. The main disadvantage is that a Java 9+ compiled class is now alongside the other .class files in the built .jar file. However, since module-info is not a valid Java identifier, you cannot reference this class directly in regular applications. Only (Java 8) code that deliberately scans the contents of a .jar file may run into UnsupportedClassVersionErrors
An alternative would be to keep the module-info.java in a separate source directory (something like src/main/java9) and construct a multi-release JAR by compiling (only) the module-info.java descriptor to the the META-INF/versions/9 directory of the .jar file (which is what, e.g., slf4j does). The main disadvantage I see with this approach is (besides the wonky IDE support) that the multi-release semantics are defined only on the .jar level. However, with maven, your typical lifecycle is compile -> test -> package so you would need to move certain (unit-) tests to the integration-test phase in order to test your module definitions (such as service definitions via uses and provides).

Hi @mtf90 , thanks for submitting the PR #1343 , the naming looks good to me.
And thanks for detailed explaination for JMPS solution too, I can see a feasible path top JPMS now. Fury currently dependends on guava, which doesn't support JPMS https://github.com/google/guava/issues/2970. We may need to hold this until guava support it or we remove guava dependencies.