Art explainer crashes the kfserving controller

/kind bug
What steps did you take and what happened:
http://86308603-argo-argo-5ce9-1162466691.us-west-2.elb.amazonaws.com/workflows/kubeflow-test-infra/kubeflow-kfserving-presubmit-e2e-1306-d0f3432-5360-280e?tab=workflow&nodeId=kubeflow-kfserving-presubmit-e2e-1306-d0f3432-5360-280e-159579293&sidePanel=logs%3Akubeflow-kfserving-presubmit-e2e-1306-d0f3432-5360-280e-159579293%3Amain
{"level":"info","ts":1611193722.1990366,"logger":"inferenceservice-v1beta1-mutating-webhook","msg":"Defaulting InferenceService","namespace":"kfserving-ci-e2e-test","isvc":{
"containers":[{"name":"kfserving-container","image":"aipipeline/art-server:mnist-predictor","command":["python","-m","sklearnserver","--model_name","art-explainer","--model_
dir","file://sklearnserver/sklearnserver/example_model"],"resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}}]}}
2021/01/21 01:48:42 http: panic serving <IP_ADDRESS>:49516: runtime error: invalid memory address or nil pointer dereference
goroutine 757 [running]:
net/http.(*conn).serve.func1(0xc000be6320)
        /usr/local/go/src/net/http/server.go:1772 +0x139
panic(0x175c340, 0x28ab210)
        /usr/local/go/src/runtime/panic.go:975 +0x3e3
github.com/kubeflow/kfserving/pkg/apis/serving/v1alpha2.(*InferenceService).ConvertFrom(0xc000275680, 0x1bcf900, 0xc000af6000, 0x1bcf900, 0xc000af6000)
        /go/src/github.com/kubeflow/kfserving/pkg/apis/serving/v1alpha2/inferenceservice_conversion.go:346 +0x2eb
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.(*Webhook).convertObject(0xc00002ddb0, 0x1baca80, 0xc000af6000, 0x1bac9c0, 0xc000275680, 0x1bac9c0, 0xc000275680)
        /go/pkg/mod/github.com/zchee/sigs.k8s-controller-runtime@v0.6.1-0.20200623114430-46812d3a0a50/pkg/webhook/conversion/conversion.go:142 +0x7bc
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.(*Webhook).handleConvertRequest(0xc00002ddb0, 0xc000e40540, 0xc000754cc0, 0x0, 0x0)
        /go/pkg/mod/github.com/zchee/sigs.k8s-controller-runtime@v0.6.1-0.20200623114430-46812d3a0a50/pkg/webhook/conversion/conversion.go:107 +0x1f8
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.(*Webhook).ServeHTTP(0xc00002ddb0, 0x7f6851c8fad8, 0xc000b88190, 0xc000ea7300)
        /go/pkg/mod/github.com/zchee/sigs.k8s-controller-runtime@v0.6.1-0.20200623114430-46812d3a0a50/pkg/webhook/conversion/conversion.go:74 +0x10b
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1(0x7f6851c8fad8, 0xc000b88190, 0xc000ea7300)
        /go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:40 +0xab
net/http.HandlerFunc.ServeHTTP(0xc000315e60, 0x7f6851c8fad8, 0xc000b88190, 0xc000ea7300)
        /usr/local/go/src/net/http/server.go:2012 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0x1bddf00, 0xc0005f62a0, 0xc000ea7300)
        /go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:100 +0xda
net/http.HandlerFunc.ServeHTTP(0xc0003e0000, 0x1bddf00, 0xc0005f62a0, 0xc000ea7300)
        /usr/local/go/src/net/http/server.go:2012 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x1bddf00, 0xc0005f62a0, 0xc000ea7300)
        /go/pkg/mod/github.com/prometheus/client_golang@v1.6.0/prometheus/promhttp/instrument_server.go:76 +0xb2
net/http.HandlerFunc.ServeHTTP(0xc0003e0270, 0x1bddf00, 0xc0005f62a0, 0xc000ea7300)
        /usr/local/go/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc000601cc0, 0x1bddf00, 0xc0005f62a0, 0xc000ea7300)

What did you expect to happen:
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

Istio Version:
Knative Version:
KFServing Version:
Kubeflow version:
Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
Minikube version:
Kubernetes version: (use kubectl version):
OS (e.g. from /etc/os-release):

/cc @Tomcli @drewbutlerbb4 @animeshsingh

/cc @Tomcli @drewbutlerbb4 @animeshsingh

I think it is probably due to the fact that art explainer is not implemented on v1alpha2 so dst.Spec.Default.Explainer is null
https://github.com/kubeflow/kfserving/blob/master/pkg/apis/serving/v1alpha2/inferenceservice_conversion.go#L346

I think it is probably due to the fact that art explainer is not implemented on v1alpha2 so dst.Spec.Default.Explainer is null
https://github.com/kubeflow/kfserving/blob/master/pkg/apis/serving/v1alpha2/inferenceservice_conversion.go#L346

I thought v1alpha2 is going to be deprecated in the future releases so I didn't add it to the PR. Not sure is there a way to opt-out the converter to avoid this error, but I can add the v1alpha2 api if that's required.

I thought v1alpha2 is going to be deprecated in the future releases so I didn't add it to the PR. Not sure is there a way to opt-out the converter to avoid this error, but I can add the v1alpha2 api if that's required.

@Tomcli I think we do not need to add back to v1alpha2, we just need a null check if the explainer does not exist on v1alpha2.
if dst.Spec.Default.Explainer != nil {
dst.Spec.Default.Explainer.ServiceAccountName = src.Spec.Explainer.PodSpec.ServiceAccountName
dst.Spec.Default.Explainer.MinReplicas = src.Spec.Explainer.MinReplicas
dst.Spec.Default.Explainer.MaxReplicas = src.Spec.Explainer.MaxReplicas
if src.Spec.Explainer.ContainerConcurrency != nil {
dst.Spec.Default.Explainer.Parallelism = int(*src.Spec.Explainer.ContainerConcurrency)
}
}

@Tomcli I think we do not need to add back to v1alpha2, we just need a null check if the explainer does not exist on v1alpha2.
if dst.Spec.Default.Explainer != nil {
dst.Spec.Default.Explainer.ServiceAccountName = src.Spec.Explainer.PodSpec.ServiceAccountName
dst.Spec.Default.Explainer.MinReplicas = src.Spec.Explainer.MinReplicas
dst.Spec.Default.Explainer.MaxReplicas = src.Spec.Explainer.MaxReplicas
if src.Spec.Explainer.ContainerConcurrency != nil {
dst.Spec.Default.Explainer.Parallelism = int(*src.Spec.Explainer.ContainerConcurrency)
}
}


make verbosity disabled by default

Users can still enable verbosity by re-running a CI job with debugging logs enabled.
Not outputting debug info in production will save users CI time.

speed improves by a negligible amount (~20 ms):

I'll be setting up a benchmark test in the cpp-linter/cpp_linter_rs repo that will use more complex and laborious scenarios. Currently, I'm just running the benchmark tests in repo root (using tests/demo/demo.* files and default checks).

I have another speed-up trick up my sleeve: Parse clang-tidy and clang-format stdout from buffered strings instead of writing/reading to a cached files.
I only used cached files for easier development debugging. I rarely use the cached files for anything now. Unfortunately, this isn't a simple change given how the markdown comment is constructed from the parsed output.


Scheduling a container to a specific minion

Is there a recommended way to force schedule a pod to a particular minion? Some docs suggest that labels could be used for this, but its not clear how.

Yes, use the host field in the pod spec.  There are several e2e tests that do this if you need sample code.


Todoist: Use name of containing subproject as Toggl project

Todoist displays entries of a subproject below the main project's entries. Currently toggl-button just uses the main project's name as the Toggl project. This commit adds the ability to use the subproject name.
Any depth of project nesting is handled. We determine the name of the project, its parent, grandparent, and so on up the project hierarchy. Then we pass an array of these project names to the createTimerLink function so that each may be tried successively until one of them matches a Toggl project. (The matching doesn't happen until we click the Start Timer and it sends a message to the extension's background code, which I've enhanced slightly to accept a list of project names to look for.)
We determine the parent of a Todoist project by inspecting the indentation of their entries in the sidebar menu.

To see the functionality I'm adding:

Create two Todoist projects with names that match two of your Toggl projects.
Add a task to each Todoist project.
In Todoist, make one of the projects a sub-project of the other (drag the project name in the sidebar)
Now click the name of parent project in the sidebar to view it afresh. You'll see tasks from both the parent project and sub-project (each with their own project heading).
Click the 'Start timer' link next to a task in the sub-project.
Verify that the Toggl timer entry that just started has the project named after the sub-project rather than the parent project.

Now I got it working :+1:
LGTM except for the small code comments.


Include OpenMP support on Windows

Followup after #246 is merged

Requires successful build of the Windows wheels with OpenMP.


Fix change in vardumper class.

Q
A

Is bugfix?
✔

New feature?
✔

Breaks BC?
✔

Tests pass?
✔

Thanks!


Fixed pfi::network::mprpc not to time out when timeout_sec is 0

Current server and client of pfi::network::mprpc time out immediately when timeout_sec is 0 or smaller than 0.
This PR fixes this problem.

Looks good to me. Thanks!


#2012 - Fix 404 page when going to org profile on mobile

What does this pull request aim to achieve? 💯
closes #2012
Please be concise and link any related issue(s):

[x] I have checked that no one else is working on similar changes.
[x] I have read the latest README and understand the development workflow.
[x] The name of this branch is: feature/<branch_name>, or hotfix/<branch_name> if this is a hotfix, where <branch_name> briefly describes the issue(s) resolved and is prefixed with the issue number(s) (e.g. 1127-1130-update-git-branching-model). The branch is created in a cloned version of this repo, not a forked repo.
[x] This branch is rebased/merged with the latest staging (or the latest production if it is a hotfix).
[x] There are no merge conflicts.
[x] There are no console warnings and errors.
[x] On the right hand side of this PR, I have linked the appropriate issue(s) under "Issues" and added a project progress under "Projects".
[x] The title describes the issue(s) being addressed using format: <issue_numbers> - <brief_issue_description> (e.g. #1127/#1130 - Update Git Branching Model)

Tested on Android mobile
Able to switch to organization account and view the profile.
Works as expected.

Tested on iPhone and works.


Update Readme

Here are the details for updating the README:

Project Overview: In the README, please provide an overview of the project, explaining what it does and its main features. This will help visitors quickly understand the purpose of the project.

Contributing Guidelines: Update the CONTRIBUTING.md file as well. Specifically, please provide clear and detailed instructions on how to set up the project and get it running locally. Ensure that the setup process is well-documented and beginner-friendly.

Feel free to reach out if you have any questions or need further information. Your contributions to improving our project documentation are greatly appreciated!

Can you assign me for this?

Can you assign me for this?

Assignee: @Madhvik007
Hi Madhvik007,
Thank you for offering to update the README! I've assigned the task to you.
Here are the details for updating the README:

Project Overview: In the README, please provide an overview of the project, explaining what it does and its main features. This will help visitors quickly understand the purpose of the project.

Contributing Guidelines: Update the CONTRIBUTING.md file as well. Specifically, please provide clear and detailed instructions on how to set up the project and get it running locally. Ensure that the setup process is well-documented and beginner-friendly.

Feel free to reach out if you have any questions or need further information. Your contributions to improving our project documentation are greatly appreciated!

Hello @ajaynegi45  can you please assign this to me?

Hello @ajaynegi45 can you please assign this to me?

Assignee: @riddhi-283
Hi @riddhi-283 ,
Thank you for offering to update the README! I've assigned the task to you.
Here are the details for updating the README:

Project Overview: In the README, please provide an overview of the project, explaining what it does and its main features. This will help visitors quickly understand the purpose of the project.

Contributing Guidelines: Update the CONTRIBUTING.md file as well. Specifically, please provide clear and detailed instructions on how to set up the project and get it running locally. Ensure that the setup process is well-documented and beginner-friendly.

Feel free to reach out if you have any questions or need further information. Your contributions to improving our project documentation are greatly appreciated!

@Madhvik007 and @riddhi-283 are you currently working on this..?

Can you assign me for this?

Hi @BijoySaga,
Thank you for offering to update the README! I've assigned the task to you.
Feel free to reach out if you have any questions or need further information. Your contributions to improving our project documentation are greatly appreciated!

Hi, can you assign this issue to me?


v0.5.2, v0.5.3, v0.6.0 Release Tracker

Anything you want to discuss about vllm.
We will make a triplet of releases in the following 3 weeks.

[ ] v0.5.2 on Monday July 15th.
[ ] v0.5.3 by Wednesday July 23rd.
[ ] v0.6.0 after Monday July 29th.

The reason for such pace is that we want to remove beam search (#6226), which unlocks a suite of scheduler refactoring to enhance performance (async scheduling to overlap scheduling and forward pass for example). We want to release v0.5.2 ASAP to issue warnings and uncover new signals. Then we will decide the removal in v0.6.0. Normally we will deprecate slowly by stretching it by one month or two. However, (1) RFC has been opened for a while (2) it is unfortunately on the critical path of refactoring and performance enhancements.
Please also feel free to add release blockers. But do keep in mind that I will not slow the release for v0.5.* series un

July 23rd is Tuesday. Do you mean July 24th?

v0.5.2 has been released: https://github.com/vllm-project/vllm/releases/tag/v0.5.2

Hello. Can #4409 be included in any of the next releases? Or at least, can I get an explanation as to why it can't be included (maybe I can help in some way?).
Once the wheel size limit increase was approved, and #6394 was merged, wheel size should not be an issue.
I am currently waiting for PyPI staff to approve the wheel size limit increase request to publish the patched triton to PyPI (pypi/support#4295).
It would be nice to see support for Pascal GPUs in vLLM. Many people use them because they are cheap.

Can this PR be added in v0.5.3?
https://github.com/vllm-project/vllm/pull/5036

@AlphaINF unlikely given the current state of the PR at the moment (still being reviewed). but I'm looking very much forward to this PR as well!

@simon-mo thanks!

@simon-mo  for this "async scheduling to overlap scheduling" do we have a plan？

hello, when will v0.6.0 release? I'm looking forward to https://github.com/vllm-project/vllm/pull/5036 and MiniCPM-Llama3-V-2_5

Would it be possible to get #6594 merged in before the next release is due? @joerunde @Yard1

#6463 is not available in v0.5.5 docker image. Can you please have it available for 0.6.0?


Add @emillon's changelog check action

This PR adds a github action to check that the changelog has been updated unless the no changelog label is added.
Thanks to @emillon for working on this!

It seems to work as intended!

@gpetiot @pitag-ha could you please have a look at this so that we can merge it quick and make sure the changelog never goes out of date again?
I think the travis failures are completely unrelated so we should just ignore them. I'm looking how to restart the builds but I don't seem to have sufficient permissions for that atm!


why the downchannel should send directives interface every seconds

in your project,we send directives interface to amazon serve evert seconds,when we catch IOException,we send the serve request immediately in Downchannel.java

Yes, I thought if the call to enqueue fails then rather than starting again I should make the call again to that method and send the request again to amazon server. Is this what you are asking?


dataset

can  i have the original dataset mentioned in the train_gpu script

sure, you can download the dataset in https://www.kaggle.com/datasets/alxmamaev/flowers-recognition


[Event Hubs] Processor post-release

Summary
The focus of these changes is to update the project version metadata and change log.  Because of the GeoDR beta package version being a minor higher, this must be done by hand because automation gets confused.

API change check
API changes are not detected in this pull request.


beta release preparation

[x] fix dpp sort order
[x] align dpp content items to top
[x] configure sentry for cms app

:tada: This PR is included in version 1.0.0-beta.2 :tada:
The release is available on:

GitHub release
v1.0.0-beta.2

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

GitHub release
v1.0.0

Your semantic-release bot :package::rocket:


withAuthenticationRequired causing infinite loops when calling wrapped component's effect

Describe the problem
Since upgrading to 1.2.0 from 1.0.0, it seems that creating a ProtectedRoute using withAuthenticationRequired leads to infinite remounting of components.
I am using React 16.13.1, react-router-dom 5.2.
This problem does not happen with auth0-react 1.0.0
export const ProtectedRoute = ({
                                component,
                                ...args
                              }: {
  component: ComponentType;
  exact?: boolean;
  path: string;
}) => {
  const Comp = withAuthenticationRequired(component, {});
  // @ts-ignore
  return <Route {...args} render={(props) => <Comp {...props} />} />;
};

Usage that will cause it to break
// imagine this is inside a switch in react-router
<ProtectedRoute component={MyComponent} path="/whatever" />

// now this is our MyComponent, it calls an async thunk

function MyComponent() {
  const fetchResults = async () => {
    try {
      dispatch(someAsyncThunk());
    } catch (err) {
    }
  };

  useEffect(() => {
    fetchResults();
  }, []);

  return (
I will just keep calling my effect!
)
}

Hi @type
Can you change your ProtectedRoute component to:
export const ProtectedRoute = ({
                                component,
                                ...args
                              }: {
  component: ComponentType;
  exact?: boolean;
  path: string;
}) => {
  return <Route {...args} render={(props) => {
    // wrap the component inside the render function,
    // to stop withAuthenticationRequired getting recreated
    // on every render
    const Comp = withAuthenticationRequired(component, {});
    return <Comp {...props} />;
  }} />;
};

Let me know if that resolved your problem

Hi @adamjmcgrath , unfortunately this does not fix it. Tested your provided code against 1.2.0 (failed) and 1.0.0 (succeeded), so same result as with the code I provided.

My bad, I had an axios interceptor that got left in on accident in the App.jsx component that was using getAccessTokenSilently(). This was working okay with 1.0.0 but broke with 1.2.0. But not really a bug!

Hello,
We had the same issue, and I think it's problematic that we can't use getAccessTokenSilently() in App.jsx while also using redux in protected components.
In case it's useful to anyone, here is how we implemented  without withAuthenticationRequired to avoid infinite loops :
const ProtectedRoute = ({
  component: Component,
}: {
  component: React.ComponentType;
}) => {
  const AuthMetadata = useSelector((state: State) => state.auth.entity);
  const { isAuthenticated, isLoading, loginWithRedirect } = useAuth0();

  useEffect(() => {
    // * Useful later to know company in the generic callback page
    // * see getCompanyFromUrlORLocalStorageRedirectUrl();
    if (!isAuthenticated) {
      localStorage.setItem(
        LOCALSTORAGE_ENUM.REDIRECT_URL,
        window.location.href,
      );
    }
  }, [isAuthenticated]);

  const {
    returnTo = `${window.location.pathname}${window.location.search}`,
    onRedirecting = (): JSX.Element => <></>,
    loginOptions = {},
  } = {
    loginOptions: buildLoginOptions(AuthMetadata),
  } as WithAuthenticationRequiredOptions;

  useEffect(() => {
    if (isLoading || isAuthenticated) {
      return;
    }
    const opts = {
      ...loginOptions,
      appState: {
        ...loginOptions.appState,
        returnTo: typeof returnTo === 'function' ? returnTo() : returnTo,
      },
    };
    (async (): Promise<void> => {
      await loginWithRedirect(opts);
    })();
  }, [isLoading, isAuthenticated, loginWithRedirect, loginOptions, returnTo]);

  return isAuthenticated ? <Component /> : onRedirecting();
};```

It would be awesome to have a hook helper like useLoginRedirectEffect() to do somethings like this

const ProtectedRoute = ({
  component: Component,
}: {
  component: React.ComponentType;
}) => {
  const AuthMetadata = useSelector((state: State) => state.auth.entity);
  const { isAuthenticated, isLoading, loginWithRedirect } = useAuth0();
  const options = {
    loginOptions: buildLoginOptions(AuthMetadata),
  };
  const beforeRedirect = () => {
    if (!isAuthenticated) {
      localStorage.setItem(
        LOCALSTORAGE_ENUM.REDIRECT_URL,
        window.location.href,
      );
    }}
  const [ onRedirecting ] = useLoginRedirectEffect(options, beforeRedirect)
  return isAuthenticated ? <Component /> : onRedirecting();
};


test: set up cypress component testing and test footer

Description
This PR configures Cypress component testing and adds a component test for the footer, as well as testing the footer from the home page end-to-end test.
It also adds a custom cy.validateLink() command based on the needs of this project and does a few other little things, full details are below.
Note: it reuses the same test code for the component test and the end-to-end test to aid in refactoring - the test code does not depend on any details of the underlying UI library.
Related Issue

Resolves: #122

Motivation and Context
Chad mentioned to me that there were some open issues for Cypress tests targeting various components. In a Next.js application there are some components and ideas that are best to test from end-to-end, but many tests can be written at the component level just fine.
Detailed notes on changes (in order of the diff)

Add some Cypress things to .gitignore
set up component testing config for Next + webpack
run validateFooter in homepage e2e test
define custom validateLink command

might be too much? but comes in handy

add component-index.html, this is where Cypress mounts component tests
add component.ts - support file for component testing with custom mount command to wrap components in ChakraProvider so they have what they need when rendering
add validateFooter helper function for desktop/mobile footer testing

this is DOM-only, framework/library agnostic test code, works in e2e and component tests

update scripts for Cypress in package.json, and update cypress
add footer.cy.tsx - component test that tests the footer at different sizes
add some data-cy attributes to the footer to make it easy to separate mobile and desktop

I think a better solution would be to only have one set of elements in the DOM at a time, currently the layout that is not being shown is hidden with CSS only, so we need to do this little workaround

changes to tsconfig.json to make Cypress related files work correctly with TS

I haven't noticed any negative side-effects to this change but it would be worth checking out before merging in case some types clash somewhere or something.

How Has This Been Tested?

Use the new npm scripts to confirm:

npm run cypress:open:e2e should start you out at the Choose a Browser screen in e2e mode. Pick a browser and open e2e testing, then open homepage.cy.ts and note that it calls the new footer validation util to test the footer
npm run cypress:open:component should start you out at the Choose a Browser screen in component testing mode. Pick a browser and open component testing, then open footer.cy.tsx and note that it also calls the new footer validation util to test the footer in desktop and mobile sizes
npm run cypress:run:e2e should run the homepage e2e test in headless mode, not open the cypress UI
npm run cypress:run:component should run the footer component test in headless mode, not open the cypress UI

Screenshots (if appropriate):
Both testing types of Cypress are configured now:

Component test running for footer on mobile size:

Homepage e2e test also testing footer:

Hey @marktnoonan, there seems to be an issue with the deployment of this branch. The error is:
./cypress/support/component.ts:20:19
--
Type error: Cannot find module '../../src/styles/globals.css' or its corresponding type declarations.

Any idea what could be causing this?

@chadstewart It looks like there is a typescript check that doesn't run by default locally, but runs pre-deployment, TS needed to be told it is OK to import a CSS file, when I run tsc after adding the declaration file it is ok.
On thing I didn't do in this PR is add component tests to the github actions for this project - wanted to make sure you were OK with the addition of the Cypress component tests first.

@chadstewart It looks like there is a typescript check that doesn't run by default locally, but runs pre-deployment, TS needed to be told it is OK to import a CSS file, when I run tsc after adding the declaration file it is ok.
On thing I didn't do in this PR is add component tests to the github actions for this project - wanted to make sure you were OK with the addition of the Cypress component tests first.

Nah, it's cool man. Honestly, you know what you're doing much more than I do in terms of Cypress so I trust your judgement on what should be added and the set up needed.
Only thing I'd ask is could you elaborate a bit more on the why of some of the config? To be honest, this is primarily for me cause I want to learn Cypress and just want to know why I'd want to do some of this stuff in future projects.
Otherwise, pretty much consider yourself Chief Cypress Officer here. 😂


Ruff check and format

I added ruff as a linter and formatter. I tried both black and ruff over the past weeks and i prefer ruff because it's faster, easier and more clear to use i think.
This PR fixes the lint errors and reformats the packages.
https://docs.astral.sh/ruff/
There is no CI check yet, so before committing, please run ruff check then ruff format from the repository root.

Already merged this to develop


fix megengine duplicate value error

fix duplicate value error of megengine  visualization
please help review, thanks~

@YSF-A please share a sample model file to validate this change.


Modify code to reproduce

Hi @PatWalters,
I wanted to try re-running the code in this package. I ran into a few snags:

The code didn't keep track of the ChEMBL version
The file names in the different scripts didn't match up. I used  pystow to create a standard directory and file structure.
The code didn't work - the descriptors weren't saved in the models. I was able to change the LogDPredictor class to use the uru rdkit descriptors to solve this

Overall, I think the introduction of pystow to pick file paths outside of the git directory is really important for reproducibility, since it means you can rely on things being in a certain place (e.g., if you don't run the code from inside the repo's folder, unexpected stuff might happen). Now, you have some nice chembl 31 results, too!

Thank you!


Add readme of how to create all your classes to SFDC

I really wanted to try use your dml mocking framework in my sandbox.
I've tried creating apex classes one by one in these order :

DML classes
TestingUtils
Factory classes
Repo classes

However at step 3 when creating the RepoFactory class, I get this error:
force-app/main/default/classes/factory/RepoFactory.cls  Invalid type: FieldLevelHistoryRepo (8:36)
So I tried creating the FieldLevelHistoryRepo class , I get this error :
force-app/main/default/classes/repository/FieldLevelHistoryRepo.cls  Invalid type: AggregateRepository (1:68)
force-app/main/default/classes/repository/FieldLevelHistoryRepo.cls  @Override specified for non-overriding method: Set<String> FieldLevelHistoryRepo.addSelectFields() (32:42)

So I tried creating the AggregateRepository class, I get this error :
force-app/main/default/classes/repository/AggregateRepository.cls  Invalid type: Repository (1:66)
force-app/main/default/classes/repository/AggregateRepository.cls  @Override specified for non-overriding method: String AggregateRepository.getFinalQuery(List<Query>) (82:29)
force-app/main/default/classes/repository/AggregateRepository.cls  @Override specified for non-overriding method: Set<String> AggregateRepository.addSelectFields() (65:42)

So I'm essentially stuck trying to create all these classes effectively. Help pls?

Is there a better way than creating these apex classes one by one?
Is there a more simplified framework that doesn't involve Aggregation, FieldHistory?

@victorng8 do you have much experience with SFDX? The easiest way to deploy this framework is using the CLI. While I am toying with the idea of creating a base unlocked package (which would allow for the addition of things like field level history and aggregates being provided as plugins), I haven't had the time or chance to take on that work.
It's very uncommon to be deploying or creating classes one by one - and you'll struggle to do so in most any kind of project. I would highly recommend deploying the entirety of the directory using the CLI and then deleting the classes you're not interested in (if applicable).

Oops sorry, I forgot I could deploy folder of all the apex classes in one go via VSCode even without the CLI (Right click folder and Deploy to Source org)

Not to worry, and the vs code extension uses the CLI to do just that!


Allow prefetch with IntermediateM2MField

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

12 of 13 (92.31%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+1.1%) to 81.973%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

import_export_extensions/fields.py
12
13
92.31%

Totals

Change from base Build<PHONE_NUMBER>4:
1.1%

Covered Lines:
1255

Relevant Lines:
1531

💛  - Coveralls


dataset

where can we download DSIFN-CD dataset

Hi @KDSBBS,
I have uploaded my processed LEVIR-CD and DSIFN-CD datasets to the DropBox and you can download the final datasets here:

LEVIR-CD-256: click here to download
DSIFN-CD-256: click here to download

Accordingly, I have updated the readme file as follows:
https://github.com/wgcban/ChangeFormer/blob/main/README.md#links-to-download-processed-datsets-used-for-trainvaltest
It is still uploading and will be available to download after a couple of hours.
I also recommend you to see this issue: https://github.com/wgcban/ChangeFormer/issues/12
Let me know if you have more questions!
Best,
Chaminda

This seems possibly resolved, if so please close the issue.
Otherwise here is a link for the dataset:
https://pan.baidu.com/s/19TRsdPiRym9DoV76DukQ5g -Password: hetu
source:
https://github.com/GeoZcx/A-deeply-supervised-image-fusion-network-for-change-detection-in-remote-sensing-images/tree/master/dataset


Add new subservers package

Note: substitutes part of the work in #442, server status will be added in a different PR

itest need to be added

@guggero: review reminder
@positiveblue, remember to re-request review from reviewers when ready


Sodium 0.5.x is now under conflicts instead of breaks

The reasoning is so painfully dumb that I needed to open a PR for this literal two-line change.
In short, Sodium 0.5.x will no longer stop you from booting, but you'll be warned about a potential conflict in the log file.
This is to undo the damage of another commit that wrongly added it to breaks, meaning it stopped you from booting unless you extracted, edited, and reinserted this file. (You shouldn't have to edit a mod's jar file to make it work properly)

No need to be rude sheesh

Nobody reads logs. Letting them run together will only cause a million new bug reports to be opened.

Nobody reads logs. Letting them run together will only cause a million new bug reports to be opened.

@TropheusJ dude, if it's a flywheel bug just force it to not be used on sodium 0.5.x and above.
it's literally one setting change. (but i dont know how to do it)
(pinging because i know you're just ignoring this now)

That's not really how it works. Disabling Flywheel, while it might help with some issues, always has the chance to introduce worse rendering bugs between Create and Sodium or just Create itself, not to mention the performance loss that comes with disabling Flywheel, which Sodium can barely remedy because it doesn't have the optimizations built in for Create like Flywheel does.
The dev team is stressed because they've been trying to fix everything for BlanketCon23, on top of other IRL shit I'm sure, it'll get fixed in due time.

Disable flywheel and then you get a million issues about miserable performance.
Requiring 0.4.10 for now is the most clear option.

I've made a fix for Flywheel on Sodium 0.5.2, it'll be PR'd shortly.

https://github.com/Jozufozu/Flywheel/pull/200


Gatling library should not bundle logback-test.xml

Gatling should not bundle logback-test.xml file. If Gatling is used in Maven project it's very difficult to correctly override logging configuration if not impossible.
Maven picks up multiple XML configurations and logs are populated with things like:
08:26:35  05:26:35,336 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback-test.xml] at [file:/srv/jenkins/workspace/load-testing_main/src/test/resources/logback-test.xml]
08:26:35  05:26:35,337 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs multiple times on the classpath.
08:26:35  05:26:35,337 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [file:/srv/jenkins/workspace/load-testing_main/target/test-classes/logback-test.xml]
08:26:35  05:26:35,337 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback-test.xml] occurs at [file:/srv/jenkins/workspace/load-testing_main/src/test/resources/logback-test.xml]
08:26:35  05:26:35,466 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
08:26:35  05:26:35,467 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
08:26:35  05:26:35,473 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONSOLE]
08:26:35  05:26:35,482 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
08:26:35  05:26:35,534 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN
08:26:35  05:26:35,534 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONSOLE] to Logger[ROOT]
08:26:35  05:26:35,534 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:26:35  05:26:35,535 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@25bbf683 - Registering current configuration as safe fallback point

It is a general best practice that libraries do not include any logging XML configuration but set it programmatically if needed. Or at least rename a file to something like logback-gatling.xml wrapped into <included> so it can be used as default and overridden if necessary (this is how Spring does it).

Please check your logs properly:
Resource [logback-test.xml] occurs at [file:/srv/jenkins/workspace/load-testing_main/target/test-classes/logback-test.xml]
Resource [logback-test.xml] occurs at [file:/srv/jenkins/workspace/load-testing_main/src/test/resources/logback-test.xml
The logback-test.xml file is NOT bundled in Gatling binaries.
The issue happens because both resources and test-classes are in the classpath.
Please provide a standalone reproducer if you think the issue is in a Gatling component.

Thank you for your response. It looks like I was wrong. Garling indeed doesn't bundle logback-test.xml file.
However, there is a question then, why both src/test/resources and target/test-classes are on the testing classpath? I suppose it is a bug here? https://github.com/gatling/gatling-maven-plugin/blob/31c776c2cb3c24c2cb6ca1a71172dd91f051245b/src/main/java/io/gatling/mojo/AbstractGatlingMojo.java#L86
I don't follow the logic in gatling-maven-plugin but it looks like it adds resources folder to the test classpath depending on if gatling.conf exist or not. That folder can contain all sorts of resource files, including logback XML configuration, so such test classpath modification is wrong IMHO.
I just did a quick reproducible case with https://github.com/gatling/gatling-maven-plugin-demo-java

Clone the repo
Run mvn clean gatling:test . No issues with logback overloading.
Delete src/test/resources/gatling.conf which is basically empty commented out file.
Run mvn clean gatling:test again. This time you will get warnings about duplicate logback XML configuration.

Fix released in 4.3.5.


A2-7-3: Exclude compiler generated functions created from lambdas

Affected rules

A2-7-3

Description
Exclude compiler generated functions created from lambdas from the requirement for documentation.

Fixed by https://github.com/github/codeql-coding-standards/pull/266.


Persist filters and scroll position

whenever browsing for novels filters and scroll position are reset whenever I go back after reading the novel summary.
I would appreciate if this can be fixed/added.
Thanks

Just updated
The filter are persistent now.
please close the issue, thanks


Gateway firmware 1.5.4 unsupported

I updated Xiaomi Mijia smart multi mode gateway to the latest version 1.5.4 on MI home app.
I added Xiaomi door / window sensor 2 in Xiaomi gateway 3 (latest version 2.0.5), which shows that the sensor is unavailable.
So I don't know whether it's caused by my updating Xiaomi Mijia smart multi mode gateway or other problems.

Me too. I can't add aqara T1 module after upgrading the firmware. Later, I added it in the home assistant

А у меня после обновления не включается ZHA, точнее на девайссе то он включается... но интеграция ZHA не видит девайс... буду ковырять дальше. Алекс, я знаю что у вас другие дела, а можете как-либо добавить полный саппорт шлюза датчиков? Динамик простаивает, инструкций по нему особо и нет...

1.5.4._0084 firmware doesn't open port 8888,which causes the ZHA can't work. Any idea ?

1.5.4._0084 firmware doesn't open port 8888,which causes the ZHA can't work. Any idea ?

Downgrade

I found that 1.5.4 silabs_ncp_bt has built-in logger maybe contains some useful logs?
Some logs:
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmgku5ks48xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmgku5ks48xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4106,"edata":"26"}],"frmCnt":98,"gwts":1652375740}
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmul07eg44xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmul07eg44xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4106,"edata":"17"}],"frmCnt":3,"gwts":1652375740}
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmgku5ks48xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmgku5ks48xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4102,"edata":"2003"}],"frmCnt":99,"gwts":1652375741}
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmul07eg44xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmul07eg44xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4102,"edata":"D002"}],"frmCnt":4,"gwts":1652375741}
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmgku5ks48xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmgku5ks48xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4100,"edata":"EA00"}],"frmCnt":100,"gwts":1652375742}
May 13 01:15:50 rlxlinux user.notice <BT>: ble report: blt.3.vmul07eg44xx, "method":"_async.ble_event","params":{"dev":{"did":"blt.3.vmul07eg44xx","mac":"3F:59:C8:53:XX:XX","pdid":1115},"evt":[{"eid":4100,"edata":"FE00"}],"frmCnt":5,"gwts":1652375742}

silabs_ncp_bt help:
usage:silabs_ncp_bt -[pnrtlvh]

arguments:
    -n  --normal           run in normal start mode
    -r  --restore      restrore mode, clean target flash
    -t  --test         test mode(mac: get ble mac, adv: start adv)
    -h  --help         print this help
    -l  --log_level    set log level(0:error, 1:warning, 2:notice, 3:info, 4:debug)
    -p  --port         specify the serial port to connect target(default: /dev/ttyS1)

silabs_ncp_bt startup in launcher.sh:
if [ -e ${MIIO_PATH}wifi.conf ];then
        if [ -e /tmp/debug_mode ]; then
            silabs_ncp_bt -p /dev/ttyS1 -n 2>&1 | logger -t "<BT>" &
        else
            silabs_ncp_bt -p /dev/ttyS1 -n > /dev/null 2>&1 &
        fi
    else
        if [ -e /tmp/debug_mode ]; then
            silabs_ncp_bt -p /dev/ttyS1 -r 2>&1 | logger -t "<BT>" &
        else
            silabs_ncp_bt -p /dev/ttyS1 -r > /dev/null 2>&1 &
        fi
    fi

https://github.com/AlexxIT/XiaomiGateway3/pull/672
use this pull

I found that the latest version 3.0.0 supports 1.5.4, so I upgraded my multi-mode gateway and found that many entities were invalid. Do I need to do other operations? My gateway version is 1.5.4_0090, HA version 2023.1.7

Fixed in latests integration version


Run the extension locally

Hi there.
I'm trying to run the extension locally, but there is no manifest.json.
I tried to create a new one, but there is some permission missing...
Could you provide a manifest sample ?

My bad. It's on the public folder. I got how the extension works.


Several new lines after progress bar

As soon as progress bar starts lots of white space is there before the final log

Config:
var barOpts = {
		width: 50,
		total: fileSize
	};
var bar = new ProgressBar(' Uploading [:bar] :percent :etas', barOpts);

Full Code: https://github.com/arjunkomath/Amazon-S3-CLI-Client/blob/master/index.js

I experienced this when I was generating ticks beyond the total for the progress bar.

I got same problem, in my program, there are several TCP clients, and the number of new lines after the progress bar complete is based on how many clients. If there are 10 concurrent clients, there will be 10 new lines after the progress bar, and if there are 100 clients, there will be 100 new lines, which is very annoy.
✗ npm run simulator -- -h <IP_ADDRESS> -c 10 -n 100

><EMAIL_ADDRESS>simulator /Users/taowang/lab/shui-device/server
> node test/simulator.js "-h" "<IP_ADDRESS>" "-c" "10" "-n" "100"

info: Receiver Server: "<IP_ADDRESS>:7778"
info: Starting 10 devices for 100 requests...
  Sending requests [====================================================================================================]  100% 0.0s

info: Finished.

I might locate the issue, the code in /lib/node-progress.js at line 96
  // progress complete
  if (this.curr >= this.total) {
    if (this.renderThrottleTimeout) this.render();
    this.complete = true;
    this.terminate();
    this.callback(this);
    return;
  }

This part will call the this.terminate() no matter whether it's been called before. And the new line is coming from the this.terminate():
ProgressBar.prototype.terminate = function () {
  if (this.clear) {
    this.stream.clearLine();
    this.stream.cursorTo(0);
  } else this.stream.write('\n');
};

so, if the bar.tick() is called when the bar completed, which is quite common for some async calls, such as  setTimeout() or setInterval(), it will always write a new line. There should be some code to avoid multiple calls to this.terminate().
The current workaround is that manually checking whether the progress is finished or not, and avoid call bar.tick() if the progress is finished already. A single new line may be manually added depends on the case.

I had the same issue, bar.tick() was being called even after the progress is finished. Fixed it 👍
Thanks!


Add perf metrics support for WhisperStaticPipeline

@as-suvorov could you have a look, please?

I think we should add features_extraction_durations as well to be aligned with stateful pipeline. Example


Removing upper bound pin on HDF5

What is the change?
Removing the upper bound pin on the HDF5 library.
Why is the change being made?
We recently had to put a temporary stop-gap in place for the HDF5 library, because its latest release was bugged on Windows. Their team now says this problem is resolved, so I thought it was time to remove this version upper bound on their tool.

Checklist

[x] This PR has only one purpose or idea.
[x] Tests have been added/updated to verify any new/changed code.

[x] The code style follows good practices.
[x] The commit message(s) follow good practices.

[x] The release notes have been updated if necessary.
[x] The documentation is still up-to-date in the doc folder.
[x] The dependencies are still up-to-date in pyproject.toml.

bump


Feed - [Mobile - IOS] - When "Make Surgical Masks: Materials and..." is clicked the content does not fit to page

When "Make Surgical Masks: Materials and..." is clicked the content does not fit to page
Steps to Reproduce:
Pre-requisite: Have a user created to sign in
1 - Using mobile (IOS - I used Iphone 6S and simulator 7, 8, X) - staging.fightpandemics.work
2 - Click on 3 line Sign in/Register -> follow sign in steps
3 - Click on Request help -> Medical> Enter any zip/address -> Skip
4 - On Feed: Scroll down to "Make Surgical Masks: Materials..." -> click on it or View More
Current Results: Content does not fit to page
Expected Results: Content need to fit to page

@dabartlett can you include the link of the post? -> Posts are shown as nearest to my location, and there are a couple thousand so I won't be able to scroll to find this since I am not sure where the location is. Better if you include the post link here. I want to test cuz I think this is being resolved...

I actually fond it since I figured you were looking at posts near Nevada or the US in general, and yeah I tested and this is the same issue as #928, and it is being addressed there. I will close this but feel free to test it on the Build URL of this PR -> https://github.com/FightPandemics/FightPandemics/pull/989

ok - apologies - I see that it is fixed and here is the link just in case: https://staging.fightpandemics.work/post/5eea73164b96a349da599533


Listen to IPv6 address

Hey!
Tried to set reproxy to listen on [::]:8080 (and :::8080 too) address but it doesn't want to start with some errors logged:
reproxy  | 2022/03/31 21:17:03.924 [INFO]  activate http proxy server on "[::]:8080"
reproxy  | 2022/03/31 21:17:03.925 [ERROR] proxy server failed, listen tcp: address "[::]:8080": too many colons in address
reproxy  | 2022/03/31 21:17:03.925 [ERROR] proxy server failed, listen tcp: address "[::]:8080": too many colons in address

Is IPv6 supported in reproxy? Could we implement if it isn't?
Thanks!

Liiks like it's working now:
2023/02/12 15:19:02.784 [INFO]  activate http proxy server on [::]:8888


Microsoft.JSInterop.JSException: Could not find 'TimeGhost.IndexedDbManager.openDb' ('TimeGhost' was undefined)

I know you already have an open ticket in this regard, but I just wanted to reinforce it.
I have version 1.5 of the component and I am using the blazor wasm (net 5) and js is still not found. And I have already configured the following reference:
<script src="_content/TG.Blazor.IndexedDB/indexedDb.Blazor.js"></script>

blazor.webassembly.js:1 fail: VerusDate.Web.Pages.Profile.Index[0]
      ProcessException
Microsoft.JSInterop.JSException: Could not find 'TimeGhost.IndexedDbManager.openDb' ('TimeGhost' was undefined).
Error: Could not find 'TimeGhost.IndexedDbManager.openDb' ('TimeGhost' was undefined).
    at https://localhost:5001/_framework/blazor.webassembly.js:1:1287
    at Array.forEach (<anonymous>)
    at e.findFunction (https://localhost:5001/_framework/blazor.webassembly.js:1:1247)
    at b (https://localhost:5001/_framework/blazor.webassembly.js:1:2989)
    at https://localhost:5001/_framework/blazor.webassembly.js:1:3935
    at new Promise (<anonymous>)
    at Object.beginInvokeJSFromDotNet (https://localhost:5001/_framework/blazor.webassembly.js:1:3908)
    at Object.w [as invokeJSFromDotNet] (https://localhost:5001/_framework/blazor.webassembly.js:1:64218)
    at _mono_wasm_invoke_js_blazor (https://localhost:5001/_framework/dotnet.5.0.0.js:1:190800)
    at do_icall (https://localhost:5001/_framework/dotnet.wasm:wasm-function[10595]:0x194e46)
   at Microsoft.JSInterop.JSRuntime.<InvokeAsync>d__15`1[[System.String, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at TG.Blazor.IndexedDB.IndexedDBManager.<CallJavascript>d__28`1[[System.String, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at TG.Blazor.IndexedDB.IndexedDBManager.OpenDb()
   at TG.Blazor.IndexedDB.IndexedDBManager.EnsureDbOpen()
   at TG.Blazor.IndexedDB.IndexedDBManager.<GetRecordById>d__21`2[[System.String, System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[VerusDate.Shared.Model.ProfileModel, VerusDate.Shared, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null]].MoveNext()

I'm encountering the same issue.  I have a "standard" Blazor WASM application that I added Blazor.IndexDB to and added openDB etc to the "Counter" page, I have buttons that Open the DB and that add 100 records to the DB.  This app is blazor wasm (net 5) and it works with no issues.
On another project, which is part of a solution that has an API project as well I've duplicated the code and added the open DB etc to the Counter page again, however, this time, when I first click the button to open the DB I get the error and it asks for  Reload, opening Developer Tools I see the error 'TimeGhost' was undefined.  If I click the Reload and then try the OpenDB button all works and I can add the 100 records.
The new WASM app in the solution I have added JWT tokens (6.8.0) , Blazored.Toast (3.1.2), BlazorInoputFile (0.2.0) and BlazoredLocalStorage (3.0.0).

@drma-dev Did you solve this issue?  I was unable to find any commit for it...

FIXED: I had the same issue and unable to find answer on any post but but try & error I conclude that as I was playing with multiple version of my app maybe my browser was mixed-up but in any case when I did CTRL-F5 to clear cache it fixed my problem

Unfortunatel, ctrl-f5 didn't fix it in my scenario--that was one of the first things I tried...

I am using .net6 blazor wasm. At first I had an issue with the JSInterop when using Reshisru.Blazor.IndexedDb,Framework. Then I installed this package found here https://github.com/wtulloch/Blazor.IndexedDB I got the issue 'TimeGhost' was undefined so I just did what @jeanpoulin have done and it fixed the issue.


Correct typo

take('abcde`, 3)

should be
take('abcde', 3)

@joechung-msft : Thanks for your contribution! The author, @ecfan, has been notified to review your proposed change.

@joechung-msft Thanks for contributing! Looks good, so signing off. #sign-off


[TESTS] Actions Helpers

Pull Request Test Coverage Report for Build 898

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
5 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-3.4%) to 67.698%

Files with Coverage Reduction
New Missed Lines
%

src/EconomicStrategy/EconomicStrategyHelpers.ts
1
57.69%

src/Scanner/Scanner.ts
1
78.38%

src/Actions/Pending.ts
3
49.04%

Totals

Change from base Build 893:
-3.4%

Covered Lines:
410

Relevant Lines:
578

💛  - Coveralls


Test OWSLib PR allowing POST for WFS getfeature requests

Description
We need to make WFS requests for info on many subregions. For a large number of regions, the GET requests exceeds the maximum size. See https://github.com/geopython/OWSLib/issues/439
Francis made a PR https://github.com/geopython/OWSLib/pull/706 to solve this, and we'd need a review of the PR, as well as confirmation that it solves our issue, illustrated in https://pavics-sdi.readthedocs.io/projects/raven/en/latest/notebooks/Subset_climate_data_over_watershed.html#Subsetting-a-gridded-climate-dataset
https://github.com/Ouranosinc/raven/blob/master/docs/source/notebooks/gridded_data_subset.ipynb
The notebook sets the number of features to 2, but it should work with 100s of features.
TODO:

[x] Local install of PR branch
[x] Add method="POST" to flyingpigeon.subset_base.get_feature
[ ] Create unit test checking that it works with large number of features (that would crash GET requests).
[ ] Confirm PR is functional for this test case - report success in OWSLib PR comments
[ ] Wait for PR to be merged and new version of OWSLib released
[ ] Make new flyingpigeon release
[ ] Update flyingpigeon version in raven requirements

If I recall correctly, I had established that there was a problem via a very large request run in a Jupyter notebook. If I can find the issue where I raised this and the notebook/environment where I established the problem existed, it'll make things much easier to formally establish that the issue is fixed.
I'll have some time available to put some work into this as we near the Symposium.

@f-PLT
Thanks for your work on getting POST support into OWSlib. I'm pinging you on this thread as this problem is RAVEN-centric.
I seem to not be grasping the differences between GET and POST request formatting. Here's a snippet from a request formatting function in a PR I'm currently working towards:
    wfs = WebFeatureService(GEO_URL, version="1.1.0", timeout=30)
    if method.upper() == "GET":
        try:
            resp = wfs.getfeature(
                typename=layer,
                bbox=coordinates,
                srsname="urn:x-ogc:def:crs:EPSG:4326",
                method=method,
            )
        except Exception as e:
            raise Exception(e)
    elif method.upper() == "POST":
        if parse(ows_version) < parse("0.20.0"):
            raise NotImplementedError("POST requires OWSlib >= 0.20.0")
        try:
            resp = wfs.getfeature(typename=layer, bbox=coordinates, method=method)
        except Exception as e:
            raise Exception(e)
    else:
        raise NotImplementedError("Method %s not implemented" % method)

Does the `WebFeatureService().getfeature() method work for POST queries? I've combed your PR and haven't been able to find anything suggesting that it wouldn't work, but I seem to only be receiving this cryptic error when the method is POST:
owslib.util.ServiceException: Could not determine geoserver request from http request org.geoserver.platform.AdvancedDispatchFilter$AdvancedDispatchHttpRequest@577bc7f

Any advice?

Yes, I did the implementation for Post in 1.1.0 and 2.0.0.
I tested with a few public WFS services, with both versions, but I did have this kind of error when testing on Boreas (https://boreas.ouranos.ca/geoserver/wfs)
An ordinary Post request to that address with an xml filter worked well, but through OWSLib, that error always popped up with Post.
It has to do with this part : https://github.com/f-PLT/OWSLib/blob/wfs-getfeature-post/owslib/feature/init.py#L314
Basically, I took this part from the Get section of getfeature(), which takes the url from the GetCapabilities. In the case of Boreas, that address is http://boreas.ouranos.ca:80/geoserver/wfs which works for method=GET, but not with POST. I tried ordinary Post requests to that specific address and got the same kind of error as before (and yours). I gather it's an accessibility issue, as the unsecured address is not reachable outside of it's deployment network, and there's apparently no redirection to https. Might be related to how POST requests work compared to GET requests.
If I got rid of the url finding cited above and went with base_url = self.url, it would then work, but then lost the automatic redirections.
All this is from memory, I'll take an other look at it on Monday to confirm things, and I could also give you the python script I used while implementing this.

def get_feature(url, typename, features):
    """Return geometry from WFS server."""
    wfs = WebFeatureService(url, version='2.0.0')
    resp = wfs.getfeature([typename], featureid=features,
                          outputFormat='application/json',
                          method="Post")
    return json.loads(resp.read())

def test_get_feature():
    url = "https://pavics.ouranos.ca/geoserver/wfs"
    typename = "public:USGS_HydroBASINS_lake_na_lev12"
    feature_pat = "USGS_HydroBASINS_lake_na_lev12.{}"

    #features = [feature_pat.format(i) for i in range(67088, 67449)]
    features = [feature_pat.format(i) for i in range(67088, 67089)]
    get_feature(url, typename, features)

Fails with
Traceback (most recent call last):
  File "/home/david/src/flyingpigeon/tests/test_subset.py", line 10, in test_get_feature
    get_feature(url, typename, features)
  File "/home/david/src/flyingpigeon/flyingpigeon/subset_base.py", line 18, in get_feature
    method="Post")
  File "/home/david/src/OWSLib/owslib/feature/wfs200.py", line 324, in getfeature
    u = openURL(url, data, method, timeout=self.timeout, headers=self.headers, auth=self.auth)
  File "/home/david/src/OWSLib/owslib/util.py", line 230, in openURL
    raise ServiceException('\n'.join([t.strip() for t in serviceException.itertext() if t.strip()]))
owslib.util.ServiceException: Could not determine geoserver request from http request org.geoserver.platform.AdvancedDispatchFilter$AdvancedDispatchHttpRequest@3b972d1a

@f-PLT Making this test function pass is our use case.

See https://stackoverflow.com/questions/24952960/how-to-correctly-request-a-geoserver-wfs-via-post

I see this header is already set in the code... stumped.

Is it possible that the URL to the fes namespace is missing ?

Got this stack-trace server-side from docker logs geoserver that looks like the same error as in https://github.com/Ouranosinc/raven/issues/297#issuecomment-731432190:
23 Nov 15:31:12 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Could not determine geoserver request from http request org.geoserver.platform.AdvancedDispatchFilter$AdvancedDispatchHttpRequest@67147182
        at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:622)
        at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:258)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:436)
        at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:177)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
        at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
        at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
        at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
        at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:84)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
        at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
        at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:152)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:87)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:48)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:509)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1104)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2445)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

I understand that OWSlib should be working with any instance of GeoServer that uses the proper standards, but I just want to raise a point here that our current instance is quite old (http://geoserver.org/release/2.9.3/). Do we know with certainty that this isn't a bug with POST in the build?
I'll try doing a quick look over the advancements in the past few years to see if this is covered anywhere. Otherwise, perhaps the three of us can chat sometime?

When I implemented the Post method, I ran accros this problem, but only while testing with Boreas (the only Ouranos Geoserver I tried).
I had no problems with other Geoservers, either external or locally deployed.
The only way I got it to work was to sacrifice this bit of code : https://github.com/f-PLT/OWSLib/blob/wfs-getfeature-post/owslib/feature/init.py#L312-L320
If I made it use the "outside" address https://pavics.ouranos.ca/geoserver/wfs, meaning only having this line instead https://github.com/f-PLT/OWSLib/blob/wfs-getfeature-post/owslib/feature/init.py#L320, it then worked with Boreas, but lost the ability to be redirected by what OWSLib got back from the GetCapabilities, which I gather could be a problem for other servers.
The url given to OWSLib is https://boreas.ouranos.ca/geoserver/wfs, but the GetCapabilities of the WFS is pointing to http://boreas.ouranos.ca:80/geoserver/wfs, which is restricted (and usually redirected?). It seems to the be the case with Pavics  : https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0
The same code is present for the Get method, but it does not interfere with the request.
I'm not familiar enough with how requests work and the finer points between Get vs Post requests, but my guess is it could be that the Post request does not handle redirection (or if it's possible to redirect it in the context of how OWSLib handles requests, which is outside of what I implemented).
It would also be consistent with this: https://stackoverflow.com/questions/31620987/redirecting-http-post-requests-to-https-post-requests#:~:text=You cannot redirect.,3. and https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
From the above, to redirect a Post request seems like something we should avoid, so the ideal solution would be for the Geoserver's GetCapabilities to list the secured address.
However, I'm pretty sure we rely on Geoserver being unsecured locally so it can interact with the other services. Maybe new flag in getFeature() to force the use of the provided url, instead of searching for the provided one in the GetCapabilities? Not sure how good a design that would be though.

I can check into it a bit more with David B. this week, see if he has a better idea.

I just want to raise a point here that our current instance is quite old (http://geoserver.org/release/2.9.3/)

Maybe we can take this opportunity to address https://github.com/Ouranosinc/pavics-sdi/issues/183?  Since right now we are unable to test Geoserver on other PAVICS instance than Boreas meaning if we perform this upgrade, where are we going to test the upgrade?

The url given to OWSLib is https://boreas.ouranos.ca/geoserver/wfs, but the GetCapabilities of the WFS is pointing to http://boreas.ouranos.ca:80/geoserver/wfs, which is restricted (and usually redirected?). It seems to the be the case with Pavics : https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0

Looks like https://github.com/bird-house/birdhouse-deploy/issues/14 a Twitcher/Magpie bug or new feature needed.

It would also be consistent with this: https://stackoverflow.com/questions/31620987/redirecting-http-post-requests-to-https-post-requests#:~:text=You cannot redirect.,3. and https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
From the above, to redirect a Post request seems like something we should avoid, so the ideal solution would be for the Geoserver's GetCapabilities to list the secured address.

I've read those 2 links and agree that redirecting POST is probably a bad idea.  The summary is POST is usually not an idempotent action (ordering pizza, casting a vote, ...) so should not be repeated (a redirect is like repeating the POST).

However, I'm pretty sure we rely on Geoserver being unsecured locally so it can interact with the other services.

I assume other services can be reconfigured to use the secure httpS connection.  httpS is pretty much the norm now.

Maybe new flag in getFeature() to force the use of the provided url, instead of searching for the provided one in the GetCapabilities? Not sure how good a design that would be though.

If the flag is optional, then why not?  You are not introducing any breaking changes and not forcing it on users either.

I assume other services can be reconfigured to use the secure httpS connection. httpS is pretty much the norm now.

I agree 100%

If the flag is optional, then why not? You are not introducing any breaking changes and not forcing it on users either.

Well, short term, yes, I agree. It's also a really quick fix.
Just to be sure we're talking about the same thing, the flag would need to be in OWSLib's getFeature function
From a design perspective, I'm wary of simply adding a flag to solve this problem.
It's not a problem internal to OWSLib. The Post request, the HTTP protocol and getCapabilities (albeit with an url that is being redirected to httpS) are doing what they're suppose to do. The problem is (most probably) on our end with the way we've configured Geoserver.
Introducing a flag argument in a public module for a very specific use case like this seems, well, out of place? I wouldn't like to see, down the line, 10 flags or more for different things unrelated to the core of the WFS standard.
Just to be clear, I'm simply voicing my concerns here. At the end of the day, we do have time constraints and I'll implement what I'm told to implement.

I'll also be double-checking again tomorrow to make sure a standard post request works with the Resquests library, and also without the url fetching done in OWSLib.

I agree with @f-PLT on not using owslib to fix issues with our own infrastructure.

@f-PLT
@huard  and I had checked over your examples the other week, and everything seemed fine when using the Requests library. For the purposes of Raven, I could potentially patch in a workaround based on your code if the priority is getting Raven up and running sooner.

Maybe we can take this opportunity to address Ouranosinc/pavics-sdi#183? Since right now we are unable to test Geoserver on other PAVICS instance than Boreas meaning if we perform this upgrade, where are we going to test the upgrade?

I can throw some time into that issue this week if it's in the interest of the greater project development. For the HTTPS configurations, not quite my tempo. I know that if we can modernize the version, the data management side of the GeoServer would be much simpler for additional instances. This is a bit out of scope in this thread but let me know if there are other aspects of GeoServer that could use a helping hand.

There is no need for a quick fix on this.
Questions:

Does updating geoserver fix this ?
If not, where is the relevant configuration for this ?

There is no need for a quick fix on this.

Good to know. If there's something else that is within Raven that needs help with, ping away. Shall I focus on https://github.com/Ouranosinc/pavics-sdi/issues/183 then ?

Hitting directly geoserver http://pavics.ouranos.ca:8087/geoserver/wfs?request=GetCapabilities&version=1.1.0 (only works inside Ouranos firewall), I am getting proper
<ows:Operation name="GetCapabilities">
<ows:DCP>
<ows:HTTP>
<ows:Get xlink:href="http://pavics.ouranos.ca:8087/geoserver/wfs"/>
<ows:Post xlink:href="http://pavics.ouranos.ca:8087/geoserver/wfs"/>
</ows:HTTP>
</ows:DCP>

Not sure why behind Nginx and Twitcher/Magpie https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0, we end up with the  wrong http protocol and the wrong port
<ows:Operation name="GetCapabilities">
<ows:DCP>
<ows:HTTP>
<ows:Get xlink:href="http://pavics.ouranos.ca:80/geoserver/wfs"/>
<ows:Post xlink:href="http://pavics.ouranos.ca:80/geoserver/wfs"/>
</ows:HTTP>
</ows:DCP>

Will attempt to put Geoserver directly behind Nginx see if the error persist.  If yes then it's Twitcher/Magpie and most likely relate to https://github.com/bird-house/birdhouse-deploy/issues/14

Digging further, geoserver is actually not behind Twitcher/Magpie, only behind Nginx.  It's the same Nginx for everything else and the usual headers for performing front proxy is there.  Will dig into why geoserver is miss configured.
As for why geoserver data is not protected behind Twitcher/Magpie, it is a little weird since Magpie already have the config to protect it https://github.com/bird-house/birdhouse-deploy/blob/cc4ca20665e5eeb3d3ef51e1588521f4b42bef4d/birdhouse/config/magpie/providers.cfg.template#L74-L88 This predates me starting at Ouranos so should we enable this protection or it's not protected by Magpie for a particular reason?  Someone remember?

Does placing it behind Magpie complicate testing for CI ensembles/notebooks? Will users/services need authentication in order to perform WMS/WFS requests if GeoServer is behind it? I can't recall if I was ever privy to this discussion (it sounds like something my predecessor would have covered).

There is no need for a quick fix on this.
Questions:

Does updating geoserver fix this ?
If not, where is the relevant configuration for this ?

For this issue, I don't think updating geoserver will do much. However, I did try Kartoza's latest geoserver image and the setup/configuration process has been streamlined. (env file and no need for JAVA_OPTS in the docker-compose)
If we need to add the plugin for NetCDF to Geoserver, it could be a good opportunity to update at the same time. It's not a priority right now though.
As for the configuration, I found this, which could be helpful : https://docs.geoserver.org/2.9.3/user/configuration/globalsettings.html#proxy-base-url
For version 2.9.3, I think it would have to be managed by the Tomcat properties, like the the max heap size https://github.com/kartoza/docker-geoserver/tree/2.9.4#setting-tomcat-properties

As for why geoserver data is not protected behind Twitcher/Magpie, it is a little weird since Magpie already have the config to protect it https://github.com/bird-house/birdhouse-deploy/blob/cc4ca20665e5eeb3d3ef51e1588521f4b42bef4d/birdhouse/config/magpie/providers.cfg.template#L74-L88 This predates me starting at Ouranos so should we enable this protection or it's not protected by Magpie for a particular reason? Someone remember?

It's also before I started working for CRIM, I'll bring it up at our next standup.

Update on this.

As for the configuration, I found this, which could be helpful : https://docs.geoserver.org/2.9.3/user/configuration/globalsettings.html#proxy-base-url

This works but it's a manual UI change.  I'd rather have everything automated.  This would be the fall back if nothing else work.

For version 2.9.3, I think it would have to be managed by the Tomcat properties, like the the max heap size https://github.com/kartoza/docker-geoserver/tree/2.9.4#setting-tomcat-properties

This is not working.  Not sure if I implemented it wrong or our geoserver do not support it (we have a custom docker build "inspired" from kartoza build).
My attempt: https://github.com/bird-house/birdhouse-deploy/commit/fddbd1efc50c686c471546178fe32c8a5fa57aae (not found too many documentation about configuring tomcat properties, I might get this wrong).
Our custom geoserver docker build: https://github.com/bird-house/birdhouse-deploy/blob/cc4ca20665e5eeb3d3ef51e1588521f4b42bef4d/birdhouse/docker/geoserver/Dockerfile
I'll spend the rest of the day finding alternative.  Else are we okay with the manual change?  I can do the manual change right now on Boreas for you guys to continue testing.

Does placing it behind Magpie complicate testing for CI ensembles/notebooks?

Should not, Thredds is being Magpie and we have e2e notebook test using data from Thredds.  Only 1 notebook test Thredds with authentication all the rest are accessing Thredds publicly.

Will users/services need authentication in order to perform WMS/WFS requests if GeoServer is behind it?

I guess yes, if some path is protected.  Else, like the current Thredds, everything is currently public so no auth required.  In the current state (not behind Magpie) everything is public as well.

I can do the manual change right now on Boreas for you guys to continue testing.

I just did the manual change.
https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0 returns
<ows:Operation name="GetCapabilities">
<ows:DCP>
<ows:HTTP>
<ows:Get xlink:href="https://pavics.ouranos.ca/geoserver/wfs"/>
<ows:Post xlink:href="https://pavics.ouranos.ca/geoserver/wfs"/>
</ows:HTTP>
</ows:DCP>

This is what we want?
In the mean time, I'll keep looking for an automated way to set it.

That looks right. I can try my hand at testing the POST capabilities using @f-PLT's scripts to get confirmation!

This just pass for me
https://github.com/bird-house/flyingpigeon/blob/7b7a5ce1dda2cde93db34667368ac224a75d58cc/tests/test_subset.py#L1-L10 (branch test_post)

Seems to now be working perfectly for me as well:
url = self.urls["pavics"]
version = "1.1.0"
wfs = WebFeatureService(url, version)

result = wfs.getfeature(typename="public:CanVec_WaterBodies",
                            maxfeatures=5,
                            outputFormat="application/json",
                            method="post",
                            )

Gives me:
{"type":"FeatureCollection","totalFeatures":128796,"features":
[{"type":"Feature","id":"CanVec_WaterBodies.1","geometry":null,"properties":{"bbox":[-125.1331,64.7883,-117.4586,67.0415]}},
{"type":"Feature","id":"CanVec_WaterBodies.2","geometry":null,"properties":{"bbox":[-90.4187,46.4087,-84.6029,48.3055]}},
{"type":"Feature","id":"CanVec_WaterBodies.3","geometry":null,"properties":{"bbox":[-116.9824,60.8071,-108.9008,62.9534]}},
{"type":"Feature","id":"CanVec_WaterBodies.4","geometry":null,"properties":{"bbox":[-87.6073,41.7607,-84.7217,46.095]}},
{"type":"Feature","id":"CanVec_WaterBodies.5","geometry":null,"properties":{"bbox":[-89.58,46.4996,-84.355,49.0172]}}],
"crs":null}

Excellent !
Trevor, we'll likely need this feature for the raven distributed model configuration we spoke about today, so this is great.

Edit: Interesting that I don't receive the geometry though... Might be more to this...
Edit2: WFS @v2.0.0 gives me the geometry. The fix seems to be exactly what we need!

By adding propertyname=None, you'll get it to work.
This is an unintended behavior, caused by a preset here : https://github.com/geopython/OWSLib/blob/master/owslib/feature/wfs110.py#L223
It looks voluntary, and works for a Get request, but for 2.0.0, it's defaulted to None : https://github.com/geopython/OWSLib/blob/master/owslib/feature/wfs200.py#L227
I'll see if I can fix it without breaking something else!

Well... turns out it looks like there's a problem with propertyname even with Get. I'll investigate and create an issue on OWSLib (or might as well just address it myself since my hands are in the code).
In the meantime, I'll fix my part of the code so the default propertyname="*" in 1.1.0 does not interfere with requests.

I got the automated way working but GetCapabilities returns https://pavics.ouranos.ca:443/geoserver/wfs (there's 443 in there).
Pretty sure it's still working, my previous https://github.com/bird-house/flyingpigeon/blob/7b7a5ce1dda2cde93db34667368ac224a75d58cc/tests/test_subset.py#L1-L10 still pass.
So just for your info if you notice something weird.  It's live on Boreas right now for testing ... sorry I do not have another server with proper test data.


 Potential interference from Optifine/Shaders? Otherwise bug with 1.12.2.

try and remove optifine and see if that fixes it.

I made a github account just to say I can reproduce this with just using lb photo realism reload on 1.12.2 and so to protect my map from 1.7.10 I don't use mapwriter currently


Addition of a simple Abstraction code in java

What does this PR does
Issue solved: #82
This PR introduces a simple example of abstraction in Java by implementing an abstract Bank class and extending it with multiple subclasses, each representing a different bank with its own get_balance() method.
The Bank class is defined as an abstract class containing an abstract method, get_balance(), which is implemented in the concrete subclasses Bank1, Bank2, and Bank3. Each subclass provides its own balance value, which is displayed when the method is called.
Key Changes:
Abstract Bank Class:
Created an abstract class Bank with the get_balance() method declared but without a specific implementation.
Concrete Bank Subclasses:
Implemented three subclasses: -
Bank1 with a balance of 1000
Bank2 with a balance of 1500
Bank3 with a balance of 2000
Main Class (abstract_bank):
A main method to instantiate each bank and display their individual balances using the overridden get_balance() method.
Purpose:
This code serves as a foundational example of abstraction, showcasing how different subclasses can implement the same method in their unique way. It improves modularity and enforces a consistent interface across different bank types.

Please add a  hacktoberfest-accepted label to this PR!
Thank You

Thanks! @shivam-sharma-03 for contributing to this repo you can also go through other repo of my acc and contribute according to your expertise & pls share it with your friends to participate and enjoy the hacktoberfest 2024.
Happy coding / happy hacking🎉✨


Adding this to KoolKits?

Hiya!
I've caught this repo from Observability News by Michael Hasuenblas - great stuff:)
I wonder, since this requires 0 code changes, whether it will be a good fit to bundle into our OSS project KoolKits.
We have docker debug images specifically geared towards debugging applications written in a specific language - you can check out the project here.
This might fit into our Go KoolKit.
Your comments much appreciated @edeNFed :)

Hi @TomGranot,
KoolKit looks like a really cool project!
Adding auto instrumentation to the Go image should be pretty easy, probably just copying the binary from this Dockerfile:
https://github.com/keyval-dev/opentelemetry-go-instrumentation/blob/master/Dockerfile
Will be happy to open a PR if you think it's a good idea.

@edeNFed Please do! That would be awesome.
If you can also add it to the README for the Go KoolKit with a quick explainer I'll appreciate that too:)

Sure no problem, I'll let you know once I finished adding to KoolKit so we can update the README file.

@edeNFed Dope. Talk soon!

Hi @TomGranot,
I tried to include the project in the KoolKit container but I had a few issues.
The main problem is that in order for this instrumentation to run we need to add privileges to the pod, for example:
          securityContext:
            runAsUser: 0
            capabilities:
              add:
                - SYS_PTRACE
            privileged: true
          volumeMounts:
            - mountPath: /sys/kernel/debug
              name: kernel-debug

I could not find a flag in kubectl debug which allows adding securtityContext or volume mounts.
I do see that kubectl debug has a "node mode" which allows the creation of pods that runs with enhanced privileges.
What do you think? Should I try to use this mode or do you have other suggestions?

@edeNFed Interesting.
To add what I know off the top of my head, kubectl debug has two modes - container-in-existing-pod and container-in-new-pod.
If I understand correctly, you were attempting to target the latter - copying a pod and debugging it. I'm pretty sure that if you use that you indeed can't change the securityContext, so using 'node mode" seems like the right path to go.
Let me know if I can help out with something!

@edeNFed Any luck testing this out?

@edeNFed NP! Just let me know if i can help.


Title fixed on two pages

Related Issue
Fixes:  #1949
Description
Changes in title according to page name.

Type of PR

[x] Bug fix
[ ] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
Before :

After :

Checklist:

[x] I have made this change from my own.
[x] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

@anuragverma108 @huamanraj @RishabhDhawad @RitiChandak Please check it and merge !

@huamanraj Please also provide labels.

@huamanraj Please also provide labels.

as gssoc guidelines there is no label for basic changes!


[Incremental] JENKINS-55523: Auto-populate Project ID in new dropdown when possible

Apologies for the PR size, I recommend looking by commit. The majority of this change is in testing, boilerplate, and a new client for retrieving the list of project IDs from the Cloud Resource Manager API.
The reason that this is a dropdown is that as far as I can tell it is not possible to do two crucial things:

Fill a f:textbox using query parameters and have that play nice with the Publisher.
Set up a new handler (i.e. onchange) for the c:select (credentials dropdown) tag when that is selected.

I suspect that doing this would require forgoing jelly altogether and writing our view a different way (i.e. JS/HTML/CSS, groovy).
I tried to minimize the impact that this change had on other elements of the KubernetesEnginePublisher and associated tests and view but I faced a few complications when testing, both through unit tests and in the browser:

When the credentialsId is selected, this then changes the "credentialsId" query parameter used when filling the project ID items. Unfortunately, the newly selected "projectId" query parameter is not then updated in the call to check credentialsId unless the credentialsId dropdown comes before projectId's

There is a similar story for Zone. The automatically selected "projectId" query paramater is not updated in both the call to fill zone items and to check zone, unless the Zone dropdown comes before the projectId dropdown.

The behavior of getComputeClient() in the descriptor is only technically correct in tests (before this change) because the descriptor is a new object for every test. In a future PR I will bring the usage of the ComputeClient in line with that of the CloudResourceManagerClient so that doFillZoneItems and doCheckZoneItems construct their ComputeClient using the ClientFactory from getClientFactory in the compute client.

In order to make a spy for ClientFactory in the CloudResourceManagerClientTest, I need both a zero argument constructor and a "makeClientFactory" method. The reason I need a spy is that unlike the ComputeClient, the only arguments involved are passed through the constructor of ClientFactory (the credentialsId), that then allows me to mock the behavior of listing service account projects depending on the value of the CredentialsId.

The implication of avoiding the dropdown is that we don't get any functionality to auto-populate the project ID. It's pretty convenient once you select a credential that everything else (implemented so far) falls into place.

One question, I'm not sure what you mean here:
"2. Set up a new handler (i.e. onchange) for the c:select (credentials dropdown) tag when that is selected."
Isn't doFill<...> called again for each of the elements after a selection has been made? Perhaps I'm misunderstanding.

The issue that is referring to is adding custom javascript which references "onchange" directly, or adding something like "<c:select onchange='...'>". The way that the view is generated from the jelly simply doesn't allow it unless I'm missing something (I tried a bunch of different ways to do this with no impact on resulting view). doFill... works as intended but only applies to dropdowns, not textboxes.

We'll need to add enabling the Resource Manager API as a pre-req to the wiki docs, link: https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=397699472102

Rebased onto develop, hopefully that fixes the windows build issue.

Also updated the wiki: https://wiki.jenkins.io/pages/diffpagesbyversion.action?pageId=165582379&selectedPageVersions=10&selectedPageVersions=9


Increase taskbar area + Horizontal server list

Awesome theme BTW, love it.
The taskbar area in windows is really small, it's difficult to left-click and drag.

Also, it would be awesome if you add Horizontal server list. after all, it was also made by you.

The dragging area is 9px which should be plenty.

Also I am well aware that I made Horizontal Server List and while making SoftX I did know it wouldn't be compatible with HSL as it changes the server list too much. Without extracting the server list code to a separate import there's no possible fix.


Modify alpha release to be more similar to prod release

:tada: This PR is included in version 4.15.0-alpha.1 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 4.15.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.15.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


prepare v0.9.0-rc1

Signed-off-by: Sven Dowideit<EMAIL_ADDRESS>

just waiting for the docker push's


There are no picks to be exported!

After picking data in refrapick GUI, I often get the error "There are no picks to be exported!". I can see the picks I have made in 'View observed traveltimes', and I can 'show/hide travel time picks from other files' but when I try to save the pick file it says there are no picks to be exported. I have tried it on a windows 10 and Mac machine and get the same error. Any suggestions would be appreciated!

Include a topography txt file in the GPS folder of your working data (even if the topography data is arbitrary), when you save the pick file proceed with the option to add topography, and then you will have the option to save the pick file in another folder - this worked for me!

Hi I also have this problem, is there a solution? I tried adding a topography.txt file, which initially solved the problem, but now after some time and a few seismic records later I am getting the same "There are no picks to be exported!" error and I can't save the pick file. Where does refrapy save the pick data before the .sgt (pick file) is created? Any help appreciated.

Hi there,
I had the same issue and I found out that it is not possible to export picks as long as at least 1 pick is present in each .dat/.sgy file.
Hope that helps.

Hi there, I had the same issue and I found out that it is not possible to export picks as long as at least 1 pick is present for each .dat/.sgy file. Hope that helps.

This did help, thanks!


Multiple Reverse Proxy Paths

I am not able to add multiple reverse proxy paths.
I have routing like /api -> http://api-server:80 and /log -> http://log-server:80
How can I add multiple revers proxy paths?

Any news on this? Would be very useful!

#99


Bazel 8 compatibility for rules_{gzip,bzip2,xz,zstd,curl,squashfs,diff}, ape and download_utils

Upstream modules now have multiple Bazel version testing in the CI now.
#3260

We don't have Mac/Windows coverage upstream...yet. The team is working on getting them sorted.
I'll work through the regressions for ape/rules_curl regressions.

There are two (what look like) infrastructure failures:
  | 2024/11/27 14:42:34 Downloading https://releases.bazel.build/8.0.0/rc4/bazel-8.0.0rc4-linux-x86_64...
  | 2024/11/27 14:42:36 could not download Bazel: failed to download bazel: failed to download bazel: could not copy from https://releases.bazel.build/8.0.0/rc4/bazel-8.0.0rc4-linux-x86_64 to /var/lib/buildkite-agent/.cache/bazelisk/downloads/_tmp/download3159232164: stream error: stream ID 1; INTERNAL_ERROR; received from peer
  | Command '['bazel', 'info', 'output_base']' returned non-zero exit status 1.
  | 🚨 Error: The command exited with status 1
  | user command error: The plugin docker command hook exited with status 1


Code split

Sidebar and map has been split
transitions has been added
mobile responsive page

Thank you @laxmanpokhrel  for organizing the code. I like the animations.
Can you please resolve the conflict first, for that pull the main branch.
And, also in the small device sidebar is not working.

@rowheat02 My bad brother 😐, will do it for sure.


Add/remove IPs dynamically

I need a way to modify the list of IPs that are being watched while the server is running. I don't think this is currently possible.
I was thinking that there could be a handler for checking if an IP address is valid. It could just return a bool for accept or reject. Ideally I could just hit my own cache to get the list of valid IPs instead of the internal one. I'll be using a Redis server I think. I think I just need to implement HandlePacket on my own.
If I'm venturing too far off your vision of this project, I can always write this myself, or even submit a pull request if you want. This library (not including the other 2) is pretty manageable.

Also, just looking for your advice on this as you've used this in production more than I have.

This is something that I have intended to implement, but havent gotten around to yet. I think the best option would be to just create an interface for the packet handler lookup. Then it would be easier to for example load the configuration from a db.

Something like this maybe https://github.com/vforteli/Flexinets.Radius.RadiusServer/commit/3001583459ad421821df24d82f5b5340d66ef399

Well I am in no rush :)
It is merged now and the package is updated on nuget