HTTP 301 redirects can't be followed

I'm trying to use Crabler to do some basic web scraping and I've run into an issue where 301 redirects aren't followed. I can't implement a way to handle this in my own code, because the redirect URL from the response isn't exposed to the on_response callback. An alternative implementation would be to change this in the Crabler::event_loop when other HTTP status codes are checked and to handle the redirect transparently. For my application, I think either approach would work, but given that Download implements similar functionality, I think implementing a similar approach for 301 redirects would make sense.

Im fine with having follow_redirect as an option in crabler somewhere. Crabler uses surf underneath (https://docs.rs/surf/2.3.2/surf/), but i havent looked into how surf client can be forced to do redirects. PRs are welcome :P

I will definitely need this functionality, so I'll try to take a stab at it. I expect I should add this as an additional field to crabler::Opt, I'm thinking of a default value of true would make sense. I would expect a webcrawler that has this level of indirection would automatically follow redirects. Do you think that's the right default?

An alternative implementation would be to expose the redirection URL and leave it up to someone's custom implementation of on_response to deal with following any redirections, since it's possible to add pages to be queried through the navigate method. That's more boilerplate for a user that "just wants to scrape all pages", but a bunch more flexibility.
If I implement it with automatically following 3xx redirects, should I hide those responses from the on_response handler if one exists? Or still expose them there? Exposing them there gets into an ordering issue, so I guess it'd make sense to run the user's callback then follow the redirect.

Yes, default value in Opts is how I would approach this! Crawler should follow redirects, unless specified otherwise for the sake of convenience i think.


Expose whether a given event is a real network request or a synthetic fetch during load

Describe the problem
There is not currently a good way to know whether a given event corresponds to a real request coming in to the server, or a synthetic request from load's fetch (or event.fetch) being called for the same origin, which just calls handle without generating a real network request.
This is important to be able to know in my particular use case of having an API gateway that sites in front of both the SvelteKit app and some external API. load fetch requests from the browser can be set to the same origin, but these requests during SSR get sent to handle and I need them to go out to the real API. Right now, I'm doing this when the URL path matches a given regex - however, this could easily match things that aren't really API requests, and if these are ultimately mapped back to the SvelteKit app, this results in an infinite loop.
Describe the proposed solution
A new flag on the event object (say, event.isSyntheticRequest) makes sense to me here.
Alternatives considered
For right now, my plan is to add a dummy x-is-ssr-api-fetch header to the appropriate requests as part of my handleFetch hook, and then in my handle hook look for that header, remove the header, and deal with the request appropriately.
Importance
would make my life easier
Additional Information
We already have event.isDataRequest. I'm not entirely thrilled about the word 'synthetic' here, but I do think some name of the form event.is*Request makes sense here.

I was just taking a peek at what might be involved in implementing this, and I realized I wasn't clear on what the desired behavior was, exactly. Should the event that's passed to handleFetch also have isSyntheticRequest: true (or whatever we call it)? (How would we know whether a given request should be marked as such? Maybe that's an argument for not setting it on any of them.)
Or should respond() in respond.js take a new parameter/option (passed in to it by create_fetch() in fetch.js) that tells it when it creates its event object, it should set isSyntheticRequest: true on it? Or something else?

Looking at this again, maybe we already have what we need with state.depth? Maybe respond() can just construct the event object with isSyntheticRequest: state.depth > 0?


GRIF-72, GRIF-91, GRIF-173 provide searching on provides

Internal query services now leverage the ?provides_name= and ?re_provides_name= url params in component registry enabling direct searching on any components child dependencies.
Note - depends on a specific component-registry-bindings branch (pip install -e git+https://github.com/RedHatProductSecurity/component-registry-bindings.git@corgi-818-test#egg=component-registry-bindings
)
Note- depends on corgi deployment (to stage and prod)
Tests will fail until both of the above are addressed

@JakubFrejlach made more changes ... let me know if we merge


Week 4 of M102 and M101P creates error

Hi,
I tried to download the videos of the MongoDB University for courses M102 and M101P this week and got errors and the download didn't complete. Command used was:
-d path -u my_email --cw --hq
Then I selected the respective course, the chapter(week) and All videos from this chapter. This works fine for chapters/weeks 1 to 3 but the script doesn't finish for week 4. Is this a known issue? Shall I post the error output I see in the command prompt of Windows 8 ? What other details do you need?
Thanks
Dmr

Just downloaded entire Chapter4: Replication from M102 without any issues. There are sometimes issues with youtube so you might try to run it again later on and let me know the outcome.
P

This should be fixed now just bump it up to latest mongo-edu v0.2.7.


Register devices on the internet

Hi, I have been playing around with resin.io and came across Eliot.  Looks like a great project you have going.  I must be missing something obvious,  from reading through the docs I was trying to work out how to deploy a specific pod to a specific device.  I could not see where you specify which device you want to deploy to.
"eli get devices" gets the devices on your network
"eli get pods" lists your pods
"eli create pod --image eaapa/hello-world testing" deploys to all devices listed in the "eli get devices" ?  How can you specify what devices you deploy to and what if the devices are not on your local network ?

Hi @grendo!
At the moment Eliot manages only single device at a time.
If there are multiple devices, in the network, the eli create pod gives error message that you need to specify the device with --device flag.
I'm thinking of how to make the device selection easy, but at the same time make the getting started as easy and smooth as possible. All ideas are welcome!


Adds support for custom context path on Jira Server

Can't wait for it anymore <3 !

@MartinBoernertVW Sorry I left this abandoned for so long—I have no defense. But I've cleaned it up, am merging it now, and submitting the update to Streamdeck!


UA Prefix

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description
The Terraform provider should offer a UA Prefix provider parameter. This would be used in Terraform modules as a way to identify the module as the API consumer. This could also be used by providers like Pulumi, Kubeform, and Crossplane which extend the TF Provider and are innacurately identified as "Terraform" agents today.

For example: https://github.com/linode/terraform-provider-linode/pull/40/commits/1d6bb71e4d3efac46af78589c11afbd008397ffc
New or Affected Resource(s)

provider

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

References

#0000

Great find @ocobleseqx! Let's do that and incorporate it into the UA, like the Google provider.
I'm not sure if we'll have the ability to define Provider Metadata for Pulumi and Crossplane in the same way, but we can address that separately (and potentially do both: a UA prefix and a Provider Metadata module_name).

Noting that this has been implemented as:
terraform {
		provider_meta "equinix" {
			module_name = "namehere"
	}
}

/ok-to-test


Record compliance events on the compliance events history API

If a policy controller sends a Kubernetes event with the policy.open-cluster-management.io/policy-compliance-db-id annotation set. The compliance event will be recorded on the compliance events history API.
Also, when the policy is disabled/deleted, a compliance event is recorded on the compliance events history API.
Note that to enable this functionality, the --compliance-api-url must be passed to this controller.
Relates:
https://issues.redhat.com/browse/ACM-6889

/hold
In case others want to take a look and have comments.  Otherwise remove the hold.
/lgtm


PIC-1070 Create PUT Preference Endpoint

In general, we also have spring boot integration tests which use H2 as the database. We have them in court case service and community API. I think we ought to have them here too.

Closing as this PR has been superseded


Handling permission denial to camera and microphone

If the user denies access to camera and microphone when the app is launched for the first time, the app enters a limbo state.
Is there a way to detect if user granted/denied access?

This is not an SCRecorder feature. You should detect that prior to opening the View that is using the Recorder. Check this SO answer: http://stackoverflow.com/questions/20464631/detect-permission-of-camera-in-ios

Thanks for the clarification!


feat(charts): add generated readme to kargo

The first part of #718. Refactor the Helm values file comments to support the bitnami-labs/readme-generator-for-helm tooling. I have manually generated the readme to confirm it works. The next steps will be to automate the generation on releases and update a section in the docs with the content.

Comments have been converted from multi-line to single-line to support the @param syntax.
I've done some minor refactoring to the contents of the parameter descriptions, mainly to take advantage of markdown syntax for the readme.
Some values have been uncommented instead of defaulting in the templates (e.g. the api.replicas was commented out but is now set to 1 the default`) so that they appear in the value column of the readme.

The only difference in the generated manifests using the default values compared to before these changes are to the admin account hash, which is due to the default value being empty quotes instead of placeholder text.
% diff helm-before.yaml helm-after.yaml
73,74c73,74
<   ADMIN_ACCOUNT_PASSWORD_HASH: $2a$10$5HiAsllXy3QjNAkjYChJSOPPD.z70o3eqL2CZ9/R0C8hIYzN.rw/.
<   ADMIN_ACCOUNT_TOKEN_SIGNING_KEY: "<signing key>"
---
>   ADMIN_ACCOUNT_PASSWORD_HASH: $2a$10$d1zy3B4xt65fjlaFwcRmeevGPp6ASl1X4p1lBrHjV32yXWC43vyvK
>   ADMIN_ACCOUNT_TOKEN_SIGNING_KEY:
732c732
<         secret/checksum: b425f7bd873326434d6e96c0ab9831ce25cd2e8c1f07a7c2355b4532ecf23cea
---
>         secret/checksum: 97bbedb624d101c522d2990c4f83cc2595d0ddbf1c6c63dfddbd0afbf8e65dc9

The readme can be generated using % bash hack/helm-docs/helm-docs.sh, assuming the user has already done npm install -g @bitnami/readme-generator-for-helm. This shouldn't be of much consequence as the process will be automated in the near future.

It would be cool if we added a github action that checked if the helm chart changed at all in a PR, and if so it would run this command to regenerate the readme file, and also copy the readme over to the website docs to keep them up to date. This way it is fully automated and we don't have to do anything.

It would be cool if we added a github action that checked if the helm chart changed at all in a PR, and if so it would run this command to regenerate the readme file, and also copy the readme over to the website docs to keep them up to date. This way it is fully automated and we don't have to do anything.

That's my plan for the next steps. But I wanted to break down the change into smaller pieces. First, get the readme and script in. Then I'll work on the action to automate it.

The changes look fine to me; the only concern is that it makes it harder to read the values file. Should we add a comment at the top of the values file to have them reference the README file for easier-to-read documentation, or do you think that is so obvious it doesn't need to be said?

It depends on where the user lands. If they go looking for the chart (like I normally do for something like Argo CD) and they land in the chart folder they'll likely stop there and use the readme. If they go to inspect the value (helm inspect values) it won't be so obvious. To account for that case, I'll add a comment to the values directing to the readme.

I promise I'll get around to looking at the actual values.yaml changes on Monday. The README that is generated from this is very nice though. 🚀

@morey-tech are you able to deal with the merge conflicts?

I resolved merge conflicts, but I bet I've gotta run code/doc gen now...


error  in<EMAIL_ADDRESS>

Describe the bug
I just installed the Notification module and I got this
error  in<EMAIL_ADDRESS>Module parse failed: Unexpected token (334:34)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|         pauseTimeout() {
|             if (this.pauseOnHover) {

            this.timerControl?.pause();

|             }
|         },
@ ./src/main.js 13:0-52 28:8-21
@ multi (webpack)-dev-server/client?http://<IP_ADDRESS>:8080&sockPath=/sockjs-node (webpack)/hot/dev-server.js ./src/main.js
To Reproduce
Steps to reproduce the behavior:
npm install --save @kyvg/vue3-notification
Add dependencies to your main.js:
import { createApp } from 'vue'
import Notifications from '@kyvg/vue3-notification'
const app = createApp({...})
app.use(Notifications)
Add the global component to your App.vue:

Config

vue: ^3.0.0 => 3.0.11
@kyvg/vue3-notification: ^2.2.5 => 2.2.5

I think it can be fixed by babel-plugin-proposal-optional-chaining plugin in 2.2.x version. So I change target to es6 and there are no more optional chaining in 2.3.0 version


add new maintainer to this repo

hi @ocefpaf / conda forge colleagues. Can we please add @nkorinek as a maintainer for this repo? i am not sure how to do that. many thanks

@conda-forge-admin, please add user @nkorinek

@lwasser https://github.com/conda-forge/earthpy-feedstock/issues/15#issuecomment-931605110 should trigger a PR and the current maintainer can merge it to add the new person.

@ocefpaf THANK YOU so so much. i will look for the PR and will merge it.

@ocefpaf THANK YOU so so much. i will look for the PR and will merge it.

The webservice may be slow. If you are in a hurry you can send a PR to add the new person's GitHub handle here:
https://github.com/conda-forge/earthpy-feedstock/blob/master/recipe/meta.yaml#L56-L59

OK. Looks like I need to do that from a new issue. See https://github.com/conda-forge/earthpy-feedstock/issues/16 and #17

closing this as i just merged #17 ! many thanks!!


Readme css var update

Fixing issue : #1337 (comment)
https://github.com/airbnb/react-dates/pull/1341#pullrequestreview-152672709

@ljharb Sorry I opened up a new PR, the last one closed after I reverted.

Coverage remained the same at 84.875% when pulling d136c69102d6bb4f6afe4e3d430856a18cd2a735 on dumblole:readme-css-var-update into ee92846ffa50b3d1fea6e016013c265dd6e9dc44 on airbnb:master.

Will this be accepted? If not, please close this issue

Anything else that's wrong?


UAR-1421 Implementation of full API validation for Remove

JIRA link
https://companieshouse.atlassian.net/browse/UAR-1421
Change description

The Remove validator class can now also perform full validation
It checks that both a filing date is NOT present and the statement field has a value of true
Containing data blocks (update and remove) must also be present
Refactored the OverseasEntitySubmissionDtoValidator class to also perform all the Update checks for Remove
Unit tests added, amended and refactored to ensure all scenarios are covered

Work checklist

[X] Tests added where applicable

Merge instructions
We are committed to keeping commit history clean, consistent and linear.  A BREAKING CHANGE can be part of commits of any type,
types other than fix: and feat: are allowed, for example build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others,
footers other than BREAKING CHANGE: <description> may be provided.

Going to address the non 'TODO comment' code smells now...


Emit part of an attribute value if the user is typing

Fixes https://github.com/dotnet/razor/issues/8734
When typing a component attribute and typing<EMAIL_ADDRESS>the compiler detects "complext content" in the attribute, raises a diagnostic, and skips emitting any C#, which means tooling can't provide any help for the user. The main issue is completion doesn't work, but there are other effects too, like the tooltip on the attribute is different, with one coming from Razor (when the . is present) and one from C# (which has better UX):

This fixes the issue for this specific case where the complex content is a C# expression followed by a ..

@dotnet/razor-compiler for 2nd review, but also please confirm if I should wait for the compiler rename before merging?

if I should wait for the compiler rename before merging

I don't think that's necessary, pausing compiler work will be important while we're working on merging assemblies into one, currently we are only renaming which shouldn't introduce any conflicts with this.

Ping @dotnet/razor-compiler for 2nd review please :D

@dotnet/razor-compiler @333fred @chsienki any change of a review?

lookiung


WEEK 0: company values research

Company: Slack
Core Values (as per their website):

Empathy
Courtesy
Thriving
Craftsmanship
Playfulness
Solidarity


Add access to CORS API

Add Java API to allow setting the CORS configuration from a java-cloudant client.
See Cloudant CORS documentation.

We won't be adding new APIs here, our new cloudant-java-sdk(beta) has operations for CORS configuration.


Replaced deprecated Swift function

index(of:) is deprecated in Swift 5, replaced with firstIndex(of:).
This change is compatible with Swift 4.2.

Thank you for this PR.
I'm going to close this PR though, as there's already one for Swift 5 support that is a bit more extensive, and keeps compatibility with Swift 4.

Thanks for the update. I realized later there was another PR for Swift 5, so I apologize for wasting time but I hope the other PR will be merged soon :-)


[WIP] Post migration actions

After the migration to redhat-ai-dev org, we need to update a few mentions and links inside the content of this repository.

WIP as I'm currently updating the docker-push.yaml section here.

WIP as I'm currently updating the docker-push.yaml section here.

Removing WIP as it was addressed in: https://github.com/redhat-ai-dev/ai-lab-helm-charts/pull/17/commits/3419c34e46e91a3276a7a11cde8b4e5b333a4bea

@johnmcollier @Jdubrick FYI as https://github.com/redhat-ai-dev/developer-images/pull/8 is now merged and the application_gitops is part of developer-images I've removed that part from the current repo.