Allow stripping query string parameters from Analytics

Trello: https://trello.com/c/LiAOrW5F/1000-remove-or-redact-values-at-the-end-of-postcode-look-up-string-that-dont-match-postcode-format
This adds a new meta tag approach to stripping PII data from being sent
to analytics. This allows for particular query string parameters to be
redacted and for an application to specify them.
To use this an application should create a meta element of:
<meta name="govuk:static-analytics:strip-query-string-parameters"
content="param-1, param-2">

Where param-1, param-2 is a comma separated list of parameter names.
Formatting for this is inspired by <meta name=keywords>
Then when analytics data is sent to GA the strings are looked for a
match of the query string parameter and any found are replaced. To illustrate:
were a url of https://www.gov.uk?param-1=secret be sent to analytics,
and param-1 was flagged for stripping, then this was be replaced to be
https://www.gov.uk?param-1=[param-1].
There are a few caveats of this approach that should be outlined:
It's just a regular expression that tries to match query strings and this
is done rather crudely. It just looks for a part of a string which is
?parameter_name=<some value> or &paramater_name=<some value>, so it is
possible that for some strings this could be a false positive. A more
thorough approach could be to parse URIs that are recognised in input
and break them down to constituent parts to identify query strings, but
it isn't clear this is needed.
This won't match a URL that is urlencoded - for example & - this
feels like something that would occur if we were showing someone
publicly the URL they used - which seems a bit strange for PII.
As the means to specify query string parameters is a comma separated list,
this technique will not support any query string that use a comma in the
parameter name. This seemed pretty low risk to me, were we to want to support
this we may want to switch to a JSON encoded approach to defining parameter
names to strip.
An application of this can be seen in: https://github.com/alphagov/collections/pull/2202

@andysellick thanks for the review. Is there anything useful I can do about keeping this in sync with the components gem?

@andysellick thanks for the review. Is there anything useful I can do about keeping this in sync with the components gem?

@kevindew I'll copy this change across into the gem. I'm pretty close now to being able to make the full switch and retire this code in static. I might ping you for a review 😄

@kevindew I'll copy this change across into the gem. I'm pretty close now to being able to make the full switch and retire this code in static. I might ping you for a review 😄


EIP-747- wallet_watchAsset

There should be a way to do a "wallet_watchAsset" request for each provider.

I found a way to do it.

@rinahorde Can you describe the solution you used for wallet_watchAsset


batch breaks on channels tw cannot see

if thread watcher is not allowed VIEW_CHANNEL perms in a category which is being used in the /batch command the whole command has a fit and breaks.
I should fix that

fixed in #6245f5024a97f9eb06225e009f6f58244c0f7867


PANIC: 1 -> Query can't return single id on Array field

Hi Prisma Team! My Prisma Client just crashed. This is the report:
Versions

Name
Version

Node
v12.18.3

OS
windows

Prisma Client
2.9.0

Logs
prisma-client {
  engineConfig: {
    cwd: '<project root>\\prisma',
    enableDebugLogs: false,
    enableEngineDebugMode: undefined,
    datamodelPath: '<project root>\\node_modules\\.prisma\\client\\schema.prisma',
    prismaPath: undefined,
    engineEndpoint: undefined,
    generator: {
      name: 'client',
      provider: 'prisma-client-js',
      output: '<project root>\\node_modules\\@prisma\\client',
      binaryTargets: [],
      previewFeatures: [],
      config: {}
    },
    showColors: false,
    logLevel: undefined,
    logQueries: undefined,
    flags: [],
    clientVersion: '2.9.0',
    enableExperimental: [],
    useUds: undefined
  }
}

This was fixed in 2.16 and no longer reproducible in 2.22 as well.
I am closing this.


Initial work on SLS Gallery and Virtual Microscope playbooks

Initial work on the playbooks for sls-repo.openmicroscopy.org and learning.openmicroscopy.org.

Nginx: looks like you're using your own omeroweb conf, there's an omero-web role variable to disable the nginx auto-config.
TLS certificates: there's some standard config that's used by most ome webservers that use the wildcard cert. E.g. https://github.com/ome/prod-playbooks/pull/147
Password vars: probably easiest to have them in plain text in the private host or group vars during development, once you've settled on the required vars put them into the separate credentials repo. If possible include a dummy default in the public playbook to facilitate future testing.
omeroweb apps: Do this in two steps. Step one installs one omeroweb with config that's not dependent on other apps. Step two installs the web-apps and their config. The IDR and training playbooks both do this in different ways, one with a web-apps role, the other by installing into the venv directly and creating a config file. Both have molecule tests so we're sure that the ordering and web restarts are correct.
After a quick look I can't see anything in the extra roles that can't be done without, though you're obviously welcome to improve the existing roles to make it easier.

I guess you're not aware that once you include a role it's handlers come into scope. That means in any tasks you can simply notify one of the restart handlers and it'll e.g. automatically restart omero-web of th at particular task changes. They're also coalesced so you can notify on all tasks that change something relevant and the handler will be called once at the end if at least one relevant task changed.

If you want to simplify the webapp install see https://github.com/ome/ansible-role-omero-web/issues/20 for a suggestion

nginx: I do want it to install and configure it somewhat, I just don't want it to be writing omero-web.conf.

Disable it in the omero web role and use either the nginx or nginx-proxy roles for full control.

I don't know if the UoD LDAP configuration is optimal but testing is made difficult by ldap0.dundee.ac.uk:636 not accepting connections from my laptop.

put them into the separate credentials repo

I guess I need access to whichever bit of the repo these should land in or it would also suffice to have somebody else add them for me.

I don't expect my roles to be adopted verbatim alongside the other OME roles but these playbooks remain a useful record of how I currently manage the SLS Gallery and Virtual Microscope.

I don't expect my roles to be adopted as they are as OME ones but these playbooks remain a useful record of how I currently manage the SLS Gallery and Virtual Microscope.

I hadn't got around to looking at the requirements file, I'll do so. How do Galaxy namespaces work, then do they become mtbc.ldap-cert and whatnot again? The omero_web_apps role on Galaxy is probably not mine; is it fairly new? I'll have to compare to see if I can use it instead.

release them to Galaxy under your namespace

Taken this approach, thank you.

Re-tested the latest state of these playbooks with the production variables and things look to be largely working. Leaving @manics to provide a final review as he requested several changes.
Overall, the state of these playbooks is now pretty readable, re-uses most of the existing roles with a few custom roles to be reviewed/rationalized so I am in favor of getting these merged alongside other production Dundee servers.

Following a discussion, the general feeling that the main remaining concern with these playbooks as they stand is the usage of personal fork roles in an organization repository. Given the progress, merging this in order to reduce conflicts as well as making future improvements smaller and more easily reviewable.
The main outstanding item is to get rid of the custom roles in time for the December server upgrade either using task within the playbook or using existing roles like ome.ssl_certificate.

get rid of the custom roles

Now https://github.com/ome/prod-playbooks/issues/216.


HA Cluster connectivity (write access)

[Enter feedback here]
Are there any pointers on how to connect to an HA cluster?  The DNS seems to point to multiple addresses, and we're periodically trying to write to the read-only replica.  Is there a particular DNS we can use in HA to indicate which db is writable?
The examples in here only have localhost....

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e53cbf1a-c62f-e9db-1e44-ee1f7004c0e4
Version Independent ID: ff9a1e3c-3e36-fab8-e226-e2515fc3e8ba
Content: Building the connection URL - JDBC Driver for SQL Server
Content Source: docs/connect/jdbc/building-the-connection-url.md
Service: sql
Sub-service: connectivity
GitHub Login: @David-Engel
Microsoft Alias: v-davidengel

@pontusvision -- Leonardo, thank you for your questions.  Please consider these resources:

SQL Server on Microsoft Q&A
DBA Stack Exchange
Stack Overflow

@David-Engel -- David, please look into this issue.

@pontusvision I think you are looking for the applicationIntent option. More details can be found in Support for high availability, disaster recovery.
Regards,
David

I am closing this issue, as it appears to be resolved.


Fonts are outlines, hard to read

I really like where this project is going, as a refugee from transmission-remote :-)
Unfortunately, both the last release and the preview don't render properly for me on Win 11.  I get fonts that are outlines (more like I can only see the anti-aliasing)
Ideas?

Thank you for giving my project a go, I didn't expect anyone to use this just yet to be honest haha.
From the looks of it you're using "Light mode" rather than "Dark mode" which I hadn't tested at all yet and I think is leading to this odd font behaviour. I had already made several changes which I think resolved this issue to begin with but looking at this screenshot I figured the whole help section can do with a little bit of a revamp.

It should look like the image above now :) in the latest publish version, let me know if it's resolved for you.

Thanks for the fix.  Seems to look normal now.
I assume to change (e.g.,) the transparency of the torrent list it's editing the "Color Test" settings, but the weird font thing is gone from before.

You can ignore the Color test section in settings.
It's there so I can see how the colors interact. And so that I have a reference to the colors available to me by default in the UI framework that's used (Avalonia). Basically the pallet I use to paint the application with :)


Review tests

Tests have gone a bit out of hand. We split the CI into three different types to accommodate the very lengthy LVAE tests, and to minimize the uploads to CodeCov.
We should review all tests to make them lighter, and offload the heavy testing to another platform (e.g. GitLab).
In addition:

generalize np.testing.assert_array_equal
review skipped tests

Maybe we want to split the tests so we have API tests separately to unit tests. Once we're at v1.0.0 we probably don't want to break the API between versions, so we could have tests that just look at function inputs/outputs and warn us if anything changes int a PR.

That's a good point. Given that we expect CAREamics to be consumed by third-party software, that would indeed make sense to make a specific CI for API and integration related tests, that would only run on linux for instance. That way we would consume less resource.


Trouble with Docker and GITLAB_MONITOR_CONFIG

Using the docker image, I set GITLAB_MONITOR_CONFIG to my config json (verified works), but when I browse to localhost, I get the "Before you can use GitLab Monitor, it has to be configured" page.
Here's how I'm running the container:
docker run --publish 80:80/tcp --env GITLAB_MONITOR_CONFIG={...json...} timoschwarzer/gitlab-monitor

I opened an interactive session, ran the commands from wrapper.sh, and verified that /usr/share/nginx/html/config.json was populated with my config before nginx started.
Does this look like a bug, or am I doing something wrong?

I did a docker pull, cleared cookies, and tried again. Now the page spins forever and I see an error in the console:

Looks like it's using default config instead of mine (and mine got jumbled). If I shell into the container, the environment variable looks right to my eye.

Can you manually GET /config.json?

Ah, you're right. Thanks :blush:
I had this in my command line:
PS> docker run --env GITLAB_MONITOR_CONFIG=$(Get-Content 'C:\path\to\config.json') ...

But no quotes ended up in the environment variable.
Escaping " with \" fixed it:
PS> docker run --env GITLAB_MONITOR_CONFIG=$((Get-Content 'C:\path\to\config.json') -replace '"','\"') ...


[Feature] prettify cpm_add_to_package_lock

I've added some prettify to the cpm_add_to_package_lock and cpm_add_comment_to_package_lock.
It's in PR #162

Hey, thanks for the PR!
I'm a bit hesitant as the package-lock was originally intended to be a small auto-generated file, and not rarely be human read or edited, but I do see that this can become more frequent. Also many projects have automatic code formatting enabled, so maybe instead of accepting the arguments in a string value, we should change the way CPMDeclarePackage works to accept the same arguments as CPMAddPackage. That way the auto code formatter would take care of any styling and be more consistent with the user's preferences.

I'm not sure how changing the way argument are pass will help. The prettify was set to help be human readable.
This is the result for unit:
# units
CPMDeclarePackage(units
    NAME units
    GIT_TAG v3.0.0.alpha-2
    GIT_REPOSITORY https://******/GitHub/units.git
    SOURCE_DIR ${CMAKE_SOURCE_DIR}/drivers/units
    OPTIONS
        "DISABLE_PREDEFINED_UNITS ON"
        "DISABLE_IOSTREAM ON"
)

In one project we have ~45 package declare inside package lock, if its not human readable it's hard to check when needed.
The string seems to be correct since if you get an option with space all the data is inside quote. readable
And we dont use formatter for cmake file that's why we decided to pretty the auto-gen instead.
To avoid causing issue to other dev I can add an option to set the human readable auto-gen,CPMHumanReadableLockFile ?

@TheLartians if we add an option to enable the human readable version could it be correct ?

@TheLartians if we add an option to enable the human readable version could it be correct ?

Hey, sorry for the late response I somehow missed your previous answer. After thinking about this a bit, I definitely see the value in a more human readable package lock. However, formatting the code is tricky and can introduce unforeseen bugs. E.g. we would need to specify ALL possible arguments that may get forwarded to FetchContent. For example, your current implementation does not support the URL argument and breaks if you try to add a package using its URL. Perhaps we could change your implementation to append all unparsed arguments as a list it may work, though we would still need to write some tests for it.
My initial thought was to use a well tested linter tool, such as cmake-format, take care of the formatting after generating the package-lock, but unfortunately it doesn't seem to resolve semicolon separated lists.

Hey, sorry for the late response I somehow missed your previous answer. After thinking about this a bit, I definitely see the value in a more human readable package lock. However, formatting the code is tricky and can introduce unforeseen bugs. E.g. we would need to specify ALL possible arguments that may get forwarded to FetchContent. For example, your current implementation does not support the URL argument and breaks if you try to add a package using its URL. Perhaps we could change your implementation to append all unparsed arguments as a list it may work, though we would still need to write some tests for it.
My initial thought was to use a well tested linter tool, such as cmake-format, take care of the formatting after generating the package-lock, but unfortunately it doesn't seem to resolve semicolon separated lists.

I can check to add a way to put unparsed data at the end without formatting and in the future we could add them to the filter, i think for URL i could add it to the oneValueArgs list. I'm not familiar with the unit test used with cpm but I will look at it  to also make some test.

I can check to add a way to put unparsed data at the end without formatting and in the future we could add them to the filter, i think for URL i could add it to the oneValueArgs list. I'm not familiar with the unit test used with cpm but I will look at it  to also make some test.

Yeah as long as all unparsed arguments still remain in the output in a valid way I think it could work. Maybe the algorithm could also be generalised without needing to explicitly list all possible arguments. E.g. if a parameter is in Snake Case (All Caps) it is most likely a named argument and should be at the beginning of a new line.
Regarding unit tests, you can check that existing tests still work by calling:
cmake -Stest -Bbuild/test
cmake --build build/test --target test-verbose

An additional unit test to check the prettified output would also be great.

Yeah as long as all unparsed arguments still remain in the output in a valid way I think it could work. Maybe the algorithm could also be generalised without needing to explicitly list all possible arguments. E.g. if a parameter is in Snake Case (All Caps) it is most likely a named argument and should be at the beginning of a new line.
Regarding unit tests, you can check that existing tests still work by calling:
cmake -Stest -Bbuild/test
cmake --build build/test --target test-verbose

An additional unit test to check the prettified output would also be great.

I base my parsing on the function: CPMAddPackage  but I think in short term we could append the CPM_ARGS_UNPARSED_ARGUMENTS at the end of the list of each item. I think cmake accept that they will be on the same line. After that i can check to do the smart format on them like you told me with the Snake Case.

I base my parsing on the function: CPMAddPackage  but I think in short term we could append the CPM_ARGS_UNPARSED_ARGUMENTS at the end of the list of each item. I think cmake accept that they will be on the same line. After that i can check to do the smart format on them like you told me with the Snake Case.

Let's move further discussion to to PR #162 itself!

Let's move further discussion to to PR #162 itself!


logstash: take any service

[x] Installs cleanly on a fresh build of most recent master branch
[x] Upgrades cleanly from the most recent release
[x] Updates documentation relevant to the changes (n/a)
[x] Rebases cleanly onto the latest master

Since the executors don't have a health check, if anything on the node is unhealthy they'll be taken out. This means we need to specify that we're OK with whatever the health happens to be.

lgtm, tested on AWS


Improve relayer pick mechanism

At the moment we always pick the same validator to relay messages. We need to improve this mechanism to pick different validators, especially if the one selected is failing to relay messages.

We will pick randomly from the top N validators.
On every failed message delivery, the success rate of a validator drops to the base score of 0.5 if it's above 0.5, otherwise it decreases as usual.


use rock for oidc-authservice on main for ckf-1.8

Closes #131
Testing notes: Because this repo's test suite has given some false passes lately, I've also tested this manually using dex-auth's tests and it worked as expected.  Once this is merged and pushed to charmhub, we should re-trigger dex-auth's CI just to demonstrate publicly how it really works

@ca-scribner this PR doesn't update for 1.8, since we have a track/1.8 for the 1.8 release. Please cherry-pick this to the correct branch

also cc @DnPlas who approved


Cargo.toml displaying user password in OpenSUSE (Tumbleweed)

Problem
I tried creating a new rust project using cargo but I've noticed that my user password is displayed next to my email in the "authors" entry of the cargo.toml file. I can't say much about where this problem is coming from, but I suppose that it may be related to some way I've configured the password manager in KDE.
[package]
name = "HelloWorld"
version = "0.1.0"
authors = ["*mail* <*password*>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
...

Steps
1.Creating new project using cargo
2.Checking on "Cargo.toml" file
3.On the "authors" entry my password is displayed
Thank you !
Notes
Output of cargo version:
cargo 1.41.0-nightly (750cb1482 2019-11-23)
Current Platform
Operating System: openSUSE Tumbleweed 20191124
KDE Plasma Version: 5.17.3
KDE Frameworks Version: 5.64.0
Qt Version: 5.13.1
Kernel Version: 5.3.12-1-default
OS Type: 64-bit

The places Cargo looks is documented in https://doc.rust-lang.org/cargo/commands/cargo-new.html.  Can you check those values to see where it is coming from?  It only uses .cargo/config, environment variables, and git configuration. There probably shouldn't be a password in any of those.

I checked .gitconfig and the issue might come from it.
[user]
	name = *mail*
	email = *password*

So finally, not an issue from cargo.
Git has certainly been configured in a wrong way on my system. Guess I'll have to check that out now.
Thank you for helping !


vegeta: 12.11.1 -> 12.12.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/tsenart/vegeta/releases.
meta.description for vegeta is: Versatile HTTP load testing tool
meta.homepage for vegeta is: https://github.com/tsenart/vegeta/
meta.changelog for vegeta is: https://github.com/tsenart/vegeta/releases/tag/v12.11.2
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 12.11.2 with grep in /nix/store/yc7vwfpg8gq8a10ajhd2p7krh9m8g5nb-vegeta-12.11.2
found 12.11.2 in filename of file in /nix/store/yc7vwfpg8gq8a10ajhd2p7krh9m8g5nb-vegeta-12.11.2

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

vegeta

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/yc7vwfpg8gq8a10ajhd2p7krh9m8g5nb-vegeta-12.11.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A vegeta https://github.com/r-ryantm/nixpkgs/archive/0940c3878ce50188676482be0fbaf27283d6db7b.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/0940c3878ce50188676482be0fbaf27283d6db7b#vegeta

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/yc7vwfpg8gq8a10ajhd2p7krh9m8g5nb-vegeta-12.11.2
ls -la /nix/store/yc7vwfpg8gq8a10ajhd2p7krh9m8g5nb-vegeta-12.11.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

vegeta

Maintainer pings
cc @mmahut for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/tsenart/vegeta/releases.
meta.description for vegeta is: Versatile HTTP load testing tool
meta.homepage for vegeta is: https://github.com/tsenart/vegeta/
meta.changelog for vegeta is: https://github.com/tsenart/vegeta/releases/tag/v12.12.0
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 12.12.0 with grep in /nix/store/l77kzmysg1y3ngmi6vhhkmc3yxrsg8h1-vegeta-12.12.0
found 12.12.0 in filename of file in /nix/store/l77kzmysg1y3ngmi6vhhkmc3yxrsg8h1-vegeta-12.12.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

vegeta

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/l77kzmysg1y3ngmi6vhhkmc3yxrsg8h1-vegeta-12.12.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A vegeta https://github.com/r-ryantm/nixpkgs/archive/286c719450c506a11e758274956f659d36005dfb.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/286c719450c506a11e758274956f659d36005dfb#vegeta

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/l77kzmysg1y3ngmi6vhhkmc3yxrsg8h1-vegeta-12.12.0
ls -la /nix/store/l77kzmysg1y3ngmi6vhhkmc3yxrsg8h1-vegeta-12.12.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

vegeta

Maintainer pings
cc @mmahut for testing.

Add a :+1: reaction to pull requests you find important.

@ofborg build vegeta


Advanced recovery - part one

First PR for the advanced recovery proposal.
I'm using a feature/advanced_recovery branch as base so we can review this first part and then work on the rest on a separate PR, as this is already quite big on its own.
Introduces the StateStore as the way stateful operators interact with their own state.
All the state is now stored in the worker global store that keeps the state in memory, and saves the snapshots to a local db.
Snapshots from all the operators are then passed to the revoery section of the dataflow that uses the Backup class to save state segments of each epoch to a durable storage.
This PR doesn't include:

Unloading of the in memory state to the local db to handle larger than memory state
Full resume (on rescale, or on problems with local db for fast resume)
Compaction
Garbage collection

Just pushed a lot of commits that address all the comments. This is almost a full rewrite of the PR, but I think all the comments made sense and it was basically needed.
Since it's a lot of work, some things might have slipped, so let me know if there's something we discussed about that was not included here.
A summary of the various fixes:

Removed StatefulLogicKind in favor of a hashmap that holds PyObjects
Split the previous StateStore into StateStoreCache, which handles the cache and the builders, and LocalStateStore which handle the connection(s) to the state store databases.
Thanks to the split between StateStoreCache and LocalStateStore, recovery is again optional.
The Backup class is now mandatory if recovery is enabled though, given our discussion for 2PC, since we want a single source of thruth for the frontier info, and that source will be the durable backup.
To help with that, I deleted the old NoopBackup default and switched it for a FileSystemBackup. It defaults to writing "durable" state into a "backup" folder inside the script where the dataflow is executed, but it doesn't create the folder on its own (it asks the user to do so if it doesn't exists). This will allow to test recovery easily, and can be used in production if the filesystem where the files are written is itself resilient. It won't be the default for our platform, but it's usable.
compactor is now a single operator, with different implementations for streams of SerializedSnapshot or ClockStreams
Renamed durable_backup operator to backup
Fixed types in RecoveryConfig, removed unnecessary string convertion.
Removed the recovery script. I left the module to export the classes there.
Changed docs references to the recovery module
I also updated the guide. That will need another pass to better explain some concepts, but at least it doesn't give wrong info.
Switched from --recovery-dir to -d/--local-state-dir. For now I left the env var unchanged though, as that will require changes in the helm chart that we'll have to coordinate with esteban
Use Py<RecoveryConfig> rather than RecoveryConfig
Simplified the resume_epoch dataflow, now using a simple map rather than a custom operator.


When user changes account type from worker to something else, they lo…

…se agency affiliations

@nancywong look good now?


Suggest: fmap a $ b ==> a <$> b

Only if b doesn't have a $ in it, and thus doesn't need bracketing.

@ndmitchell this looks like a good first issue; I like the simplicity of the suggestion. Is this still wanted?

@maartenJacobs Excuse me for jumping in 😄 I would say, go for it!

Yep, would be most welcome!

Before I dig in the wrong direction: fmap a  $ b ==> a <$> b is already implemented, right? There's a test in data/hlint.yml. Is this improvement for cases like (fmap (+ 1) . Just) 2 ==> (+ 1) <$> Just 2 then?

Hmm, so maybe this already is implemented. Trying the simplest test case:
Sample.hs:5:13-24: Suggestion: Use <$>
Found:
  fmap a $ b c
Perhaps:
  a Control.Applicative.<$> b c

So it appears it works. Either it was implemented after I created this issue and no one noticed the ticket, or it didn't fire in the example I had because of reasons like old-version, configuration setting etc. Closing this ticket - thanks for looking into it @maartenJacobs !

Looking at other first-time issues, #882 was nearly ready, plus https://github.com/ndmitchell/hlint/issues?q=is%3Aissue+is%3Aopen+label%3A"good+first+issue" has some tagged.


[Bug]: Default branch & Branch Protection do not show any entries the first time we connect an app to git

Is there an existing issue for this?

[X] I have searched the existing issues

Description
For the very first time after we try to connect an app to git, the Default branch & Branch protection fields do not show any entries inside the Settings tab of the git Modal.

Note: This happens mostly when you try to connect an app in a new workspace
Steps To Reproduce

Create a new workspace & then create a new app.
Connect the app with git.
After the connection is successful, got to the Settings tab by clicking on the Open git settings button.

Expected: We should get to see Master as the Default branch & also as the protected branch in the modal.
Actual: Both the fields are empty.
Public Sample App
No response
Environment
Production
Issue video log
No response
Version
Cloud

This issue is now resolved with the introduction of separate settings modal


Support for Dead Letter Queue Configuration

Hi,
As you may know, AWS released support for Dead Letter Queues (see https://docs.aws.amazon.com/lambda/latest/dg/dlq.html)
I may make a PR for this, but it would be a nice feature to have this config option.

Yes, this sounds like a useful feature to add. Please send us a PR!

Hey, here's the PR: https://github.com/gilt/sbt-aws-lambda/pull/28
Please review when you get a chance, thanks!


Ordinal intervals?

People are often tempted to apply an ordinal scale (sometimes implicitly via Plot.cell or Plot.bar) to temporal data. This can be misleading because gaps in the data, such as missing days, are not shown. Also, the ordinal scale can’t know the expected regularity of the temporal data, so there’s no way for it to choose an appropriate tick reduction strategy and tick labels are often overlapping.
But what if you told the ordinal scale the expected regularity of the data, say as a D3 time interval? Then the ordinal scale could explicitly compute the complete domain from the extent, e.g. d3.utcDay.range(start, stop), and show any gaps in the data. The ordinal scale could also floor the data using the interval (d3.utcDay.floor(date)) as a scale transform, such that the input dates are forced to align with the expected interval and thus be present in the domain. The ordinal scale could likely even chose an appropriate tick reduction strategy (e.g., every Sunday), at least if it’s a known time interval, or perhaps using interval.every.
Related #74.

this feels like a great idea! teases out something subtle between "ordinal" and "continuous". like, continuous scales have the nice property that they're ordered, gapless, and linear — and you can have all that, but be discrete!

People seem to overlook that marks imply specific scale types: specifically that bar (and cell) imply an ordinal scale. This makes sense if you consider that Plot has both a more explicit way of specifying scale type (the type scale option) and a more common implicit way (type inference from data).
So, maybe bar and cell shouldn’t force the ordinal scale type. Instead, if you use a quantitative domain, bars could behave like ticks (say, fixed 1.5px width) and cells could behave like dots (say, fixed 3px radius). If you really want bars with quantitative or temporal data, you have to set type = ordinal (or type = band) in the scale definition. (And maybe a bar and cell mark would automatically promote an ordinal scale into a band scale.)
Or to put it another way, when people use Plot.bar with quantitative or temporal data and see bad ticks, they think they need to fix the ticks; but if the bars are instead too skinny, it’s more suggestive that the bar needs a width. And that width can be specified either by adopting a band scale, or by binning with a rect.
I guess I also wonder if Plot.bar could somehow automatically map to Plot.rect + Plot.bin with quantitative or temporal data. That feels a little magical to me, but so did the implicit Plot.stack transform at first and I love that now.

Another thought: I think in many cases we want a rect instead of a bar here. Here’s a helper that works well as a replacement for bar:
function timebarY(data, {x, interval = d3.utcDay, ...options} = {}) {
  const X = Plot.valueof(data, x);
  return Plot.rectY(data, {
    x1: X,
    x2: X.map(x => interval.offset(x)),
    insetLeft: 0.5,
    insetRight: 0.5,
    ...options
  });
}

And then:
timebarY(data, {x: "date", interval: d3.utcHour, y: "count"}).plot()

Here’s another example that I was at first surprised by, but makes sense if you think about it. Say you have a 1-dimensional tick plot to look at the distribution:

Plot.tickX(cars, {x: "Horsepower"}).plot()

And then you realize you can’t tell how many ticks are overlapping because there’s occlusion, so you think, oh, I’ll stack the ticks.

Plot.tickX(cars, Plot.stackY({x: "Horsepower"})).plot()

Then you get confused: Where are the 0.5’s coming from? Why is is stacking down? And it’s because Plot.tickX implies that y is a band scale, so each tick represents the midpoint of the stacked interval, and band scales are reversed by default so they read from the top down rather than the bottom up.
So probably what you wanted instead was a rule rather than a tick. A ruleX has an x-value and a y-extent, which fits stacking perfectly (assuming each data point has a y-length of one).

Plot.ruleX(cars, Plot.stackY({x: "Horsepower"})).plot()

And in fact if you had used a rule from the beginning, it would have looked identicaly.

Plot.ruleX(cars, {x: "Horsepower"}).plot()


Please support configuring environment variables for the cluster-proportional-autoscaler container

E.g. to enable configuring Go runtime with GOMAXPROCS and GOMEMLIMITS environment variables.
Optionally, consider setting GOMAXPROCS and GOMEMLIMITS env vars automatically relative to the configured resources (CPU requests, and memory limits, respectively), when resources are configured.
With current default resource configuration (20m CPU requests and limits) and no GOMAXPROCS configured, there's lots of CPU throttling. Upstream chart doesn't set default resource configuration, so partially issue could be solved with https://github.com/coredns/helm/issues/103 - at least it would move upstream.

I understand your point, but at this point I wouldn't want to make any changes until we get #103 done


Bower support

This is awesome, thanks for this. Can you add support for Bower?

Implemented in 0.5


Update non-windows.md

Added a note about  support for AWS and GCP-native compute
Fixes#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/11189

@JohanFreelancer9 Please copy review

@mjcaparas Please merge this PR. Thanks


Migrate and update the excess property checking docs

Fixes #2240 and #1875
Migrates and does an editing run over the handbook v1 version of the excess property check docs

The lack of documentation for excess-property checks in the v2 handbook seems like a glaring omission. Would be great to see this merged!
As mentioned in #2240, I also think it would be nice to document the fact that excess properties are allowed in object literals if they're spread in with the ... operator.


lightbox: Download manager fails to download from {image_url}.

This is a follow-up to #3123 (and I think the last step of #2618)!
After a user grants EXTERNAL_WRITE_STORAGE permission in order to download an image, this error is thrown (screenshot from an emulator running the debug build):

(@zulipbot add "area: android")
(@zulipbot claim)

Quick note from our web searching just now:
https://developer.android.com/reference/android/app/DownloadManager.html#STATUS_FAILED

STATUS_FAILED
Value of COLUMN_STATUS when the download has failed (and will not be retried).
Constant Value: 16 (0x00000010)

Not a ton of information in that code, but that at least is very likely what the 16 is.

This reproduces only on zulipchat.com, so I bet the issue is that the download isn't being authenticated properly (as documented in #2618).

Posting the corresponding thread on czo here: https://chat.zulip.org/#narrow/stream/243-mobile-team/topic/Android.20auth.20on.20zulipchat.2Ecom

Relaying over information from the conversation in the thread above:

This bug wasn't reproducing on the dev server, meaning that there's some difference between zulipchat.com and the dev server. Specifically, the Django server on zulipchat.com wasn't recognizing an image download request's Authorization header; the server was responding with 401 and logging unauth. On the dev server, the header was coming through fine, and the mobile client was successfully downloading the image.
@timabbott did a bit of investigation into the nginx configuration and found that the API auth headers weren't being passed to any API routes that didn't start with /api (which include /user_uploads). Changes to the configuration were merged as https://github.com/zulip/zulip/commit/ededdc512b2260edfb0b5bc08291ed81f0d0c788.
After the nginx change, the server on zulipchat.com recognized the authentication / logged<EMAIL_ADDRESS>via ZulipMobile) and returned a status code of 302.

So, the authentication issue may have been fixed with https://github.com/zulip/zulip/commit/ededdc512b2260edfb0b5bc08291ed81f0d0c788, now there's one more issue with the redirect. I'll start debugging by just hardcoding the dev environment to always serve a redirect to a fixed image.

Status code 16 occurs when proper authentication is not provided, although I can't find a reliable doc for this.

This issue is now fixed! Thanks to @agrawal-d in #4089.
We also had a previous issue #2115 which in retrospect is pretty clearly the same issue. That one didn't have much detail or any useful labels on it, which is probably why we didn't notice it at the time we filed this one. I've now marked that one as a duplicate of this.


Refactor the data inside the blockchain kv store

Codecov Report

Merging #47 (2484410) into master (5a97c05) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #47      +/-   ##
==========================================
- Coverage   77.98%   77.95%   -0.04%
==========================================
  Files          69       69
  Lines        4397     4391       -6
==========================================
- Hits         3429     3423       -6
  Misses        968      968

Impacted Files
Coverage Δ

src/blockchain/mod.rs
71.17% <100.00%> (-0.24%)
:arrow_down:

src/db/keys.rs
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add support to reverse proxies

We actually generate the AssertionConsumerServiceUrl using the HTTP Request host. This won't work in scenarios where reverse proxies are used, since the request host will be different from the "public" host that users can reach.
We basically need to change the following line
https://github.com/microsoft/SPID-and-Digital-Identity-Enabler/blob/63ef10fe45b82566d90d4fc0101f0a254603be5d/WebApps/Proxy/Microsoft.SPID.Proxy/Services/Implementations/SAMLService.cs#L32-L36
We could use the x-forwarded-host header (https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-6.0) or, eventually, just put the right host in config.

I have an implementation proposal on the support-reverse-proxies branch.
In Program.cs
https://github.com/microsoft/SPID-and-Digital-Identity-Enabler/blob/16796aab0017613f310c20af311a17506d336d3b/WebApps/Proxy/Microsoft.SPID.Proxy/Program.cs#L13-L16
Where ConfigureForwardedHeadersOptions() is
https://github.com/microsoft/SPID-and-Digital-Identity-Enabler/blob/16796aab0017613f310c20af311a17506d336d3b/WebApps/Proxy/Microsoft.SPID.Proxy/Program.cs#L78-L111
Then in the appsettings we can define the ForwardedHeaders section and set all the required options. The ForwardedHeaders are defaulted to All (Which means X-Forwarded-For | X-Forwarded-Proto | X-Forwarded-Host)
https://github.com/microsoft/SPID-and-Digital-Identity-Enabler/blob/16796aab0017613f310c20af311a17506d336d3b/WebApps/Proxy/Microsoft.SPID.Proxy/appsettings.json#L161-L163


Replace all strings with constant values

We need to replace all strings (ex API paths or query names) to constant values

@namkyu1999  Can I take this issue?


Explain how to get bootstrap.cs for local use

Downloading the github sample doesn’t include the bootstrap CSS. The layout CSHTML expects a local copy of the bootstrap CSS file when in “Development”, so I think the docs should be updated to mention this. I had to add the CSS file manually in my project to resolve this.
Location of _Layout.cshtml: /Docs/aspnetcore/signalr/get-started/sample/Pages/_Layout.cshtml

@shahedc  We should just add in all the bootstrap items, to avoid explaining things that aren't SignalR in the doc.

@rachelappel  I updated the .gitignore so it's trivial to update the sample to include wwwroot/lib
see Update SignalR samples to include wwwroot/lib #6433


"kubectl auth can-i" miss documentation for "--as" parameter

What would you like to be added:

When runningkubectl auth can-i -h, make mention that the --as= parameter is possible.
When runningkubectl auth can-i -h, display two examples of the usage of the --as= parameter, one for a normal user, and another one for a service account located in a particular namespace.

Why is this needed:

The --as parameter is missing in the CLI help doc.
The usage of  --as with a service account is not straight forward for a beginner, given that some prefixes should be mentioned before the actual name of the service account (eg. --as system:serviceaccount:namespace:name)

--as is a global flag that does not just apply to kubectl auth can-i that's why you can't find it. Try,
kubectl options

That being said, if the description needs to be improved, one can find where to change over here

There's a pretty detailed explanation of this in the docs https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation.

We can probably add a link to that docs page in the command description or examples.
Something like "This command pairs nicely with impersonation. See https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation."
@chrisnegus want to grab this one?
https://github.com/kubernetes/kubernetes/blob/33aba7ee025dfddcc140d20e9ca353ffd425f1ae/staging/src/k8s.io/kubectl/pkg/cmd/auth/cani.go#L75

/assign orsenthil

I took the suggestions, and made the code change here - https://github.com/kubernetes/kubernetes/pull/103791
The docs itself has an example of impersonation using --as here - https://github.com/kubernetes/website/blob/main/content/en/docs/reference/access-authn-authz/authorization.md#checking-api-access

Thank you all very much for the explanation and the links, and happy that this issue was found valuble and made its way to the official code.
I have less than a year xp on Kubernetes, if my feedback might interest you as a new K8s user, even though --as is a global parameter, all the examples I have seen so far used --as paired withkubectl auth can-i, so that's why I expected to see at least the keyword --as in the CLI documentation of can-i.

/triage accept


Look into how to allow multi-licensed items

Spawned from https://github.com/citation-file-format/citation-file-format/pull/124/files#r632449575
Look into how to allow multi-licensed items. Options are to change to using an array, or to use SPDX license expressions

Related discussion https://github.com/citation-file-format/citation-file-format/issues/105

Assigned @sdruskat because he opened a PR for this already.

#157 merged, closing.


Add, use WITNESS_SCALE_FACTOR constant

I likely missed some constant uses but this is a start.

Codecov Report

Merging #392 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #392      +/-   ##
==========================================
+ Coverage   84.89%   84.93%   +0.03%
==========================================
  Files          40       40
  Lines        7278     7286       +8
==========================================
+ Hits         6179     6188       +9
+ Misses       1099     1098       -1

Impacted Files
Coverage Δ

src/blockdata/constants.rs
85.39% <ø> (ø)
:arrow_up:

src/blockdata/transaction.rs
93.92% <100%> (ø)
:arrow_up:

src/util/bip158.rs
93.09% <0%> (ø)
:arrow_up:

src/consensus/encode.rs
85.37% <0%> (+0.21%)
:arrow_up:

src/blockdata/block.rs
79.85% <0%> (+1.02%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2cba819...2916685. Read the comment docs.


DOC: optimize: add notes for max function evaluation constraint violation

Reference issue

Fix #14818
Ref https://github.com/scipy/scipy/pull/14805#issuecomment-937413098
What does this implement/fix?
As discussed and reported in https://github.com/scipy/scipy/pull/14805#issuecomment-937413098 and #14818,
"TNC" and "L-BFGS-B" support max function evaluation constraint, but these may violate the limit because of evaluating gradients by numerical differentiation.
So, I improved the docs for clarifying the limitation.

This PR looks ok to me. There's always going to be this sort of issue, and I don't think it's really all that solvable short of a harsh abort, which has the potential to lose minimiser progress. I force pushed after a rebase because of the privatisation refactor.


Add Cyclic Sort

Cyclic Sort algorithm is one of the most important algorithms, I think we should add it.

Please assign me this issue.

kindly assign this to me . I have an very efficient algorithm for this problem , i want to contribute on this


feat(loader): Extract mode can work in either extracted CSS or style tag

What kind of change does this PR introduce? (bugfix, feature, docs update, improvement)
feature
What is the current behavior? (You can also link to an open issue here)
When I disable css extracted option by without using the extact-text-webpack-plugin, the final generated sprite url is rendered as background: url("[object object]"); instead of extracted sprite file URL with symbol id at the end.
What is the new behavior (if this is a feature change)?
The extract mode should support the case people don't want the css be extracted, e.g. can use them in style tag.
Does this PR introduce a breaking change?
Nope.
Please check if the PR fulfills contributing guidelines
Checked.

It's about interop with style-loader in extract mode, right? I don't want additional config option, because it can be fixed via esModule: false. I'll think how to detect such cases automatically, but it'll take some time (#196).
@CXHtml in your case it can be fixed by setting esModule: false

I have background: url([object Object]) in disabled extract mode.
    .test
      height: 100px
      width: 100px
      background: url('~assets/images/svg-icons/like-icon.svg')

module.exports = {
  test: /\.(scss|sass|css)$/i,
  use: ExtractTextPlugin.extract({
    fallback: 'style-loader?esModule=false',
    use: [
      { loader: 'css-loader', options: { minimize: env.NODE_ENV === 'production' } },
      'postcss-loader',
      'sass-loader'
    ]
  })
}

Extract model also disabled for all styles.

esModule: false should be set as svg-sprite-loader config option

@kisenka If I disable in svg-sprite-loader:
background: url(<symbol viewBox="0 0 22 22" id="like-icon"><path class="st0" d="M20.9 15.9c…-.3-.4v-7c0-.2.2-.4.4-.4h2.6c.2 0 .<IP_ADDRESS>v7c-.1.3-.2.4-.4.4z" /></symbol>);

@m1neral why you disable extracting when using CSS? What you expect to get in this case?

@kisenka Nothing, I thought HMR would not work in extract mode of some loaders. Thank you, everything is working.

@m1neral you're welcome!


Some SSE instructions

I've implemented some SSE instructions. The llvm.x86.sse.add.ps, llvm.x86.sse.sub.ps, etc symbols were unresolved external symbols, so I had to use the + - * / operators for them.

Thanks so much @kocsis1david!


Class create time

Added visual to see when class gets created in the chat
hours: .5h

should be PR to develop and not master/


Slow to get fresh logs

Hello,
Since I upgraded to logtrail-6.2.3-0.1.27, It seems that logtrail parse all my index content before getting me to the last document (In logtrail screen I saw all my documents like a cat of several files).
Is it because I restared elasticsearch or something like that?
Regards,
Martinho MOREIRA

Nevermind I cleared my cookies and now it's ok.
;)


Update prophecy parser to handle participants

Depends on https://github.com/populationgenomics/sample-metadata/pull/188

Codecov Report

Merging #186 (7eae4ca) into dev (ccfe333) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #186   +/-   ##
=======================================
  Coverage   43.51%   43.51%
=======================================
  Files          57       57
  Lines        4084     4084
=======================================
  Hits         1777     1777
  Misses       2307     2307

Impacted Files
Coverage Δ

api/routes/participant.py
37.34% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ccfe333...7eae4ca. Read the comment docs.


Add action to test code snippets in documentation

As suggested here we should allow to have code blocks that can run as is, to make it easier to check them we should have github action that runs them.
Not sure if we should check the output of the snippets, I think for now we can just check if they run as python scripts :)

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

This might be a tool we could use https://github.com/anko/txm

We have a PR for this :) #3394
Linking it so this issue can eventually be closed


Links in Description are broken.

@hexydec
The clickable links @ https://github.com/hexydec/tokenise#description need updated. They lead to blank pages for me. They seem to be the right links, but the protocol in them loads http, rather than https, and then results in a blank rendered browser page for me.
Just wanted to let you know.

Thanks mate, will fix.


style: abstain icon on proposal votes page

The abstain icon on proposal votes page should use choice-bg color and not border color, and the sign in the circle should also use same color (and not black)

Nevermind this is not relevant actually, this is not abstain, it's unsupported choice from Snapshot


Some dels are uneffective

In the Notebook, you sometimes do del of either a local variable or a function's argument just before return.
IMHO it does nothing there. Both of them are garbage collected exactly there, unconditionally.

I'm not an expert on python garbage collection. This is what I get from ChatGPT:
Is del query Necessary?
In most cases, explicit del statements are not required because Python’s garbage collector (GC) will eventually free the memory for variables that go out of scope. However:
When working with large models or tensors (like in deep learning), waiting for the GC can lead to inefficiencies and OOM errors.
Explicitly calling del gives you precise control over when memory is released, which is particularly important in long-running processes like model training.

Yes but local variables are collected at the return, aren't they?


Exported CZML's id field is blank

Describe the bug
When the layer is exported as a CZML file, its id is blank.
To Reproduce
Steps to reproduce the behavior:

Go to Earth editor
Add a layer
Click "Export" tab
choose "CZML" file type
Export it
Open the file

Expected behavior
The layer should be exported as CZML file without id field

This issue is out of date. So I will close this.


Compile with dotty AND scala

To compile with scala run from the root directory, like usual. To compile with dotty run from the core directory.
cd core
sbt compile
sbt test:run

The lack of structural types caused most trouble, in particular the Closeable type. The best solution I could come up with was to replace Closeable with a CanClose type class, which involved quite a bit of boilerplate.  In turn,ManagedResource lost the nice <: Closeable constraint.

As I proposed several times, if you do disable this do the same for the beginning line in methods (which is preserved). It's all about consistency, I don't care which method is used.


Use the certificate role to create the cert and the key

Introduce a variable ha_cluster_pcsd_certificates to set the certificate_requests.
Add a test test script tests/tests_cluster_basic_custom_pcsd_tls_cert_role.yml

I borrowed the mssql README in https://github.com/linux-system-roles/mssql/pull/125 written by @spetrosi. Thanks, Sergei!
@spetrosi, @tomjelinek, this is another piece of the effort to use other roles in the ha_cluster role. If you could review it, I'd greatly appreciate it.

[citest]

[citest]

[citest]

[citest]

[citest]

The tests_cluster_basic_custom_pcsd_tls_cert_role.yml is failing on my nodes with the following error:
The conditional check '__certificate_provider_vars[__certificate_provider].packages is defined
' failed. The error was: error while evaluating conditional (__certificate_provider_vars[__certificate_provider].packages is defined
): '__certificate_provider' is undefined

The error appears to be in '/root/.ansible/collections/ansible_collections/fedora/linux_system_roles/roles/certificate/tasks/main.yml': line 22, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  block:
    - name: Ensure provider packages are installed
      ^ here
fatal: [rh86-node2]: FAILED! => {"msg": "{% set unique_providers = [] %}\n{% for item in certificate_requests %}\n{%   set _ = unique_providers.append(\n       item.provider |\n       d(__certificate_provider_default)\n     ) %}\n{% endfor %}\n{{ unique_providers | unique }}\n: {% for _cert in ha_cluster_pcsd_certificates %}\n{%   set _ = _cert.__setitem__('name', '/var/lib/pcsd/pcsd') %}\n{% endfor %}\n{{ ha_cluster_pcsd_certificates }}\n: [AnsibleMapping([('name', '{{ __cert_name }}'), ('common_name', '{{ ansible_host }}'), ('ca', 'self-sign')])]: 'ansible_host' is undefined"}

I cannot test this PR. I would like to be able to run the tests before I approve the changes. I'm not sure if the problem is in my setup or elsewhere. Since the CI passes, it may be a problem in my setup. But I was unable to figure out what the problem is and how to resolve it. Any help appreciated.

Hi @tomjelinek, I'm trying to reproduce the problem. Could you please tell me the os of the control host and the managed host? And the version of ansible? When testing locally, we usually use qemu. How about your case? Thanks!

Hi @tomjelinek, I'm trying to reproduce the problem. Could you please tell me the os of the control host and the managed host? And the version of ansible?

Sure. Sorry for not providing them right in the first post.
OS of both control and managed nodes is RHEL 8.6.0.
Ansible version:
# ansible-playbook --version
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Jan 14 2022, 11:04:20) [GCC 8.5.0 20210514 (Red Hat 8.5.0-7)]. This feature will be removed from ansible-core in version 2.12.
 Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ansible-playbook [core 2.11.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Jan 14 2022, 11:04:20) [GCC 8.5.0 20210514 (Red Hat 8.5.0-7)]
  jinja version = 2.10.1
  libyaml = True

Collection version:
# ansible-galaxy collection list
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Jan 14 2022, 11:04:20) [GCC 8.5.0 20210514 (Red Hat 8.5.0-7)]. This feature will be removed from ansible-core in version 2.12.
 Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

# /root/.ansible/collections/ansible_collections
Collection                Version
------------------------- -------
ansible.posix             1.4.0
community.general         5.6.0
fedora.linux_system_roles 1.27.0

# /usr/share/ansible/collections/ansible_collections
Collection               Version
------------------------ -------
ansible.posix            1.4.0
community.general        3.8.9
redhat.rhel_system_roles 1.16.2

When testing locally, we usually use qemu. How about your case? Thanks!

I'm developing and testing the role on qemu VMs which I also use for pcs development. I don't create new VMs for running the role tests.

Thank you, @tomjelinek! At the first glance, I noticed you are using ansible 2.11.

ansible-playbook [core 2.11.12]

Could it be possible to upgrade it to the newer version? (I'd think at least 2.12 is available.)

Thank you, @tomjelinek! At the first glance, I noticed you are using ansible 2.11.

ansible-playbook [core 2.11.12]

Could it be possible to upgrade it to the newer version? (I'd think at least 2.12 is available.)

Sorry, not an issue of the ansible version :p, but of ansible_host. I read "ansible_host is a connection variable, which is not included in hostvars if it is not explicitly declared." The reason it works for the CI is it is declared in the standard-inventory-qcow2...
Here's the debug print with my reproducer.
TASK [redhat.rhel_system_roles.certificate : DEBUG] ************************
task path: /usr/share/ansible/collections/ansible_collections/redhat/rhel_system_roles/role
s/certificate/tasks/main.yml:36
Wednesday 12 October 2022  16:17:03 -0400 (0:00:00.053)       0:00:36.861 *****
fatal: [myself]: FAILED! => {}
MSG:
{% set unique_providers = [] %}
{% for item in certificate_requests %}
{%   set _ = unique_providers.append(
       item.provider |
       d(__certificate_provider_default)
     ) %}
{% endfor %}
{{ unique_providers | unique }}
: {% for _cert in ha_cluster_pcsd_certificates %}
{%   set _ = _cert.__setitem__('name', '/var/lib/pcsd/pcsd') %}
{% endfor %}
{{ ha_cluster_pcsd_certificates }}
: [{'name': '{{ __cert_name }}', 'common_name': '{{ ansible_host }}', 'ca': 'self-sign'}]:
'ansible_host' is undefined

Good news is ansible_hostname does not have the problem. If I replace ansible_host with ansible_hostname, the tests_cluster_basic_custom_pcsd_tls_cert_role.yml test passes. Let me make the change. Thanks for finding it out!

[citest]

Good news is ansible_hostname does not have the problem. If I replace ansible_host with ansible_hostname, the tests_cluster_basic_custom_pcsd_tls_cert_role.yml test passes. Let me make the change. Thanks for finding it out!

Thanks for fixing the issue. Now the role passes the previously failing task. However, it still doesn't work for me. It gets stuck at fedora.linux_system_roles.certificate : Ensure certificate requests task. I found out that it runs the following command on managed hosts and the command gets stuck:
/usr/bin/getcert resubmit -N CN=rh86-node2 -c local -w -f /var/lib/pcsd/pcsd.crt -D  -A  -E  -g 2048 -K  -u keyEncipherment -u digitalSignature -U <IP_ADDRESS>.<IP_ADDRESS>.1 -U <IP_ADDRESS>.<IP_ADDRESS>.2 -U  -B  -C

When I run the command manually, it gives me this error and exits:
Error parsing Kerberos principal name "-u": Configuration file does not specify default realm.

Certmonger version is certmonger-0.79.13-5.el8.x86_64. I don't use kerberos on my VMs. When pcsd certificate and key already exists, the Ansible task says ok and tests pass.

Good news is ansible_hostname does not have the problem. If I replace ansible_host with ansible_hostname, the tests_cluster_basic_custom_pcsd_tls_cert_role.yml test passes. Let me make the change. Thanks for finding it out!

Thanks for fixing the issue. Now the role passes the previously failing task. However, it still doesn't work for me. It gets stuck at fedora.linux_system_roles.certificate : Ensure certificate requests task. I found out that it runs the following command on managed hosts and the command gets stuck:
/usr/bin/getcert resubmit -N CN=rh86-node2 -c local -w -f /var/lib/pcsd/pcsd.crt -D  -A  -E  -g 2048 -K  -u keyEncipherment -u digitalSignature -U <IP_ADDRESS>.<IP_ADDRESS>.1 -U <IP_ADDRESS>.<IP_ADDRESS>.2 -U  -B  -C

When I run the command manually, it gives me this error and exits:
Error parsing Kerberos principal name "-u": Configuration file does not specify default realm.

Certmonger version is certmonger-0.79.13-5.el8.x86_64. I don't use kerberos on my VMs. When pcsd certificate and key already exists, the Ansible task says ok and tests pass.

Hmmm, on my test host from 1minutetip, I don't have kerberos nor certmonger configured.
# rpm -qa | egrep krb5
krb5-libs-1.18.2-14.el8.x86_64
# rpm -qa | egrep certmonger
#

This is my task output, which was successful.
TASK [redhat.rhel_system_roles.certificate : Ensure certificate requests] ******
task path: /usr/share/ansible/collections/ansible_collections/redhat/rhel_system_roles/roles/certificate/tasks/main.yml:109
Thursday 13 October 2022  12:22:15 -0400 (0:00:01.219)       0:00:49.742 ******
changed: [myself] => (item={'name': '/var/lib/pcsd/pcsd', 'common_name': 'ci-vm-10-0-137-248', 'ca': 'self-sign'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "item": {
        "ca": "self-sign",
        "common_name": "ci-vm-10-0-137-248",
        "name": "/var/lib/pcsd/pcsd"
    }
}
MSG:
Certificate requested (new).
META: role_complete for myself

I wonder how I can reproduce the problem... Again, I'm on 8.6 and using ansible 2.12.
# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)
# ansible-playbook --version
ansible-playbook [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True

[citest]

[citest pending]

@nhosoi I merged a branch adding support for qnetd and qdevice. Communication between those cluster components is TLS encrypted and the role sets up TLS certificates. Could you take a look at that to check if this is something we want to do with the certificate role? Thanks.

@nhosoi, Thank you for helping me with the getcert issue! With a bit of experimentation and reading documentation, I was able to figure out what the problem was (see bellow). I fixed my VMs and updated my tests scripts. Your patch now passes all tests on my systems.
When one creates a certificate and key using certmonger (which the certificate role does), certmongerstores information about the certificate and key in its database. If the certificate and key files are then deleted while keeping the corresponding database records intact (which I did to test the role), then the getcert command gets stuck. And so does the Ansible task running it. I managed to fix it by running getcert stop-tracking --certfile=/var/lib/pcsd/pcsd.crt --keyfile=/var/lib/pcsd/pcsd.key. That removed the database records related to pcsd certificate and key.
I imagine this may be difficult to debug for customers using the role and not knowing what it does behind the scenes. If the issue cannot be resolved by the role, I think it should be at least documented. What do you think?

Thank you soooo much for figuring out the issue of getcert, @tomjelinek!
I could reproduce the getcert hung issue by removing /var/lib/pcsd/pcsd.key and /var/lib/pcsd/pcsd.crt before calling the 2nd certificate role.
Let's ask @rjeffman what's the best way to workaround it...
@rjeffman, could you please advise? We are trying to use the certificate role from the ha_cluster role. @tomjelinek found out if we create the key and cert pair by the certificate role, remove them manually, then run the certificate role again, the role hangs in /usr/bin/getcert resubmit command line invoked from module_utils/certificate_lsr/providers/certmonger.py.
We'd like to learn how we could avoid this hanging. Manual removing the key and cert pair should be avoided? But once it happens, we need some way to reset it... @tomjelinek figured out it's fixed by calling getcert stop-tracking --certfile=/var/lib/pcsd/pcsd.crt --keyfile=/var/lib/pcsd/pcsd.key, but it's hard for the other roles that use the certificate role to do...
If needed, we can move to https://github.com/linux-system-roles/certificate/issues to continue the discussion. Please let us know. Thanks!!

@nhosoi I merged a branch adding support for qnetd and qdevice. Communication between those cluster components is TLS encrypted and the role sets up TLS certificates. Could you take a look at that to check if this is something we want to do with the certificate role? Thanks.

Will do. Thanks! BTW, I rebased on top of your merges and squashed.

@nhosoi, @tomjelinek, I need a few days to look into it. Unfortunately, not before next Thursday (Oct 18).
Let's keep the discussion here, in one place, at least until we understand it is a certificate role issue (and not something else).

[citest]

@tomjelinek, I got one question regarding the key and cert pairs for qnetd and qdevice. You prepared ha_cluster_pcsd_private_key_src and ha_cluster_pcsd_public_key_src to copy to the pair to the pcsd node. But not for qnetd and qdevice. Are there any reason why you did not? Is it because it's better rely on the pair generated by qnetd and qdevice? Thanks!

I think qnetd only accepts certificates signed by its certificate authority. More details can be found in corosync-qdevice(8) man page, section 'MODEL NET TLS CONFIGURATION', and corosync-qdevice-net-certutil script.
Pcs reimplements corosync-qdevice-net-certutil, and ha_cluster role reimplements that.
Since the certificates are stored in NSS database, a simple copy may not work. It's possible to get a signed certificate from qnetd CA, send it to all nodes, and let them import it into their NSS databases (pcs does that). Doing it the same way in the role would just add more code, so I decided it's better to leverage Ansible and run the script for setting certificates up on all nodes.

I think qnetd only accepts certificates signed by its certificate authority. More details can be found in corosync-qdevice(8) man page, section 'MODEL NET TLS CONFIGURATION', and corosync-qdevice-net-certutil script.
Pcs reimplements corosync-qdevice-net-certutil, and ha_cluster role reimplements that.
Since the certificates are stored in NSS database, a simple copy may not work. It's possible to get a signed certificate from qnetd CA, send it to all nodes, and let them import it into their NSS databases (pcs does that). Doing it the same way in the role would just add more code, so I decided it's better to leverage Ansible and run the script for setting certificates up on all nodes.

Thanks for your inputs, @tomjelinek . Certs are stored in NSS... Indeed, it'll be harder or even impossible to make it work with the certificate role. Currently, certomonger is the only a provider the certificate role supports. So, most likely the CA in the NSS database is out of the scope even if it's retrieved and placed as a crt format...
I agree with your decision. Let's finish this task with pcsd support only. Could you please merge this pr, @tomjelinek? I'd like to have an advice from @rjeffman regarding the cert removal issue, but it can be done in a separate vehicle (it might be in the certificate role). Thanks!

Let's finish this task with pcsd support only. I'd like to have an advice from @rjeffman regarding the cert removal issue, but it can be done in a separate vehicle (it might be in the certificate role). Thanks!

Sounds good to me.


Add qlcommonmark 1.1

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} reports no offenses.
[x] The commit message includes the cask’s name and version.

Additionally, if updating a cask:

[ ] If the sha256 changed but the version didn’t,
provide public confirmation by the developer: {{link}}

Additionally, if adding a new cask:

[x] Named the cask according to the token reference.
[x] brew cask install {{cask_file}} worked successfully.
[x] brew cask uninstall {{cask_file}} worked successfully.
[x] Checked there are no open pull requests for the same cask.
[x] Checked the cask was not already refused in closed issues.
[x] Checked the cask is submitted to the correct repo.

https://github.com/caskroom/homebrew-cask/issues/36890
This has 8 9 stars and it's a cask request from a different user, not self-submitted so I'm not sure if it's too obscure or not.
QLCommonMark

Ping @vitorgalvao for final decision.

and it's a cask request from a different user

I'm the author of QLCommonMark, and the user who submitted it beat me to it by an hour.  If that has has any bearing...

Let’s not make a habit of it (saying this as a note to people in the future who might want to use this PR as reference) but merge it this time. It’s one star under the threshold and the work is already done.


Added "-d" or "--concat_dims" argument

The default for merging individual kerchunk records is concat_dims=["time"]. There are cases where you want to concatenate over other dimensions as well.
This PR resolves #48 as it allows you to send in the list of concatenation dimensions at the command-line, e.g.:
kerchunk_tools create ... -d ensemble_member,time ...
kerchunk_tools create ... --concat-dims ensemble_member,time ...

And it can also be sent into the kerchunk_tools.indexer.Indexer.create(...) method.
The default value for concat_dims is still ["time"] so this change should be backwards-compatible.

@dwest77a: please can you take a slow look at this PR as I've manually merged in the conflicts between our cli.py and indexer.py files. Please check my changes look consistent with yours. Thanks


Added Secret Key Cryptography

Hi,
Stjepan from Manning here again. This time I've added the latest Cryptography title in our catalog: Secret Key Cryptography by Frank Rubin. Thank you for considering it.
Best,

Thank you.
Best,
On Wed, Feb 2, 2022 at 1:29 PM Daniel Romero @.***>
wrote:

Merged #55 https://github.com/infoslack/awesome-web-hacking/pull/55
into master.
—
Reply to this email directly, view it on GitHub
https://github.com/infoslack/awesome-web-hacking/pull/55#event-5995409751,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANDIEMJ2VPF35BY75BUKPDLUZEPTJANCNFSM5NL3RQQA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Stjepan Jureković
Manning Publications - Marketing
@.***


[Improvement] Mentor Registor Form

There is a need to improve the Mentor register form.

Can't handle profile pic of large size. [If not able to take a large image then put condition then if the limit exceeds]
On click button it took so a loader can be made so mentor gets to know ok!

Debug!

I wish to contribute to this. Assign this to me with the gssoc'23 label.

sure!

R u working? if any issue feel free to ask.

Yeah I'm working on this.

Hii? @Swetha-Baskaran are you active on this?

Yes, I'll give the pr by today

issue resolved? @Swetha-Baskaran

Will you pls confirm what is maximum image file size should be allowed to upload?

We take image from google and other you can keep a minimal size for regular person to upload. like 2mb.

our inspiration: https://mentro.tech/mentor/apply

@Swetha-Baskaran ? are you there

Yeah I'm here. One more thing, will you give more details about the loader?
And the I made the message text box as fixed for now. You want to make it's height equal to the height of the para? Even it's so long?

Make a PR and provide the screenshot will review it @Swetha-Baskaran !!

@Swetha-Baskaran we are thinking as some what like this -https://mentro.tech/mentor/apply

no response closing as of now @anmode


Front pag echanges

Added a good baseline for the front page in terms of aesthetics/changes. Tweaks will be made ot make them look nicer.

i forgor to spell page right :(


Refactor using JS classes

In this milestone, the previous features (add and remove books from list/local storage) were refactored to use JavaScript classes.
-Refactored previous JavaScript code
-Updated styles to match medium wireframe
Note : eslint disable commands were used in accordance with guidelines provided in the previous milestone.

Thank you for the review and the suggestions @Whoistolu. Accordingly, the following updates have been made :

The deployment link has been updated to reflect 'coming soon'. [OPTIONAL 1]
The test.js file has been renamed to main.js [OPTIONAL 2]
eslint-disables have all been removed [REQUIRED 1 and REQUIRED 2]


I have been missing a timeout to devices over TCP connection. This is an attempt at implementing this functionality.

The code for tcp connection has been tested extensively to adb devices and have worked without any known issues, however the unittests for tcp connection could need input which would be much appreciated.

Coverage decreased (-1.08%) to 47.041% when pulling 1aaa895e9b882acd6e872154151d0dd606e05773 on farthornas:master into cb4ae46fc5d2b30254551ea71b3be4b64879c216 on google:master.

Coverage decreased (-0.7%) to 47.442% when pulling 2251459c22efd6fe12aa19960513b20a8260a5ae on farthornas:master into cb4ae46fc5d2b30254551ea71b3be4b64879c216 on google:master.

Coverage decreased (-0.8%) to 47.347% when pulling 660cf2b0d6a8c2ad34861c863ac5b734bfd0d000 on farthornas:master into cb4ae46fc5d2b30254551ea71b3be4b64879c216 on google:master.

Coverage decreased (-0.7%) to 47.41% when pulling 8ba24bf2a1bdf61062b30c408ffe1d9933c92208 on farthornas:master into cb4ae46fc5d2b30254551ea71b3be4b64879c216 on google:master.


some youtube videos has 302 infinite loop

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.03.20. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.03.20

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
192:youtube_downloads zbb$ youtube-dl --proxy socks5://<IP_ADDRESS>:1080 -v https://www.youtube.com/watch?v=TtvVHokhSoM
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--proxy', u'socks5://<IP_ADDRESS>:1080', u'-v', u'https://www.youtube.com/watch?v=TtvVHokhSoM']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.03.20
[debug] Python version 2.7.13 - Darwin-16.4.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {u'http': u'socks5://<IP_ADDRESS>:1080', u'https': u'socks5://<IP_ADDRESS>:1080'}
[youtube] TtvVHokhSoM: Downloading webpage
[youtube] TtvVHokhSoM: Downloading video info webpage
[youtube] TtvVHokhSoM: Extracting video information
[youtube] TtvVHokhSoM: Downloading MPD manifest
[debug] Invoking downloader on u'https://r5---sn-i3b7kn76.googlevideo.com/videoplayback?expire=1490122150&initcwndbps=1236250&ei=RiHRWPfMHYbD4QK7_KrwCg&dur=2507.430&mv=m&mt=1490100447&ms=au&requiressl=yes&ip=<IP_ADDRESS>&ipbits=0&mn=sn-i3b7kn76&mm=31&pl=24&mime=video%2Fmp4&id=o-ABxwIXjtTXUYbpdzxIpaJP9YbyEJWpmrO2pF7FOLrOIw&lmt=1488748718277559&key=yt6&itag=22&upn=ySIRfr_OPA8&signature=A395367D9D87D57CDED2590002DB1136AEB30AEE.4D8C4D36411D05466E82B8F80A408DECFBC6A96C&ratebypass=yes&source=youtube&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire'
[download] Got server HTTP error. Retrying (attempt 1 of 10)...
[download] Got server HTTP error. Retrying (attempt 2 of 10)...
[download] Got server HTTP error. Retrying (attempt 3 of 10)...
[download] Got server HTTP error. Retrying (attempt 4 of 10)...
[download] Got server HTTP error. Retrying (attempt 5 of 10)...
[download] Got server HTTP error. Retrying (attempt 6 of 10)...
[download] Got server HTTP error. Retrying (attempt 7 of 10)...
[download] Got server HTTP error. Retrying (attempt 8 of 10)...
[download] Got server HTTP error. Retrying (attempt 9 of 10)...
[download] Got server HTTP error. Retrying (attempt 10 of 10)...
ERROR: unable to download video data: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1791, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1733, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 356, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 61, in real_download
    data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2094, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 644, in http_error_302
    self.inf_msg + msg, headers, fp)
HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

Format 22 you are requesting is not available on YouTube side. Choose another format.
Duplicate of #3721.


Missing Go Tool

The editor prompts me to install gocode and cover tool, however I have already installed them manually via go get. When i try to install it again in the editor the following message appears:
The go tool is required to perform a get. Please ensure you have a go runtime installed: http://golang.org.
So I'm pretty sure there's something wrong with the detection, but I can't get any info what is being actually detected in the system: in other issues I've seen you suggested to look into Packages > Go-plus > ... and there should be an appropriate option to get this info (can't remember the exact name of the option right now - but when I looked it up, I didn't have it). I don't even have the go-plus menu under Packages, only go (maybe this is irrelevant). I attached a screenshot to show what I have there.

I also found no option to check what has been detected in go-config and go-plus packages Settings.
To sum up: is there a way I can see what go-config actually detected?

Launch atom from your terminal if you are on atom < 1.7.0 or on Linux.

Yeah already tried it and the result is the same unfortunately :(
Sorry for forgeting to write about my configuration - my OS is Mint Rafaela (practically an Ubuntu 14.04) and Atom 1.6.2

Try the suggestions in this: https://github.com/joefitzgerald/go-plus/issues/386#issuecomment-199359955. Remember, you can press ctrl-shift-i to get to the chromium debugger, from which you can go to the console and write process.env to debug.

I found the solution: I had  long time ago installed go using apt, and this binary was the source of problem - still don't know why exactly, but removing it and having a once again clean install from tarball did the trick.
If this information have any significant matter for you for the future: GOPATH and PATH were OK inside process.env before I removed this package from apt.
Thank you for very quick responses and have a good day :)

Glad to hear you got it resolved!

I had the same issue on OSX where I installed go with brew. Turned out it fixed it for me when getting rid of the GOROOT env var. See also http://stackoverflow.com/a/35823325/3479043 and the linked blog post of Dave Cheney.


Fix the PhoneNumber serialization bug

Contributing to the Azure SDK
Brief description: Fixed the logic of PhoneNumberIdentifier to always maintain the original phone number string whether it included the leading + sign or not.
More details: https://skype.visualstudio.com/SPOOL/_workitems/edit/2981911
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

#assign-reviewer: @petrsvihlik @maximrytych-ms @AikoBB @ostoliarova-msft

API change check
API changes are not detected in this pull request.


Set LC_ALL on Unix

Use the DSL instead of edits via the UI.

@donat I actually don't know what other environment variables should be moved there. What do you have in mind? Do think something else will be coming?

We have a few parameters that might make sense to define at the same place:

Java home
requires-investigation=false
REPO_MIRROR_URL
etc.

There won't be new ones afaict.


refactor STD_CARGO_KG

Description
Add STD_CARGO_KG to vehicle mass once, instead of every single time we set vehicle mass.

special handling for Civic in selfdrive/car/__init__.py to prevent circular references
special handling for comma body in selfdrive/car/interfaces.py so it doesn't carry a human around

GitHub will tell you this PR is +5, but if you listen to your heart 👩🎤 it's -140.
Verification
#!/usr/bin/env python3

from selfdrive.car import gen_empty_fingerprint
from selfdrive.car.car_helpers import interfaces
from selfdrive.car.fingerprints import all_known_cars

if __name__ == "__main__":
  for car in all_known_cars():
    CarInterface, CarController, CarState = interfaces[car]
    car_fw = []
    fingerprints = gen_empty_fingerprint()
    CP = CarInterface.get_params(car, fingerprints, car_fw, experimental_long=False, docs=False)
    print(f"{CP.carFingerprint}: {CP.mass=}")

mass_before.txt
mass_after.txt
Differences
Some cars with mass listed in pounds (and therefore multiplied by CV.LB_TO_KG) change very slightly because the cereal Float32 conversion is now done before adding STD_CARGO_KG instead of afterward.
 In this case, we're essentially fixing a bug.
GENESIS G90 2017: CP.mass=2200.0
GENESIS G90 2017: CP.mass=2336.0
HYUNDAI SONATA 2019: CP.mass=2039.80322265625
HYUNDAI SONATA 2019: CP.mass=2175.80322265625
HYUNDAI TUCSON 2019: CP.mass=1596.643798828125
HYUNDAI TUCSON 2019: CP.mass=1732.643798828125
HYUNDAI VELOSTER 2019: CP.mass=1613.88037109375
HYUNDAI VELOSTER 2019: CP.mass=1749.88037109375
KIA FORTE E 2018 & GT 2021: CP.mass=1613.88037109375
KIA FORTE E 2018 & GT 2021: CP.mass=1749.88037109375
KIA K5 2021: CP.mass=1464.1949462890625
KIA K5 2021: CP.mass=1600.1949462890625
KIA K5 HYBRID 2020: CP.mass=1464.1949462890625
KIA K5 HYBRID 2020: CP.mass=1600.1949462890625
KIA OPTIMA 4TH GEN FACELIFT: CP.mass=1613.88037109375
KIA OPTIMA 4TH GEN FACELIFT: CP.mass=1749.88037109375
KIA OPTIMA 4TH GEN: CP.mass=1613.88037109375
KIA OPTIMA 4TH GEN: CP.mass=1749.88037109375
KIA OPTIMA HYBRID 2017 & SPORTS 2019: CP.mass=1613.88037109375
KIA OPTIMA HYBRID 2017 & SPORTS 2019: CP.mass=1749.88037109375

Additional fix: the base mass for HYUNDAI SONATA 2019 was wrong. The value of 2039 kg appears to be a gross (maximum) weight, it should be more like 1536 kg (average of curb weights according to the Internets). This is a pretty hefty difference and it might have had a negative effect on params learning/tuning. This had to be addressed in this PR; it's one of the cars that wasn't adding STD_CARGO_KG in the past and adding it now would exacerbate the problem.

You can fix any incorrect masses in a new PR! Like to keep refactors free from behavior-changes

You can fix any incorrect masses in a new PR! I like to keep refactors free from behavior-changes, no matter small

Agreed, I didn't realize there would be functional changes until I fully validated the outcome.
The mass fixes are set up in #29418, once that's merged I'll rebase and revalidate this PR.

This version of the refactor pushes the Honda Civic parameters down into the car port, so it's no longer a special case with the potential to create circular references. I'm happier with this, @adeebshihadeh hopefully this resolves your concern.
I revalidated all cars after #29418 and found the only remaining differences trace back to floating-point precision issues, which have been previously identified and can be safely ignored.
mass2_before.txt
mass2_after.txt
GMC ACADIA DENALI 2018: CP.mass=2110.486083984375
GMC ACADIA DENALI 2018: CP.mass=2110.48583984375
LEXUS RX 2016: CP.mass=2168.545654296875
LEXUS RX 2016: CP.mass=2168.5458984375
LEXUS RX 2020: CP.mass=2168.545654296875
LEXUS RX 2020: CP.mass=2168.5458984375
LEXUS RX HYBRID 2017: CP.mass=2168.545654296875
LEXUS RX HYBRID 2017: CP.mass=2168.5458984375
LEXUS RX HYBRID 2020: CP.mass=2168.5458984375
LEXUS RX HYBRID 2020: CP.mass=2168.545654296875
TOYOTA ALPHARD 2020: CP.mass=2088.713623046875
TOYOTA ALPHARD 2020: CP.mass=2088.71337890625
TOYOTA ALPHARD HYBRID 2021: CP.mass=2088.713623046875
TOYOTA ALPHARD HYBRID 2021: CP.mass=2088.71337890625

LGTM, nice cleanup! @sshane anything blocking this?

I believe we were in agreement on that last item of feedback, I just haven't done it yet.

https://github.com/commaai/openpilot/pull/29459


$upstream_response_time might contain multiple, comma-separated values

Describe the bug
If there are multiple upstream backends and more than one will be tried by nginx, $upstream_response_time will contain a comma-separated list of times.
If this is happening, the log line will be parsed incorrectly
nginx documentation for $upstream_response_time:
keeps time spent on receiving the response from the upstream server; the time is kept in seconds with millisecond resolution. Times of several responses are separated by commas and colons like addresses in the $upstream_addr variable.

To Reproduce
Add multiple upstream backends (e.g. PHP-FPM) to nginx, break the first one and request a site using this backend.
Expected behavior
Log line is parsed properly and proper metrics are available
Configuration file (remove section, if not applicable):
namespaces:
  - name: nginxlog
    format: "$server_name $remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $upstream_response_time $pipe $ssl_protocol/$ssl_cipher $request_id $host"
    source:
      syslog:
        listen_address: "udp://<IP_ADDRESS>:8514"
        format: "rfc3164"
        tags: ["nginx"]
    relabel_configs:
      - target_label: "vhost"
        from: "server_name"
      - target_label: "ssl_protocol"
        from: "ssl_protocol"
      - target_label: "user_agent"
        from: "http_user_agent"
        whitelist: ["KeepAliveClient"]

Example log file (remove section, if not applicable):
www.example.com <IP_ADDRESS> - - [27/Oct/2021:06:00:14 +0200] "GET /websocket HTTP/1.1" 502 552 "-" "SomeUserAgentString" 0.000 0.000, 0.000, 0.000, 0.000 . TLSv1.3/TLS_AES_256_GCM_SHA384<PHONE_NUMBER> www.example.com
www.example.com <IP_ADDRESS> - - [27/Oct/2021:06:00:15 +0200] "GET /websocket HTTP/1.1" 502 552 "-" "AnotherUserAgent" 0.000 0.000, 0.000, 0.000, 0.000 . TLSv1.3/TLS_AES_256_GCM_SHA384<PHONE_NUMBER> www.example.com
www.example.com <IP_ADDRESS> - - [27/Oct/2021:06:00:15 +0200] "GET /websocket HTTP/1.1" 502 150 "-" "SomeUserAgentString" 0.001 0.000, 0.000, 0.000, 0.000 . TLSv1.3/TLS_AES_256_GCM_SHA384<PHONE_NUMBER> www.example.com

Metrics output (remove section, if not applicable):
Not exactly matching the log lines above, but the error is clearly visible, looking at ssl_protocol, which should have a value like TLSv1.3 only
nginxlog_http_response_time_seconds_hist_bucket{method="GET",ssl_protocol="0.000, 0.000 . TLSv1.3",status="502",user_agent="other",vhost="www.example.com",le="0.1"} 1116
nginxlog_http_response_time_seconds_hist_bucket{method="GET",ssl_protocol="0.000, 0.000 . TLSv1.3",status="502",user_agent="other",vhost="www.example.com",le="0.25"} 1116
nginxlog_http_response_time_seconds_hist_bucket{method="GET",ssl_protocol="0.000, 0.000 . TLSv1.3",status="502",user_agent="other",vhost="www.example.com",le="0.5"} 1116
nginxlog_http_response_time_seconds_hist_bucket{method="GET",ssl_protocol="0.000, 0.000 . TLSv1.3",status="502",user_agent="other",vhost="www.example.com",le="1"} 1116

Environment:

Exporter version: 1.9.0
OS (e.g. from /etc/os-release): Ubuntu 20.04.3 LTS
Deployment method (e.g. Docker image, deb/rpm package, self-compiled, ...): binary
Others:

@lhanjian Thanks for your suggestion, which I tend to agree with. #221 adds support for multiple values, which are summed into a single observation.


Failure to run GCP Bucket Brute with authentication

Good Morning,
I'm receiving an error when I attempt to scan a GCP bucket while supplying credentials and I'm not sure where I've gone wrong.
I run
'python3 gcpbucketbrute.py -k <BUCKET_NAME> -f sa.pem'
(Just for clarity, BUCKET_NAME is replaced with my bucket name)
And I receive the following error:
raise PicklingError(
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
I downloaded the service account credentials from GCP after creating and haven't modifed that file. It was originally a .json file which also didn't work.
Thanks,
Joe

hi @joebromley,
Same issue here, the temporal solution i have found, was using the docker image uploaded by other user, and mentioned by @janmasarik so kudos for him, the docker image name: s14ve/gcpbucketbrute, so the error should be related to newer versions of python, hope this help you running authenticated requests.
Regards
@takionexmachina

Had same issue, got it working with Docker container
Dockerfile
FROM python:3.8

RUN git clone https://github.com/RhinoSecurityLabs/GCPBucketBrute.git
WORKDIR /GCPBucketBrute
RUN pip3 install -r requirements.txt

ENTRYPOINT ["python3",  "gcpbucketbrute.py"]

Build:
docker build . -t gcpbucketbrute-image

Run (replace values accordingly):
docker run --rm -it --mount type=bind,source="<enter_path_to_credentials.json",target=/GCPBucketBrute/creds.json gcpbucketbrute-image --check <enter-bucket-name> -f creds.json