Provide guidance (or means) for how PyPI users can provide us with contact information

Currently when we discover problematic clients hitting PyPI, we have very little means to contact those users to ask them to stop or to work with them to remediate the problems they're causing us. Due to the fact our API is almost wholly unauthenticated, the only real information we have about clients is what is provided in the request itself, which typically boils down to User Agent and IP address, neither of which is typically super useful in tracing back to a responsible party (particularly since custom scripts/tooling that hit's PyPI often will use a generic HTTP client user agent).
This generally means that whenever we have a client generating problematic traffic, our only real option is to block their IP addresses, and wait for them to reach out to us when they discover they can't access PyPI. This "works", but it's non optimal for either side of the equation.
It would be really good if we could, at a minimum, provide guidance for what consumers of PyPI could do to maximize the chances we can proactively contact them to help them remediate issues before we have to resort to IP blocking them, or even better, provide some specific means to supply contact information (or do both).
Presumably this guidance would largely be "correctly set your user agent to identify the script/tool that's being used, and provide an URL or email address that we can use to contact you at), something as simple as FooBot (https://example.com/contact/) would go a long way to providing that capability.
This is still somewhat problematic in that you have reusable tooling like pip, DevPI, Bandersnatch, etc where end users really don't have control over the UserAgent (and in some cases, we don't want them to change those user agents). If we're getting bad requests from say pip, there are two possible causes. One possible cause is that pip itself is broken in some way, and needs to be fixed. Problems of that nature should be redirected to the pip developers, since they're the ones who control what pip actually does. Another possible cause is that someone is using pip in such a way that it's generating bad requests (for example, they have pip running in a loop just doing pip install <random non existent package name>, which isn't really pip's fault, but is a fault of the user of pip, and in those cases we would want to route those problems to the person using pip, not to pip developers themselves.
If we wanted to go further than the "don't use generic UA" guidance, I could imagine a few options:

Expand our existing API keys (or add new ones) such that they can be used to access things like /simple/ and such.
This could be implemented in a few ways, at the most minimal level we could programmatically ignore them and simply use them as a means of looking up contact information. We'd basically be asking pip, etc to allow configuration of this, and send it along, but otherwise don't use it. Our guidance could then include sending that data along with all requests. Unfortunately if it's optional AND ignored, I suspect most people will just never actually do that.
We could also expand the above, and start rate limiting the API for unauthenticated users (with a generous rate limit) and allow people to provide an API key to remove (or increase) the rate limits. If we carefully set the limits, this could allow the common case of individual users to still do pip install unauthenticated, but provide a sort of feedback mechanism that would require large users to effectively register with us so we have contact information for them.

Define a mechanism that tools like pip can implement for end users to directly specify contact information. This could be something like an additional item in the user agent, or something like the From header. These tools could then provide the means for end users to configure them with additional information to tell PyPI what their contact information is.
Similarly to the API key option, we could either just ignore the header, and only use it manually when we're investigating problematic requests OR we can attempt to use some forcing function, that adds some sort of rate limit to requests without this mechanism, and requests with it get a higher or unlimited rate limit.

Personally, I think the right approach here is:

Document that all clients accessing PyPI should have a meaningful UserAgent.
Document that requests to PyPI SHOULD have a From header indicating an email address that the "owner" of the request can be contacted at, and get pip/poetry/bandersnatch/etc to implement this.

I'm on the fence about rate limiting, I worry that without the rate limit uptake will be low but I don't think we should require random small time uses of the API to include contact information. I would say I'm probably +0 on rate limiting requests that don't have a From header, assuming we size the rate limit correctly and we phase it in.

Two quick comments from Dependabot's point of view:

I think the second of the two options presented would be preferable from our perspective. While most of our traffic is unauthenticated, we do issue authenticated requests on behalf of our users, and in those cases I think it'd still be useful for you to be able to correlate the traffic back to Dependabot. Additionally, because Dependabot sometimes runs user-provided code, it would be possible for users to exfiltrate our API key and masquerade as us.
Pointing to a GitHub repo's issues page would probably work better for us than an email as we don't really use email distribution lists internally, and an issue is more likely to get routed to the right people and get attention quickly. I could imagine that being the case for other projects too.

The authenticated option is an option I was already kind sketch on the idea to start with, as I don't really like the idea of ignoring the contents of the auth and just checking it exists, and validating it on every request would be a massive increase in the load on our origin. I think the problem of user provided code (which isn't a dependabot only problem) on top of all of that sort of kills that idea (the only way around it I could think of, is some sort of proxy that adds the auth header outside of the control.... but that's even more jank added).
To be a valid From header, it'd have to be an email address, but we could also just add something custom like Contact that can be an email OR url. If we didn't want to define our own header, we could even do something like support From for an email address and/or Link: <https://exampe.com/>; rel=contact or From and Contact-URL or something.
I believe the reason the From header restricts it to emails, is so automated tooling can be written to send emails to that address, which obviously wouldn't work if it allowed arbitrary URLs. I could see some value in that for making it easier to evolve PyPI in the future, you could imagine situations where we deprecate some API, and then use the From header to automatically send emails to people using that API, to tell them that their thing might break soon as we remove something...
That being said, it would be easy enough to do an email OR url field, and simply document that there is a preference for email because automated deprecation notices / warnings can be sent to them, but allow URLs if the user is happy managing that themselves, but still wants to make themselves easy to contact in the case of problematic requests. IOW, providing an email is more of a value add for the user, so they can pick if they want that value add or not.


Caching user playlists locally

By now loading user playlists by looking up all the tracks from GMusic is a very, extremely slow operation. In my case I have something like 20000 tracks in my playlists and this can really take up to 20 minutes for having mopidy+gmusic up & running. Wouldn't it be a good idea to cache trackIDs -> info locally, so we don't have to perform the GMusic lookup every time?

+1 this would be great to have

duplicate of #28

@berot3 oh, I forgot I already reported that issue :P

Closing as duplicate of #28.


what types do ID and Long String refer to in JAVA or ParquetFile?

hello, 👋 . I'm using ldbc to do some benchmark on our graph. however graphs don't support csv import. I need to convert csv files to parquet files. So I want to know what do ID and Long String in graph schma exactly refer to.

Hi @geralt-write-code, which LDBC workload are you interested in using?
We have three major workloads for database benchmarks:

Interactive 1.0: OLTP benchmark with inserts, uses the Datagen version contained in this repository
Interactive 2.0: OLTP benchmark with inserts and deletes, uses the Spark-based Datagen
BI 1.0: OLAP benchmark with inserts and deletes, uses the Spark-based Datagen

The Spark-based Datagen is capable of generating parquet files, so using that would be the simplest approach.
If you would like to use the old Hadoop-based Datagen and do the conversion manually, the meaning of the types is the following:

String variable length text of size 40 Unicode characters
Long String variable length text of size 256 Unicode characters
Text variable length text of size 2000 Unicode characters

(https://ldbcouncil.org/ldbc_snb_docs/ldbc-snb-specification.pdf, page 15)

Hi @geralt-write-code, which LDBC workload are you interested in using? We have three major workloads for database benchmarks:

Interactive 1.0: OLTP benchmark with inserts, uses the Datagen version contained in this repository
Interactive 2.0: OLTP benchmark with inserts and deletes, uses the Spark-based Datagen
BI 1.0: OLAP benchmark with inserts and deletes, uses the Spark-based Datagen

The Spark-based Datagen is capable of generating parquet files, so using that would be the simplest approach.
If you would like to use the old Hadoop-based Datagen and do the conversion manually, the meaning of the types is the following:

String variable length text of size 40 Unicode characters
Long String variable length text of size 256 Unicode characters
Text variable length text of size 2000 Unicode characters

(https://ldbcouncil.org/ldbc_snb_docs/ldbc-snb-specification.pdf, page 15)

hello @szarnyasg , I'm using Interactive 1.0 workloads and thank you for your reply! This will be very helpful for my benchmark! And maybe I'll also try Interactive 2.0 in the future

No problem, glad it helped. If you're looking for a convenient Paquet writer, it might be worth checking out DuckDB's Parquet features.


html element lesson: url image(the cat) not working.

http://bit.ly/fcc-relaxing-cat
this link does not open

Works fine for me. It is hosted on Amazon's services.  It sounds like it's a local networking issue.  Could be related to proxies or VPN, or possibly a plugin you are running in your browerser.
Closing as unable to reproduce.
Thanks and Happy Coding!

it worked after maybe three lessons. and thank you.


Allow slow concat time to be configurable

This is related to #36 #38. Currently, the only way to suppress the message:
[WARN] (broccoli-uglify-sourcemap) Minifying: 'assets/somefile.js' took: 21087ms (more than 20,000ms)
is by using the --silent option, which also suppresses some other messages which I don't want to suppress. I propose that we allow the "slow" concat time to be configurable (it is hard coded at 20000ms right now).  Then users can suppress this message only by increasing the time.

I wonder why it is taking so long. I upgraded from broccoli-uglify-js to this and uglify time increased from 9000ms to 24000ms. I had to revert to broccoli-uglify-js since this has slowed down development drastically.

Any recent update on this issue?

still nothing :)?

Seems like a great pull request to make 😸


Commit for #23 - Implement synthetic traffic filtering from GSM

We are sending the changes to resolve issue #23
This was worked with a team composed by myself, @jorgemariomm and @oscargarciacolon.

@gzepeda Thank you for the feature update! Can you please create a new pull request against masterRC1. I will close this for now, as we are updating master with rc2 changes soon. Once we have the features in masterRC1, it shouldn't be difficult to merget to the actual master. We shall review the new pull request. Thanks.

Will do. Thanks!


OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME must be always set to false

Describe the bug
I couldn't find a configuration where I can set OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME to true (or leave it on the default true for newer oCIS versions)
Steps to reproduce

use the oCIS chart and deployment example from https://github.com/owncloud/ocis-charts/pull/701
set verfiyHostnameIncomingRequests to true

Expected behavior
I'm still able to perform the OCM invitation flow.
Actual behavior
The OCM invitation flow fails:
proxy-697dcd77c7-wj25f proxy {"level":"debug","service":"proxy","policy":"ocis","method":"POST","prefix":"/ocm/","path":"/ocm/invite-accepted","routeType":"prefix","time":"2024-10-21T05:34:28Z","line":"github.com/owncloud/ocis/v2/services/proxy/pkg/router/router.go:222","message":"rewrite hook found"}
ocm-79f6b766f7-wm5bj ocm {"level":"info","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/interceptors/auth/auth.go:195","message":"skipping auth check for: /ocm/invite-accepted"}
ocm-79f6b766f7-wm5bj ocm {"level":"warn","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/interceptors/auth/auth.go:248","message":"core access token not set"}
ocm-79f6b766f7-wm5bj ocm {"level":"info","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/interceptors/providerauthorizer/providerauthorizer.go:83","message":"skipping provider authorizer check for: /ocm/invite-accepted"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rhttp","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/pkg/rhttp/rhttp.go:248","message":"http routing: head=ocm tail=/invite-accepted svc=ocm"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","request-id":"dc0423ebcdaa0a3b7a6355c30b6e340d","path":"/invite-accepted","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/services/ocmd/ocm.go:112","message":"ocm routing"}
gateway-6769fc757c-jqq5m gateway {"level":"debug","service":"gateway","pkg":"rgrpc","traceid":"2cc6a60028df8dfb6120c1261ced26f5","method":"/cs3.gateway.v1beta1.GatewayAPI/IsProviderAllowed","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/grpc/interceptors/auth/auth.go:122","message":"skipping auth"}
proxy-697dcd77c7-wj25f proxy {"level":"info","service":"proxy","proto":"HTTP/1.1","request-id":"dc0423ebcdaa0a3b7a6355c30b6e340d","traceid":"2cc6a60028df8dfb6120c1261ced26f5","remote-addr":"<IP_ADDRESS>","method":"POST","status":403,"path":"/ocm/invite-accepted","duration":3.592633,"bytes":70,"time":"2024-10-21T05:34:28Z","line":"github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/accesslog.go:34","message":"access-log"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rgrpc","traceid":"2cc6a60028df8dfb6120c1261ced26f5","method":"/cs3.ocm.provider.v1beta1.ProviderAPI/IsProviderAllowed","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/grpc/interceptors/auth/auth.go:122","message":"skipping auth"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rgrpc","traceid":"2cc6a60028df8dfb6120c1261ced26f5","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/pkg/ocm/provider/authorizer/json/json.go:153","message":"Comparing 'ocis.kube.owncloud.test' to 'ocis2.kube.owncloud.test'"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rgrpc","traceid":"2cc6a60028df8dfb6120c1261ced26f5","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/pkg/ocm/provider/authorizer/json/json.go:153","message":"Comparing 'ocis2.kube.owncloud.test' to 'ocis2.kube.owncloud.test'"}
ocm-79f6b766f7-wm5bj ocm {"level":"debug","service":"ocm","pkg":"rgrpc","traceid":"2cc6a60028df8dfb6120c1261ced26f5","user-agent":"grpc-go/1.66.1","from":"tcp://<IP_ADDRESS>:44426","uri":"/cs3.ocm.provider.v1beta1.ProviderAPI/IsProviderAllowed","start":"21/Oct/2024:05:34:28 +0000","end":"21/Oct/2024:05:34:28 +0000","time_ns":1672148,"code":"OK","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/grpc/interceptors/log/log.go:69","message":"unary"}
ocm-79f6b766f7-wm5bj ocm {"level":"error","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","request-id":"dc0423ebcdaa0a3b7a6355c30b6e340d","error":"error verifying mesh provider","time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/services/reqres/reqres.go:64","message":"provider not trusted"}
ocm-79f6b766f7-wm5bj ocm {"level":"warn","service":"ocm","pkg":"rhttp","traceid":"3e130641736892c87d2765979a131750","host":"<IP_ADDRESS>","method":"POST","uri":"/ocm/invite-accepted","url":"/invite-accepted","proto":"HTTP/1.1","status":403,"size":70,"start":"21/Oct/2024:05:34:28 +0000","end":"21/Oct/2024:05:34:28 +0000","time_ns":3056965,"time":"2024-10-21T05:34:28Z","line":"github.com/cs3org/reva/v2@v2.25.0/internal/http/interceptors/log/log.go:112","message":"http"}

Setup
oCIS 6.5 with the oCIS chart and deployment example from https://github.com/owncloud/ocis-charts/pull/701
Additional context

@wkloucek could you help run it in a minikube on mac
For ocis I added  to the charts/ocis/values.yaml
hostAliases:
- ip: "<IP_ADDRESS>"
  hostnames:
  - "ocis.kube.owncloud.test"

What should I add for the ocm?

@wkloucek Can you share the providers.json file you're using?

@wkloucek Can you share the providers.json file you're using?

https://github.com/owncloud/ocis-charts/pull/701/files#diff-8f0f77694b9f9323083180dd6bc91adf38f4b3dabb38ca2661330f0e15fc0109R140-R214

Ok, I guess there a multiple issues at play here:

When OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME is enable the code currently expects the host value in the services section to just contain a hostname with port. All our examples seem to be using a URL there:

{
    "endpoint": {
        "type": {
            "name": "Webdav",
            "description": "ocis.kube.owncloud.test Webdav API"
        },
        "name": "ocis.kube.owncloud.test Example - Webdav API",
        "path": "https://ocis.kube.owncloud.test/dav/",
        "is_monitored": true
    },
    "api_version": "0.0.1",
    "host": "https://ocis.kube.owncloud.test/"
}

See: https://github.com/cs3org/reva/blob/edge/pkg/ocm/provider/authorizer/json/json.go#L171
That should be easily fixable. Either by fixing the code to accept URL, or by fixing our configs (not sure if that will break anything else).

It then tries to lookup the IP(s) for that hostname and compares that to the IP of the Client sending the accept-invite request, which in our case is most probably the IP of the proxy service (IIRC we're not passing the X-Forwarded-For Headers, do we). ICBW, but I don't think this can really work with an ocis setup in kubernetes.

IIRC we're not passing the X-Forwarded-For Headers, do we?

Actually, the proxy tries to set the X-Forward-For Header (https://github.com/owncloud/ocis/blob/master/services/proxy/pkg/router/router.go#L174). It just fails because in my case RemoteAddr in the incoming request is just an IP without a port but net/http/httputil/reversproxy needs a host:port to work: https://cs.opensource.google/go/go/+/master:src/net/http/httputil/reverseproxy.go;l=79 so it just doesn't set it.
But does OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME really add any value security-wise? X-Forwarded-For can be easily spoofed to contain the expected IP address.

Additionally in many setups you can not really rely on the fact the IP which a DNS lookup resolves to is actually the IP you'll see in a client connection from that very same service.

:100:

But does OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME really add any value security-wise?

From what what I understood, it might do so only in very constrained environments (no reverse proxies (which is not recommend from oCIS from what I remember).
How are we going forward?
This are the options in descending order of my preference:

remove OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME as a config option from oCIS and hardcode it internally to false. What you described above sounds like a niche use case but a documentation nightmare. Would be solved be removing it... (Remember oCIS is a opinionated way to start REVA and has no requirement to expose all config options of it)
set OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME to default false in oCIS and remove OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME as option from the Helm Chart
hardcode OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME to false in the Helm Chart

I am fine with 1.

./services/ocm/pkg/config/config.go:
 VerifyRequestHostname bool   `yaml:"verify_request_hostname" env:"OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME" desc:"Verify the hostname of the incoming request against the hostname of the OCM provider." introductionVersion:"5.0"`

That options was already introduce in 5.0. So to play by our own rules we would need to go through a deprecation cycle and could only remove the setting in the next major release after 7.0 @micbar Does that really make sense here? Is ocm really supposed to be working in 5.0?


Fix REPL not opening for rake tasks.  Closes #74

This gem was not opening a REPL when running rake tasks. Here's a minimal rails 5 repo that reproduces the issue:
https://github.com/abaldwin88/pry-rescue-bug

thanks @abaldwin88 !

@ConradIrwin Thank you for the great gem!


Rename debian packages to conform to standards

Proposed changes

Rename debian packages to conform to debian guidelines (underscores (_) were renamed to dashes (-))
Secondary changes (as a result of the renaming)

Update install script to support both new and old Debian package names (which prompted other improvements)

Support downloading artefacts either with curl or wget (depending what is available)
More streamlined output, but still showing verbose information on errors
Add --dry-run support to check which artefacts will be downloaded
Only install mosquitto if it is not already installed and use either apt-get or apk to install it. This removes the explicit dependency for apt-get (but not dpkg)
Automatically use either sudo or su if the script was not called with root permissions
Use apt-get instead of apt (as apt is meant for humans not scripts)
Add protection against half-downloaded scripts via curl (inspiration taken from docker install script)
On errors print a more helpful reason for the error along with details about the user's setup

Update uninstall script to check for both new and old Debian package names

Types of changes

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
[ ] Documentation Update (if none of the other choices apply)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

[x] I have read the CONTRIBUTING doc
[x] I have signed the CLA (in all commits with git commit -s)
[ ] I ran cargo fmt as mentioned in CODING_GUIDELINES
[ ] I used cargo clippy as mentioned in CODING_GUIDELINES
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)

Further comments

Example Install
curl -LSsf https://raw.githubusercontent.com/reubenmiller/thin-edge.io/issue-1552-debian-names/get-thin-edge_io.sh | bash -

Thank you for trying thin-edge.io!

Version argument has not been provided, installing latest: 0.8.1
To install a particular version use this script with the version as an argument.
For example: sudo ./get-thin-edge_io.sh 0.8.1
Installing for architecture arm64
Installing mosquitto as prerequisite for thin-edge.io
debconf: delaying package configuration, since apt-utils is not installed

Downloading tedge...OK
Installing tedge...OK

Downloading tedge_mapper...OK
Installing tedge_mapper...OK

Downloading tedge_agent...OK
Installing tedge_agent...OK

Downloading tedge_apt_plugin...OK
Installing tedge_apt_plugin...OK

Downloading tedge_apama_plugin...OK
Installing tedge_apama_plugin...OK

Downloading c8y_configuration_plugin...OK
Installing c8y_configuration_plugin...OK

Downloading c8y_log_plugin...OK
Installing c8y_log_plugin...OK

Downloading tedge_watchdog...OK
Installing tedge_watchdog...OK

thin-edge.io is now installed on your system!

You can go to our documentation to find next steps: https://github.com/thin-edge/thin-edge.io/blob/main/docs/src/howto-guides/003_registration.md

Consider renaming the cargo project and binary names to use - instead of _ because it is now very confusing for users (see example below). This also has a big impact on maintaining the documentation (and reading!). Alternatively we could just rename the binary (not the cargo project name)

Naming conventions for rust crates are unclear and a source of discussions and discussions
Using kebab-case for crate names is okay but these must be referenced using snake_case from the rust code. So the burden is pushed elsewhere. That said the renaming request is for binary crates - hence this issue doesn't show.

Hence, I'm okay for the renaming of the binary crates and the associated executables.

Consider renaming the cargo project and binary names to use - instead of _ because it is now very confusing for users (see example below). This also has a big impact on maintaining the documentation (and reading!). Alternatively we could just rename the binary (not the cargo project name)

Naming conventions for rust crates are unclear and a source of discussions and discussions
Using kebab-case for crate names is okay but these must be referenced using snake_case from the rust code. So the burden is pushed elsewhere. That said the renaming request is for binary crates - hence this issue doesn't show.

Hence, I'm okay for the renaming of the binary crates and the associated executables.

Ok, then I will proceed with the renaming and see how it comes out.
Though I believe that we should create a symlink to the older binary names .e.g. tedge_mapper so that existing installations which are using custom init scripts in system.toml don't break.
And then we can put a deprecation notice to remove the symlinks in 1.0.0

The commit log history includes unrelated commits and must be cleared.

Yes I had to merge the changes from the main branch as there conflicts. But yes I will squash before committing.

The commit log history includes unrelated commits and must be cleared.

Yes I had to merge the changes from the main branch as there conflicts. But yes I will squash before committing.

More than a squash, you need an interactive rebase droping/removing the commits that have been added here by accident.

I experimented with the package renaming, and the folder names can remain to include underscores _, but the renaming of the binary was not so convinient as expected.
Below are the current options are:
Option 1 (preferred) - just rename package.name
[package]
name = "tedge-watchdog"

Option 2 - keep package.name as is, but set [[bin]]
[package]
name = "tedge_watchdog"

[[bin]]
name = "tedge-watchdog"
# path is required, otherwise cargo complains about not being able to find the main.rs
path = "src/main.rs"

Do you agree with the preferred option 1?
@didier-wenzek, @albinsuresh

Do you agree with the preferred option 1? @didier-wenzek, @albinsuresh

Since option 1 is also limited to just the Cargo files and not renaming every single crate directory, I'm fine with that. Option 2 isn't too bad either as we just have a handful of binary crates which needs that extra path key.

I also prefer the 1st option.

Option 1 (preferred) - just rename package.name

[package]
name = "tedge-watchdog"

@didier-wenzek  Ok now the PR is out of its draft status, I also fixed the commit history

I'm having some second thoughts about the doc changes since the docs are published as soon as the PR is merged.
Also I would like this branch to be fully tested before it gets merges so it does not break main, I have provided a built version to @gligorisaev

Ok making the decision to merge the changes without reverting the doc changes. It will introduce some breaking changes into the docs shortly before the next 0.9.0 release (in upcoming days).


How to define a field with List<List> type?

I have been using graphql-dotnet and wanted to try hotchocolate
here is code that worked with graphql-dotnet but fails with hotchocolate
Field<NonNullGraphType<ListGraphType<NonNullGraphType<ListGraphType<IntGraphType>>>>, List<List<int>>>()                 .Name("screens")                 .Resolve(context => context.Source.Screens);
Describe the bug
tried code below
            descriptor                 .Field(context => context.Screens)                 .Type<NonNullType<ListType<NonNullType<ListType<IntType>>>>>()                 .Name("screens");
hotchocolate throws exception
Unable to resolve type reference Output: HotChocolate.Types.NonNullType<HotChocolate.Types.ListType<HotChocolate.Types.NonNullType<HotChocolate.Types.ListType<HotChocolate.Types.IntType>>>>. -
at HotChocolate.Configuration.CompletionContext.GetType[T](ITypeReference reference)
at HotChocolate.Types.FieldBase2.OnCompleteField(ICompletionContext context, TDefinition definition)    at HotChocolate.Types.OutputFieldBase1.OnCompleteField(ICompletionContext context, TDefinition definition)
at HotChocolate.Types.ObjectField.OnCompleteField(ICompletionContext context, ObjectFieldDefinition definition)
at HotChocolate.Types.FieldBase2.CompleteField(ICompletionContext context)    at HotChocolate.Types.FieldInitHelper.CompleteFields[TTypeDef,TFieldType,TFieldDef](ICompletionContext context, TTypeDef definition, IReadOnlyCollection1 fields)
at HotChocolate.Types.ObjectType.OnCompleteType(ICompletionContext context, ObjectTypeDefinition definition)
at HotChocolate.Types.TypeSystemObjectBase1.CompleteType(ICompletionContext context)    at HotChocolate.Configuration.TypeInitializer.<CompleteTypes>b__42_0(RegisteredType registeredType)    at HotChocolate.Configuration.TypeInitializer.CompleteTypes(DiscoveredTypes discoveredTypes, TypeDependencyKind kind, Func2 action)
at HotChocolate.Configuration.TypeInitializer.CompleteTypes(DiscoveredTypes discoveredTypes)
at HotChocolate.Configuration.TypeInitializer.Initialize(Func`1 schemaResolver, IReadOnlySchemaOptions options)
at HotChocolate.SchemaBuilder.Create()
at HotChocolate.SchemaBuilder.HotChocolate.ISchemaBuilder.Create()
To Reproduce
define a new field with as code below
            descriptor                 .Field(context => context.Screens)                 .Type<NonNullType<ListType<NonNullType<ListType<IntType>>>>>()                 .Name("screens");
Expected behavior
Expect without error to run and schema would be like below as graphql-dotnet does generate;
type FooType {   screens: [[Int]!]!}

Hi @repal-ganut,
we actually left support for nested listed out of version 10. It is implemented with version 11 however.
Version 11 implements the GraphQL 2020 spec and we are still working on a lot of features there.


Remove duplication

As per code climate analysis, there are similar blocks here that can be refactored.

fixed most reported issues; the remaining relate to #23


Konflux update test-component-pac-azkjot

Pipelines as Code configuration proposal
To start the PipelineRun, add a new comment with content /ok-to-test
For more detailed information about running a PipelineRun, please refer to Pipelines as Code documentation Running the PipelineRun
To customize the proposed PipelineRuns after merge, please refer to Build Pipeline customization

Pipelines as Code CI/test-component-pac-azkjot-on-pull-request has failed.
Namespace: build-e2e-rwic-tenant
PipelineRun: test-component-pac-azkjot-on-pull-request-2lllc

Task Statuses:

StatusDurationName

✅ Succeeded
5 seconds
init

✅ Succeeded
16 seconds
clone-repository

✅ Succeeded
52 seconds
build-container

✅ Succeeded
8 seconds
sast-snyk-check

✅ Succeeded
9 seconds
apply-tags

✅ Succeeded
9 seconds
clamav-scan

✅ Succeeded
6 seconds
sbom-json-check

✅ Succeeded
9 seconds
push-dockerfile

✅ Succeeded
5 seconds
deprecated-base-image-check

❌ Failed
8 seconds
clair-scan

✅ Succeeded
15 seconds
ecosystem-cert-preflight-checks

✅ Succeeded
12 seconds
show-summary

✅ Succeeded
6 seconds
show-sbom

Failure snippet:
task clair-scan has the status "Failed":
Inspecting raw image manifest quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot@sha256:6a15166ac5dcad1c2ebd71db4a9348668211dadfeecd24dde742fd162832d61e.
time="2024-07-16T13:42:59Z" level=fatal msg="Error parsing image name \"docker://quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot@sha256:6a15166ac5dcad1c2ebd71db4a9348668211dadfeecd24dde742fd162832d61e\": reading manifest sha256:6a15166ac5dcad1c2ebd71db4a9348668211dadfeecd24dde742fd162832d61e in quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot: manifest unknown"
The raw image inspect command failed

Pipelines as Code CI/test-component-pac-azkjot-on-pull-request has failed.
Namespace: build-e2e-rwic-tenant
PipelineRun: test-component-pac-azkjot-on-pull-request-h5bsb

Task Statuses:

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
15 seconds
clone-repository

✅ Succeeded
29 seconds
build-container

✅ Succeeded
6 seconds
deprecated-base-image-check

❌ Failed
11 seconds
clair-scan

✅ Succeeded
6 seconds
clamav-scan

✅ Succeeded
7 seconds
sast-snyk-check

✅ Succeeded
18 seconds
ecosystem-cert-preflight-checks

✅ Succeeded
6 seconds
push-dockerfile

✅ Succeeded
5 seconds
sbom-json-check

✅ Succeeded
8 seconds
apply-tags

✅ Succeeded
7 seconds
show-sbom

✅ Succeeded
10 seconds
show-summary

Failure snippet:
task clair-scan has the status "Failed":
Inspecting raw image manifest quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot@sha256:2de88905d75b8f2bd86bc2eb13049d6d84910c0820485b0d504985a3417d4c10.
time="2024-07-16T13:44:42Z" level=fatal msg="Error parsing image name \"docker://quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot@sha256:2de88905d75b8f2bd86bc2eb13049d6d84910c0820485b0d504985a3417d4c10\": reading manifest sha256:2de88905d75b8f2bd86bc2eb13049d6d84910c0820485b0d504985a3417d4c10 in quay.io/redhat-appstudio-qe/build-e2e-rwic-tenant/build-suite-test-application-rafd/test-component-pac-azkjot: manifest unknown"
The raw image inspect command failed

Pipelines as Code CI/test-component-pac-azkjot-on-pull-request has successfully validated your commit.
Namespace: build-e2e-rwic-tenant
PipelineRun: test-component-pac-azkjot-on-pull-request-nfh7k

Task Statuses:

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
20 seconds
clone-repository

✅ Succeeded
39 seconds
build-container

✅ Succeeded
9 seconds
sbom-json-check

✅ Succeeded
8 seconds
push-dockerfile

✅ Succeeded
9 seconds
deprecated-base-image-check

✅ Succeeded
14 seconds
clair-scan

✅ Succeeded
12 seconds
ecosystem-cert-preflight-checks

✅ Succeeded
41 seconds
clamav-scan

✅ Succeeded
7 seconds
sast-snyk-check

✅ Succeeded
8 seconds
apply-tags

✅ Succeeded
7 seconds
show-sbom

✅ Succeeded
10 seconds
show-summary


Decimal.cmp/2 is deprecated. Use compare/2 instead

Environment

Elixir version (elixir -v): Erlang/OTP 23 [erts-11.1.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]

warning: Decimal.cmp/2 is deprecated. Use compare/2 instead
  lib/ecto/changeset.ex:2155: Ecto.Changeset.validate_number/6

Correct. We are keeping it for backwards compatibility but we will drop it soon. Thanks!

This should be closed due to #3504

Thanks @v0idpwn!


kernel: disable pci shpc hotplug for arm64

As shpchp used for pci hotplug on arm64 initialized
its bottom half work as a delay work for 5 seconds, pci bus
rescan triggered between up half and bottom half of shpc interrupt
handling will fail. so disable shpc and let bus rescan
to do the device hotplug on arm64.
Fixes: #499
Signed-off-by: Jianyong Wu<EMAIL_ADDRESS>@jodh-intel @Pennyzct @bergwolf @grahamwhaley

@devimc for input around bus rescanning pci and stuff I think....

As far as I know, x86 use ACPI to do pci device hotplug, so it will no this issue I think.

/test

@devimc help retest, thank you.

@jongwu - here you go...
/retest

@jodh-intel need retest, thank you!

/test


dependencies: sympy 1.11 compatibility

Small PR than fixes the newly introduced bug in sympy for _print_Add. We should keep that bug fix in our printer even once 1.12 is released so that we stay compatible with sympy 1.11

and ready to go

No there is still a bug with str(expr) looking into best way to fix it

Ok sorry for the delay, looks like we GTG.
Thecnically fixed in sympy (https://github.com/sympy/sympy/pull/24030) from original issue (https://github.com/sympy/sympy/issues/24022) but from looking at issues/PRs and links I thnk is safer to keep it internal.

@mloubout to be extra clear, I don't mind merging our PR now, so it's a go for me 👍 Above I was just assessing alternatives


fix: wrong var type in response_rewrite e2e

Please answer these questions before submitting a pull request

Why submit this pull request?

[x] Bugfix

[ ] New feature provided

[ ] Improve performance

[ ] Backport patches

Related issues
see https://github.com/apache/apisix/issues/5370

Codecov Report

Merging #754 (e2b0e9d) into master (da30386) will increase coverage by 68.30%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           master      #754       +/-   ##
============================================
+ Coverage   31.69%   100.00%   +68.30%
============================================
  Files          65         1       -64
  Lines        6639         1     -6638
============================================
- Hits         2104         1     -2103
+ Misses       4280         0     -4280
+ Partials      255         0      -255

Impacted Files
Coverage Δ

cmd/ingress/ingress.go

pkg/apisix/consumer.go

pkg/kube/translation/annotations.go

pkg/kube/translation/apisix_route.go

pkg/apisix/apisix.go

pkg/ingress/manifest.go

pkg/ingress/compare.go

pkg/apisix/cluster.go

pkg/apisix/schema.go

pkg/api/router/router.go

... and 56 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da30386...e2b0e9d. Read the comment docs.


optionsFromCapabilities() treading on layer [& config] specified image format?

I'm passing a WMTS capabilities object to WMTS.js : optionsFromCapabilities().
Near the end of the function, it builds a list of URLs. I do not have any OperationsMetadata in my capabilities so my URL list is empty.
The URL list is then built from the layer's collection of ResourceURL tags. As this is done in one function / scope, the .forEach used to iterate through the list is stamping on format. This ends up in function return result.
This behaviour doesn't match the function comment, and treads on both a) the first layer format b) the config format (if specified).
The code in question:
if (urls.length === 0) {
  requestEncoding = WMTSRequestEncoding.REST;
  l['ResourceURL'].forEach(function(element) {
    if (element['resourceType'] === 'tile') {
      format = element['format'];
      urls.push(/** @type {string} */ (element['template']));
    }
  });
}

If this behaviour is unintended, I'm happy to raise a PR / fix.
I'm not sure if this is compounded by my resource URLs being different image formats. This means the map rendered is somewhat Frankenstein, comprised of tiles of differing formats.

@gknapp feel free to submit a pull request (ideally with a new test in https://github.com/openlayers/openlayers/blob/master/test/spec/ol/source/wmts.test.js). Thanks!


Does this support django 1.4 version

I am trying to integrate this version of library for my django 1.4 version. got this error.
Traceback (most recent call last):   File "./django/core/handlers/wsgi.py", line 241, in __call__     response = self.get_response(request)   File "./django/core/handlers/base.py", line 178, in get_response     signals.got_request_exception.send(sender=self.__class__, request=request)   File "./django/dispatch/dispatcher.py", line 172, in send     response = receiver(signal=self, sender=sender, **named)   File "./django/db/__init__.py", line 68, in _rollback_on_exception     transaction.rollback_unless_managed(using=conn)   File "./django/db/transaction.py", line 144, in rollback_unless_managed     connection.rollback_unless_managed()   File "./django/db/backends/__init__.py", line 225, in rollback_unless_managed     self._rollback()   File "/usr/local/lib/python2.7/dist-packages/django_postgrespool2-0.2.0-py2.7.egg/django_postgrespool2/base.py", line 105, in _rollback AttributeError: 'DatabaseWrapper' object has no attribute 'wrap_database_errors'

Not working for me on Django v2.0 also.

This version doesn't support django 1.4


core-api: Introduce new plugin extension API

Fixes #3530
This adds the final building blocks, wires them up in the app, exporting a couple of new functions.
Planning to merge after release on the 17th and then start porting some plugins in master.
Some proper documentation to come either in this or followup PR as well, but we're still not pushing for usage of this quite yet.

Hm what's up with those type dependencies


Feature: Add request hooks

Allow users to pass request hooks.
Related issue: https://github.com/OHIF/Viewers/issues/2374
PR in OHIF: https://github.com/OHIF/Viewers/pull/2408

Example using node-retry to add retry functionality to XHR:
    const client = new api.DICOMwebClient({
      url: server.qidoRoot,
      headers: DICOMWeb.getAuthorizationHeader(server),
      errorInterceptor: errorHandler.getHTTPErrorHandler(),
      requestHooks: [
        (request, metadata) => {
          function faultTolerantRequestSend(...args) {
            const operation = retry.operation({
              retries: 10,
            });
            /**
             * retries: 5,
             * factor: 3,
             * minTimeout: 1 * 1000,
             * maxTimeout: 60 * 1000,
             * randomize: true
             */

            operation.attempt(function(currentAttempt) {
              const originalOnReadyStateChange = request.onreadystatechange;

              request.onreadystatechange = function() {
                originalOnReadyStateChange.call(request);
                if (request.status === 429 || request.status >= 500) {
                  operation.retry(new Error('Attempt failed!'));
                }
              };

              console.debug(`${metadata.url} (attempt: ${currentAttempt})`);
              request.open(metadata.method, metadata.url, true);
              originalRequestSend.call(request, ...args);
            });
          }

          const originalRequestSend = request.send;
          request.send = faultTolerantRequestSend;

          return request;
        },
      ],
    });

@igoroctaviano this is super cool, but not easy to understand:
const pipe = functions => args => functions.reduce((arg, fn) => fn(arg), args)

If we go for this approach, I would suggest adding

Documentation (in the docstring and in the source code) and several examples to describe and demonstrate how this feature is intended to be used
Runtime type checks (instanceOf, typeof, etc.) to avoid cryptic error messages, for example if a caller passes an object that is not callable
Unit tests
Type annotations (we can use the ones we already have in SliM and include them in the library)

Yes, and while we need to be idiomatic to the languages, it would be great if the functionality and perhaps even some of the naming were consistent with the discussion here:
https://github.com/MGHComputationalPathology/dicomweb-client/pull/50

@igoroctaviano could we call the option requestHooks rather than requestInterceptors? I think the term hook would be more intuitive and is used by other libraries for similar purposes. What do you think?

Sure, I took it from axios library, where they call these request interceptors but I'm fine with hooks, updated.

@hackermd just pushed an example and updated the existent one to point to dcmjs server.

It needs to be rebased but otherwise is this ready to be merged?

It needs to be rebased but otherwise is this ready to be merged?

Yes, just waiting review + merge.

Any further comments @hackermd or should we merge?

@pieper can you please merge this? Thanks!

@pieper can you please merge this? Thanks!

@Punzo @igoroctaviano I merged the PR. Let me know if we should draft a new release.

@pieper can you please merge this? Thanks!

@Punzo @igoroctaviano I merged the PR. Let me know if we should draft a new release.

yes, please! thanks!

@Punzo @igoroctaviano I released version 0.8.0 (https://github.com/dcmjs-org/dicomweb-client/releases/tag/v0.8.0) and uploaded the new version of the package to npm (https://www.npmjs.com/package/dicomweb-client/v/0.8.0)


Mock async methods are not returning corrent values for toBeCalledTimes

Only the first spy method call is correctly "spied".
Steps to reproduce:
// testing.ts
export interface PartyProvider {
  getPartyType: () => Promise<void>;
  getSongs: () => Promise<void>;
  start: () => Promise<void>;
}

export class Testing {
  constructor(private partyProvider: PartyProvider){}

  async execute(): Promise<void>{
    await this.partyProvider.start();
    await this.partyProvider.getSongs();
    await this.partyProvider.getSongs();
    await this.partyProvider.getPartyType();
  }
}

// testing.spec.ts
import { mock, mockReset } from 'vitest-mock-extended';
import { describe, test, expect, it, beforeEach } from "vitest";
import { PartyProvider, Testing } from "./testing"

describe('Party Tests', () => {
    test('testing called times in mock', () => {
        const mockProv = mock<PartyProvider>();

        const sut = new Testing(mockProv)

        expect(sut.execute()).resolves

        expect(mockProv.start).toBeCalledTimes(1); // should be true
        expect(mockProv.getSongs).toBeCalledTimes(2); // should be true
        expect(mockProv.getPartyType).toBeCalledTimes(1); // should be true

    });
});

when i run vitest i got error:
 FAIL  testing/test.spec.ts > Party Tests > testing called times in mock
AssertionError: expected "spy" to be called 2 times
 ❯ testing/test.spec.ts:14:35
     12|
     13|         expect(mockProv.start).toBeCalledTimes(1);
     14|         expect(mockProv.getSongs).toBeCalledTimes(2);
       |                                   ^
     15|         expect(mockProv.getPartyType).toBeCalledTimes(1);
     16|

  - Expected   "2"
  + Received   "0"

if i invert the call order to first call one time this.partyProvider.getSongs(); i got this error
 FAIL  testing/test.spec.ts > Party Tests > testing called times in mock
AssertionError: expected "spy" to be called 1 times
 ❯ testing/test.spec.ts:13:32
     11|         expect(sut.execute()).resolves
     12|
     13|         expect(mockProv.start).toBeCalledTimes(1);
       |                                ^
     14|         expect(mockProv.getSongs).toBeCalledTimes(2);
     15|         expect(mockProv.getPartyType).toBeCalledTimes(1);

  - Expected   "1"
  + Received   "0"

My environment:
"vitest": "^0.29.2",
"vitest-mock-extended": "^1.0.10"
Node: v16.15.0

It seems to be related to the .resolves  assertion. It will work out if run like this
describe('ToBeCallesTimes', () => {
    test('should report correct called times on mock', async () => {
        //given
        const mockProv = mock<PartyProvider>();
        const sut = new Testing(mockProv);

        //when
        await sut.execute();

        // then
        expect(mockProv.start).toBeCalledTimes(1);
        expect(mockProv.getSongs).toBeCalledTimes(2);
        expect(mockProv.getPartyType).toBeCalledTimes(1);
    });
});

I'm trying to figure out why it behaves like this.

Yeah, it seems to work now, mb, i started learning vitest and ts recently.


Add AVIF decoding support

With #599, we now support AVIF encoding, provided through the imagecrate by ravif.
It would be great to also support AVIF decoding. Here, the path forward is not entirely clear yet. A pure Rust crate doesn't exist yet, which may be a blocker for inclusion in image itself. We could however use dav1d with rust bindings ourselves (and ask about the opinion of the image authors on using dav1d bindings, perhaps behind a feature).

Opt-in support for decoding AVIF was added to image in 0.23.13, which was merged in #760.


Tighten Selection Management in GridModel

Disable selection more explicitly in rendered agGrid
Always round-trip selection activity in ag-Grid back to grid at least once.

Hoist P/R Checklist
Pull request authors: Review and check off the below. Items that do not apply can also be
checked off to indicate they have been considered. If unclear if a step is relevant, please leave
unchecked and note in comments.

[x] Caught up with develop branch as of last change.
[x] Added CHANGELOG entry, or determined not required.
[x] Reviewed for breaking changes, added breaking-change label + CHANGELOG if so.
[x] Updated doc comments / prop-types, or determined not required.
[x] Reviewed and tested on Mobile, or determined not required.
[x] Created Toolbox branch / PR, or determined not required.

If your change is still a WIP, please use the "Create draft pull request" option in the split
button below to indicate it is not ready yet for a final review.

Pull request reviewers: when merging this P/R, please consider using a squash commit to
collapse multiple intermediate commits into a single commit representing the overall feature
change. This helps keep the commit log clean and easy to scan across releases. PRs containing a
single commit should be rebased when possible.

Not quite ready to merge this -- investigating one subtlety with the group rows we are skipping in the new keyboard handling code.


Add AZ failure: ECS

See rudpot/scenario-az branch for existing work

@rudpot , I will pick this up.


Add Auto-Variant Counting for VCF Files and Multi-Collection Processing

This PR includes the following updates:

Adds an auto-variant counting feature for VCF files.
Adds a script to process all collections simultaneously.

Hi @mansego ! Thank you very much for this. I tested everything and worked fine. The new features will help a lot on reducing the time for converting all the files, a big appreciation to your work. On the other hand, we will continue testing the tool after merging the PR to check everything works like expected to continue improving the tool.
Thank you for this awesome contribution,
Oriol


Can't find Apple Watch 5

My series 7 shows up fine but my girlfriend's series 5 doe snot. Any tips on how I can get both working? I changed the Lamba to the code suggested in a similar thread and captured the following here is some info I've gathered
[V][api.connection:763]: Hello from client: 'Home Assistant 2022.2.5 (<IP_ADDRESS>)'
[D][api.connection:779]: Home Assistant 2022.2.5 (<IP_ADDRESS>): Connected successfully
[D][homeassistant.sensor:024]: 'input_number.upstairs_harssi_present': Got state -77.00
[V][sensor:074]: 'HA RSSI Present Value': Received new state -77.000000
[D][sensor:125]: 'HA RSSI Present Value': Sending state -77.00000  with 1 decimals of accuracy
[V][mqtt:405]: Publish(topic='upstairs/sensor/ha_rssi_present_value/state' payload='-77.0' retain=1)
[D][homeassistant.sensor:024]: 'input_number.upstairs_harssi_not_present': Got state -85.00
[V][sensor:074]: 'HA RSSI Not Present Value': Received new state -85.000000
[D][sensor:125]: 'HA RSSI Not Present Value': Sending state -85.00000  with 1 decimals of accuracy
[V][mqtt:405]: Publish(topic='upstairs/sensor/ha_rssi_not_present_value/state' payload='-85.0' retain=1)
[D][ble_adv:052]: Possible Apple Watch: data[2]=0x0e data[3]=0x18
[D][ble_adv:053]:   rssi -86
[D][ble_adv:054]:   mac 5E:A9:00:64:95:97
[D][ble_adv:052]: Possible Apple Watch: data[2]=0x0e data[3]=0x18
[D][ble_adv:053]:   rssi -87
[D][ble_adv:054]:   mac 5E:A9:00:64:95:97
[D][ble_adv:052]: Possible Apple Watch: data[2]=0x0e data[3]=0x18
[D][ble_adv:053]:   rssi -86
[D][ble_adv:054]:   mac 5E:A9:00:64:95:97
[D][ble_adv:052]: Possible Apple Watch: data[2]=0x0e data[3]=0x18
[D][ble_adv:053]:   rssi -86
[D][ble_adv:054]

I don't have an answer for you. Though I would like to see your whole config settings. I have series 3 and 7 but atm my setup isn't working properly. I have posted it here. I am not sure what I am doing wrong.

Hi @Zipties -
It looks like your girlfriends series 5 is being detected, but possibly was locked at the time? Usually data[3]=0x18 means the watch is locked, and it changes to 0x98 when unlocked (i.e. on wrist and pincode typed in.) Or does the log output always say data[3]=0x18 even when unlocked?
On line 60 of the yaml, you can change it to
if (data_flags == 0x98 || data_flags == 0x18) {

This will detect both locked and unlocked devices.

I confirmed it wasn't locked, and even borrowed it for a bit to make sure. After working on it longer, I believe the device in the logs may not have been hers, but I am sure the screenshots of the BLE scanner app was her watch.

Was the watch detected when you changed the lambda to look for both 0x28 and 0x18?
If you want to be fairly sure the watch you are testing is the one being detected, bring it within 5-10 cm of the ESP32, and the RSSI should be about -45 dBm.


Feature request: Prevent manipulation

When using the plugin with a limited emoji set for users to choose from, it can very easily be manipulated:
A user simply has to edit the title-attribute of any of the displayed emojis in the picker to an emoji-shortcode of his choice (doable with the browser developer tools), then click the altered emoji, and without any checks, his custom emoji is inserted as a reaction to the post.
This allows manipulation and trolling, and since there are no ways for the staff to edit or remove reactions, the selected emoji should be checked against the list of allowed emojis, before it is added to the post.

@DoomDesign Yes, fair point. Is it something you've seen in the wild? In any event, I'll look at adding serverside validation for this.

@angusmcleod No, we discovered this while checking the plugin on a staging forum, to see if there are any issues that could get abused in a larger, more tech-savvy community like ours.
Thank you for looking into this!

Any possibility that you add this check in the plugin? Not only users can inject maliciously invalid emote code to display what they want, but users using translation browser extension also do it without understanding why they post broken reactions.
I had a few "wild" examples, but since threads are now locked there is no more reactions on them. But it happens in the wild sometimes.

@Zorrototo I'll take a look next week, but keep in mind this is no longer actively maintained and there is an official Reactions Plugin now.


unplugging and replugging webcam breaks webcam

Hello,
If i use this library to have take input from a usb webcam everything works fine. But when i unplug the webcam and replug it I am stuck with this message:

And in the logs I see this:

How can I stop the webcam from being disposed after being unplugged? Because I think after I replug my webcam it should reconnect to the webcam java input

Hello there @TamjidTayeb.
Please take a look at The Webcam Capture API Handbook section 6.1 where Webcam Discovery Service is described.
It explains how to access it, stop, and start it again.
By stopping discovery service your webcam won't be disposed, but there is no guarantee if it will be re-instantiated again after it's re-connected. This is completely random and works differently on a different operating systems. I do remember cases where application crashed when I tried to do thinks on a webcam which was disconnected. So, yeah, I'm sure you will have completely new field of scenarios to research and test to identify the problematic ones :)
From my point of view I would solve this by reacting to the disconnection event, e.g. close the panel, and open it again or re-create it again with a new webcam instance after the webcam is connected again.
Here is the sample code to listen for connection/disconnection events (from the handbook).
List<Webcam> webcams = Webcam.getWebcams();
for (Webcam webcam : webcams) {
    // print all currently connected webcams
    System.out.println(webcam.getName());
}

// will print message whenever webcam is connected or disconnected
// the Webcam here is a class, not a webcam instance (addDiscoveryListener is a static method)
Webcam.addDiscoveryListener(
    new WebcamDiscoveryListener() {
        public void webcamFound(WebcamDiscoveryEvent event) {
            System.out.println(event.getWebcam().getName() + " connected");
        }
        public void webcamGone(WebcamDiscoveryEvent event) {
            System.out.println(event.getWebcam().getName() + " disconnected");
        }
    }
);

Hi,
Thanks for your quick reply. I have tried to use the webcamdiscoverylistener to cleanly reconnect but i'm still getting issues, could you please have a look at this and let me know what I'm doing wrong?
https://stackoverflow.com/questions/66485557/how-to-handle-webcam-unplug-then-replug-for-sarxos-java-webcam

@TamjidTayeb By looking at your code I suppose this happened because you did not remove old WebcamPanel from JFrame and therefore it displays the old data. You created new one in webcamFound(..) but this is also wrong, because you only created it and you did not add it to the JFrame. If component is not added to the JFrame it will not be displayed (you won't see it).
    @Override
    public void webcamFound( WebcamDiscoveryEvent event ) {
        if( webcam == null ) return;
        Webcam newWebcam = event.getWebcam();
        if( newWebcam.getName().equals( webcam.getName()) ) {
            close();
            webcam = Webcam.getWebcamByName( webcam.getName() );
            webcam.open();
            panel = new WebcamPanel( webcam ); // <--- you need to remove old from the frame and add new
        }
    }

You should do similar thing as you did with original WebcamPanel:
add( panel );

I'm closing this ticket. Feel free to reopen it in case of additional questions in regards to the discussed matter.


Add TexlTest for Error function

We recently had a bug added to the Error function (in #1206), and it was fixed in the next commit (#1224). This adds some additional tests to prevent the error from happening in the first place.

✅ No public API change.


Polarities: subtyping for datatypes

(x:int{x>1} * y:int{y>1}) is not a subtype of (int * int)

Was thinking with @victor-dumitrescu the other day that polarities can be seen as a form of relational refinements. For instance, a positive type constructor like list could have type:
list: a:Type -> Tot (b:Type{L a <: R a => L b <: R b})

where <: is an internalization of the subtyping judgment. Would doing polarities this way be a crazy idea? :)

#593 has some good examples where polarities are really needed.

Was discussing with @alejandroag the other day that internalizing subtyping (or any other judgment) is tricky, since the logical encoding is sound but incomplete. It seems that for the negative positions in a formula (like L a <: R a above) we would need an unsound but complete encoding. Such encodings that are "sound for counterexamples" exist (e.g. in Nitpick and Nunchaku), but it's very unclear how well one can mix the two kind of encodings. And anyway, there are better reasons to implement "sound for counterexamples" encodings ... if we ever do that, we do further experiment with mixing things.

@aseemr Wondering whether we will also get this from your polarities check for inductives.

@kyoDralliam has discovered a counterexample/paradox in the literature showing that enabling subtyping for datatypes would currently be unsound. He knows about this for a while, still it would be good to write this all down somewhere (e.g. here). From what I understand, a potential solution for this would be to remove type arguments to datatype constructors.

@catalin-hritcu @kyoDralliam Is that the counterexample from Luo's note about Russell-style universes (http://www.cs.rhul.ac.uk/~zhaohui/universes.pdf)?

It's that same counterexample. Here's is what it looks like for F* + subtyping for pairs.
type a * b : Type = | MkPair : #a:Type -> #b:Type -> a * b

let elim (a b : Type) (c : a * b -> Type) (f: x:a -> y:b -> C (MkPair #a #b x y)) (p: a * b) : C p
= match p with | MkPair #a #b x y -> f x y

let subject_reduction_failure (x y:nat) (c:int * int -> Type) (f : x:int -> y : int -> c (MkPair #int #int x y)) : c (MkPair #nat #nat x y) = elim int int c f (x,y)

this is well typed because (x,y) <: int * int but reduces to f x y which is of type c (MkPair #int #int x y) and not c (MkPair #nat #nat x y)

The current fix proposal discussed with @theolaurent and @kyoDralliam is to prevent matching inductive parameters. In Kenji's counterexample this will make it impossible to write (function (MkPair #a #b x y) -> a).

Discussed this again with @kyoDralliam and @mtzguido, and the main challenge for adding datatype subtyping is that now constraints such as a * b <: c * d will no longer reduce to equalities a = c and b = d but to subtyping a <: c and b <: d, which might be less efficient. I vaguely remember discussions with @nikswamy in Redmond in fall 2016, but I'm not sure what the conclusion was on this.

I'm now getting errors on (function (MkPair #a #b x y):
This pattern (_) binds an inaccesible argument; use a wildcard ('_') pattern

It seems a recent change from July by @nikswamy : https://github.com/FStarLang/FStar/commit/8939c3e9c6896a0ba9ecf0d7372d4c1c12edd483#diff-9a12b72662f95654a4c5ab46f41257b3R99
Does this completely prevent matching on inductive parameters, as we would need for fixing this issue? :)

@catalin-hritcu Confused by your last remark. Projecting the parameters of a datatype is exactly what we want to prevent. Why do you want to write it?

@nikswamy Sorry if this was unclear. Completely preventing matches on inductive parameters is indeed what we need for fixing this issue. And the question is whether your change completely prevents such matches already.

Status update on this issue:

As required for soundly implementing subtyping for datatype parameters, pattern matching on parameters is now disallowed (see comment above) and a fix by Nik changed the SMT encoding to forbid injectivity for some parameters of inductive type. The current issue would require extending this to all parameters, which Nik tried but that lead to regressions.
There has been no work so far on implementing subtyping for datatypes, and one worry Nik had is that this could impact typechecking performance, since subtyping constraints on the type parameters are harder to solve than the current equality constraints.

Just for the record, wanted to mention that @kyoDralliam and @theolaurent are doing that research. For instance, the French readers can see this: https://kenji.maillard.blue/Writings/mettons-de-l-ordre-dans-cic-171220.pdf

... and a more recent extended in English: https://types22.inria.fr/files/2022/06/TYPES_2022_paper_62.pdf

Let me mention that there is a now paper about the research above that will appear at ESOP'24:
Definitional Functoriality for Dependent (Sub)Types. Théo Laurent, Meven Lennon-Bertrand, Kenji Maillard.


Document how to enable serial console tty

I wanted to enable serial console tty in NixOS on a server, so i can still use the terminal even when i messed up the networking config. Sadly, it was not documented in the NixOS Manual how to do it.
So please add documentation for that to NixOS manual.
I found some hints in descriptions of options like systemd.services.<name>.enable:

Note that enable=true does not make a unit start by default at boot; if you want that, see wantedBy.

and systemd.services.<name>.wantedBy:

Units that want (i.e. depend on) this unit. The standard way to make a unit start by default at boot is to set this option to [ "multi-user.target" ].

So the needed configuration looks like this:
  # start tty0 on serial console
  systemd.services."serial-getty@ttyS0" = {
    enable = true;
    wantedBy = [ "getty.target" ]; # to start at boot
    serviceConfig.Restart = "always"; # restart when session is closed
  };

You can also use this:
{
  boot.kernelParams = [
    "console=ttyS0,115200"
    "console=tty1"
  ];
}

Required systemd unit will start automatically.
If you want to see boot log in serial console "console=ttyS0,115200" should be after "console=tty1".

Required systemd unit will start automatically.

Will it get stopped when i close the session, like in the solution above?
I had to explicitely enable restarts, but i think that can get added upstream to the generator template.
serviceConfig.Restart = "always";

Required systemd unit will start automatically.

Will it get stopped when i close the session, like in the solution above?
I had to explicitely enable restarts, but i think that can get added upstream to the generator template.
serviceConfig.Restart = "always";

Yes it will. Restarts is enabled by default, i didn't change anything.
> systemctl cat<EMAIL_ADDRESS>                                                                                misuzu@serara 10:41:47
#<EMAIL_ADDRESS>#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Serial Getty on %I
Documentation=man:agetty(8) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html
BindsTo=dev-%i.device
After=dev-%i.device systemd-user-sessions.service plymouth-quit-wait.service getty-pre.target

# If additional gettys are spawned during boot then we should make
# sure that this is synchronized before getty.target, even though
# getty.target didn't actually pull it in.
Before=getty.target
IgnoreOnIsolate=yes

# IgnoreOnIsolate causes issues with sulogin, if someone isolates
# rescue.target or starts rescue.service from multi-user.target or
# graphical.target.
Conflicts=rescue.service
Before=rescue.service

[Service]
# The '-o' option value tells agetty to replace 'login' arguments with an
# option to preserve environment (-p), followed by '--' for safety, and then
# the entered username.
ExecStart=-/sbin/agetty -o '-p -- \\u' --keep-baud 115200,38400,9600 %I $TERM
Type=idle
Restart=always
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

[Install]
WantedBy=getty.target

#<EMAIL_ADDRESS>[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/k51wcq4lqkj3la51jlpnsnnly5wwhc8i-glibc-locales-2.30/lib/locale/locale-archive"
Environment="PATH=/nix/store/hzvl3dvv8651iqlb5g6gq5hzjzmhjn7m-coreutils-8.31/bin:/nix/store/rphcpivxfm4blw36ki262yrmqygw9pcp-findutils-4.7.0/bin:/nix/store/pmwf1fz03c5say5fnqk4m748rjb8kdy7-gnugrep-3.4/bin:/nix/store/9k90a16frvqdcxzkhxcjsi2>
Environment="TZDIR=/nix/store/msbzkf26r6v5nypj1dspjk5jfnvl9l8y-tzdata-2019c/share/zoneinfo"

X-RestartIfChanged=false

ExecStart=
<EMAIL_ADDRESS>agetty --login-program /nix/store/s30n68sarsjsclb9l8risvq9zvg25gg1-shadow-4.8/bin/login  %I 115200,57600,38400,9600 $TERM

You should only need to enable serial consoles by adding them to the kernel cmdline as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-608084218.
systemd will automatically create<EMAIL_ADDRESS>symlinks while parsing the kernel cmdline - see https://github.com/systemd/systemd/blob/master/src/getty-generator/getty-generator.c.

I have bootet from usb, but it seems the minimal image also don't starts a serial console by default. It would help me right now. Would that has any negative effect when the hardware has no serial port, like failed systemd unit?
We could also detect serial console hardware from nixos-generate-config and activate it when available? My colluege said it works on other systems by default, like debian.

Yeah, enabled serial console on installation cd/usb/netboot would be great!

We might want to look at https://github.com/NixOS/nixpkgs/issues/30760#issuecomment-416093820 and the references there.

You can also use this:
{
  boot.kernelParams = [
    "console=ttyS0,115200"
    "console=tty1"
  ];
}

Required systemd unit will start automatically.
If you want to see boot log in serial console "console=ttyS0,115200" should be after "console=tty1".

This doesn't work for me. I'm running a NixOS VM via virsh and this would be needed for virsh console nixosvm to work. The need arises from the VM also having LUKS active. So, while I could use virt-viewer to enter the LUKS passphrase, I think using virsh console would be more straightforward, but for that the<EMAIL_ADDRESS>needs to start on the VM, but the systemd unit is not generated even though I have the above suggested in the VM's configuration.nix. The VM is running 20.03.

We're mixing two things here.
The initial issue is about serial console access for a running system. That can be archieved by adding boot.kernelParams as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-608084218.
Entering crypto keys during early boot is something different - we currently don't have systemd in initramfs yet, and gettys are spun up much later.
However, this should solve the initial issue, recovering from breaking the network configuration (which should probably be added to the manual, if it's not already).
The current existing tooling can currently only ask on one console (and which one might depend on the order). I'd consider asking on the graphical console a sane default if there's a graphical output. I didn't yet debug on how these options need to be set if you want to prefer the serial console for this question. @devhell if you want to do some experiments and contribute that to the documentation, that'd be appreciated!
Getting systemd into initramfs, and building systemd with cryptsetup support would allow asking for the crypto volume password using the systemd-ask-password mechanism, which should ask on all serial consoles - so just adding the consoles to boot.kernelParams should work some time in the future - but it'll still take some time till then.

Thanks @flokli. I'm sorry for mixing those two things up. I'll elaborate a bit. I was under the impression that adding boot.kernelParams to the VM's configuration would allow me to execute virsh console on the host to access the VM's console and enter the LUKS passphrase to allow the VM to finish its boot. After I added boot.kernelParams I rebuilt the VM and rebooted it, virsh console would not give me a console into the VM. So I entered the passphrase via virt-viewer and checked virsh console but I would still not get anything back from the console once the boot completed. I tried to verify via systemctl that serial-getty@ttys0 had been generated, but it had not.
I then used @davidak's solution instead and added systemd.services."serial-getty@ttyS0" to the VM's configuration, rebuild, reboot, and still no virsh console to get to the LUKS passphrase input, but after using virt-viewer to enter the passphrase virsh console started showing the rest of the boot process and subsequent login prompt, etc. Checking for serial-getty@ttys0 was also successful.
I then checked if enabling GRUB's early console activation would work as stated in 20.03's manual, using:
boot.loader.grub.extraConfig = ''
      serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
      terminal_input --append serial
      terminal_output --append serial
'';

This worked and using virsh console I was able to see the GRUB boot menu and then the LUKS passphrase prompt. I then entered the passphrase and the boot continued as expected and shortly after a login prompt was available on the console.
Just to test things out, I've then removed the explicit addition of systemd.services."serial-getty@ttys0", rebuild, rebooted, and confirmed that only the GRUB menu was visible via virsh console and no LUKS passphrase prompt. I then added the original solution via boot.kernelParams, rebuild, rebooted, and now I suddenly also got the LUKS passphrase prompt on the console. So I checked if serial-getty@ttys0 was generated, and now it was.
Finally, I removed GRUB's extraConfig so that only boot.kernelParams was left over, rebuild, rebooted, and now virsh console gives me no GRUB menu, but does give me a LUKS prompt.
I don't know why it's behaving like this, but I hope someone might come up with an idea and finds this write-up useful.
Thank you!

@devhell with just the boot.kernelParams being set, did you check with systemctl status if there was any getty spawned on the serial console?

@flokli, the first time I set boot.kernelParams nothing was spawned, then I added systemd.service."serial-getty@ttys0" and getty was spawned as I would have expected using boot.kernelParams. However, after removing systemd.service."serial-getty@ttys0", rebuilding and rebooting, the getty was still being spawned, so I'm not sure why it wasn't doing it the first time around before I added systemd.service."serial-getty@ttys0".

Can you provide a minimal configuration snippet producing an image that I can debug with libvirt?
This "works after some reboots" behaviour is something I'd really like to take a closer look at.

Here you go, I hope this is what you had in mind: https://gist.github.com/devhell/d23b0d09470894060645ad13b35e4c3c
The VM is LUKS encrypted, nothing fancy, example here, with UUIDs removed: https://gist.github.com/devhell/07df7d1902ae2ab4af6456a8c6e0989a
Hope that helps and thank you for looking into this.

Heya @flokli, just wondered if you had been able to test this?

Hey! Sorry for the silence - didn't get to this until now.
I slightly updated your configuration, and used the build-qcow2.nix to build a nixos.qcow2 image:
nix-build '<nixpkgs/nixos>' -A config.system.build.qcow2 --arg configuration "{ imports = [ ./build-qcow2.nix ]; }" -I nixpkgs=/path/to/my/nixpkgs

I copied nixos.qcow2 from the resulting store path, and used "Import existing disk image" from virt-manager. I selected "Generic default" as Operating system, and manually increased memory and cpu, but selecting "NixOS Unstable" also seemed to have the same effect.
I could both see and update Grub counting down at View -> Text Consoles -> {Serial 1, Graphical Console Spice}.

Without console=tty0 (as in your example) at the beginning of boot.kernelParams, I saw all Linux' boot output (including that from stage1/2, which would ask for the luks passphrase) at the serial console.
If I added console=tty0 at the front, that output appeared on the graphical console instead.

In both cases, getty did start up on both consoles after boot was finished, and I didn't experience different behaviour across reboots as described in https://github.com/NixOS/nixpkgs/issues/84105#issuecomment-654844464.
stage1/2 seems to only be able to ask for a password on one console.
If I removed console=ttyS0… from boot.kernelParams, there was no output during linux booting, and no getty was spawned there (which makes sense, given systemd starts gettys on all the ttys, and parses the kernel cmdline for additional consoles).
TLDR:

Grub can be used simultaneously from serial(s) and graphical consoles, using the boot.loader.grub.extraConfig from above
systemd will spin up gettys (and spin them up again if you exit your session) for all virtual terminals and consoles passed via boot.kernelParams (see http://0pointer.de/blog/projects/serial-console.html for details)
stage1/2 can only ask for a password on one console, it'll pick the first one from boot.kernelParams.

@devhell can you reproduce this?

Hey @flokli,
Thank you very much for testing this. Yes, I can reproduce it, and now it also makes sense. :)
Much appreciated!

Awesome. If you can put this somewhere into the Wiki, this'd be :heart:  :wink:

Hi, has there been any updates on starting a serial getty?
I am not getting a serial getty despite having this:
    boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ];

But in any case, I would like to have additional serial gettys on serial ports that are not included in the above kernelParams.  Options?
Some context:

I am generating an .iso image that needs these serial capabilities.
An embedded device will boot this iso image for software installation.  It does have video/usb, but we'd like to be able to use a serial port too.

Hey, I'm trying to create VMs using nix. I've followed these steps and added:
boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ]; .
However, when I try to access a vm via virsh console I am unable to interact with the VM and instead get dropped into a console with logs from systemd units. Is there a way to get an interactive console? Unfortunately, I don't have access to any GUI and networking isn't set up.
Any help would be appreciated!

Hey, I'm trying to create VMs using nix. I've followed these steps and added: boot.kernelParams = [ "console=ttyS0,115200" "console=tty0" ]; . However, when I try to access a vm via virsh console I am unable to interact with the VM and instead get dropped into a console with logs from systemd units. Is there a way to get an interactive console? Unfortunately, I don't have access to any GUI and networking isn't set up.
Any help would be appreciated!

You need to figure out what tty "virsh console" is actually using and make sure a getty is started on that one.

Those defaults work quite well for servers and virtual machines: https://github.com/numtide/srvos/blob/main/nixos/common/serial.nix
They are inspired by ubuntu and alpine + some nicer defaults to set terminal size and defaults on serial consoles

For some reason, simply adding the console kernel parameter for my raspberry pi didn't work. I'm not sure why, but systemd didn't start the serial-getty service on its own. I was able to enable the GPIO serial console using the following configuration:
  # Enables serial console through the GPIO.
  boot.kernelParams = [ "console=ttyS1,115200" ];
  systemd.services."serial-getty@ttyS1" = {
    enable = true;
    serviceConfig.Restart = "always";

  };

Note how I use ttyS1 instead of ttyS0.

I've been running into these issues as well. I think the direction these days is to not set any console-related kernel cmdline options,  so the device tree defaults can take over.
I can send a PR for this, but we need to test it doesn't break console on some common boards that might be missing it.


Get boom deployment as kustomizable resource

It would be nice to actually have the yaml definition for boom itself (the jumpstart) as a kustomize yaml. Then one could just create a simple kustomization yaml and reference the repository:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: caos-system

resources:
  - github.com/caos/boom/build/.......?ref=v1.4.2

# optional stuff

migrated to https://github.com/caos/orbos/issues/145


$emit events on user input

Feature Request
In order to respond to user input actions, currently you have to set a watch function on the model. However this also catches programmatic model updates, and can't discern whether the update was by user action or the program.
VFG should $emit an 'input' event on any user input; no arguments are required, although it would be nice to get an argument giving the field name in the model that changed.
This is helpful when a server provides form input updates asynchronously.

Can you not use the model-updated event?
<vue-form-generator ... @model-updated="onModelUpdated" />

onModelUpdated(model, schema) {
// model and schema of update
},

I believe 'model-updated' is emitted for any model change, not solely user input in the form? If so, it has the same limitation as a watch function on the model, described above.
Also I don't see any events documented in https://icebob.gitbooks.io/vueformgenerator/content/

Model updated is called by VFG whenever it processes user input, fairly sure it’s not called by reactive updates.  Though, you could test this.
I’m also not sure how an input event being fired helps with server related updates?

A quick test seems to confirm that @model-updated is emitted solely on user input. Now you just need to document that! :-)
One may need to take different actions for updates to the model pushed from a server vs input by user; e.g. inform the user that the form has changed. Hence my req for an input-only event.

As the developer, shouldn’t you know the update was caused by the server response ... allowing you to have the opposite of what you’re looking for, negating the request and solving the issue?
As for documentation, this is an actively growing project and the source is available... that’s how I found the model updated.  If you have the time, we’d welcome any contributions you could make to he documentation.
Thanks.  I’m going to close this issue now as it appears resolved.

I suggest re-opening this issue until the documentation is rev'd for @model-updated.
Do the docs need a new page for events, and if not which page would they belong to?
I'll try to submit a patch when I have time, but that will be months from now. If I were in your shoes I'd update the docs ASAP so I wouldn't have to answer this Q again :-)


StandardJS

Set up standard.js as linter. we need to make sure it works nicely with Prettier (might require some non-trivial settings)

I have started on this task. Most of the challenge so far has been around existing pieces of the build infrastructure. Is yarn run test-all intended to be the correct place to add this check?

Thanks for taking this. see also my comment to your PR here
test-all is not implemented yet, we only use test for now.
You are right that this is a bit challenging and there should be a clear separation what each tool is responsible for. Standard should not affect any styling rules like indentation and quoting and should only serve as a JS linter.

https://github.com/sheerun/prettier-standard


Use external pypolyclip package

The PR removes the vendored polyclip code and uses the external pypolyclip package, now available on PyPI.
cc: @Russell-Ryan, @rosteen

The CI tests were failing due to pre-existing codestyle failures.  I went ahead and fixed those in the PR so tests would pass.

The test failure is unrelated.


Peering Request

Full Name
YUZHEN NETWORK INC.
ASN
AS211390
PeeringDB Integration

[X] My PeeringDB Entries are current
[ ] Peer at all available exchanges (and skip the Peer Address(es) field below)

Peer Address(es)
2a12:dd47:f601::1
2a12:dd47:f602::1
2a12:dd47:f603::1
2a12:dd47:f604::1
2a12:dd47:f605::1
2a12:dd47:f606::1
2a12:dd47:f607::1
Multihop

[X] Multihop

Code of Conduct (TODO)

[X] I agree to follow this project's Code of Conduct

This request is somewhat problematic.  I believe we could peer at a regional exchange, but I do not see any overlap at exchanges where we exist.  For multi-hop however, I believe the hop-count and TTL will make these not feasible.


Issue SCC v0.7.2 on Openshift

Error:
"container has runAsNonRoot and image will run as root (pod: "dynakube-ea-oneagent-5lgw5_dynatrace(f60eda44-a753-4581-b2a5-8ba4b7819d8b)", container: dynatrace-oneagent)"
Deployed CR:
apiVersion: dynatrace.com/v1beta1 kind: DynaKube metadata:   name: dynakube-ea   namespace: dynatrace spec:   activeGate:     capabilities:       - routing       - kubernetes-monitoring       - dynatrace-api     resources:       limits:         cpu: 2500m         memory: 1.5Gi       requests:         cpu: 1500m         memory: 1Gi   apiUrl: 'https://server123.de/e/8e6300ef-656b-4fe0-92ab-cce168a73f2b/api'   oneAgent:     classicFullStack:       args:         - '--set-host-group=mygroup'       oneAgentResources:         limits:           cpu: 500m           memory: 1.5Gi         requests:           cpu: 300m           memory: 512Mi       tolerations:         - effect: NoSchedule           key: node-role.kubernetes.io/master           operator: Exists   trustedCAs: dynakube-ca
It seems that there's an issue with the scc's for unprivileged and privileged service account. They are granted to the same service account(unprivileged).
That should be checked.

How did you install the Operator? Is this just Openshift or using OLM?
Was it an upgrade from a previous version? If yes, what version?

We installed the operator via the default process:
oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.7.2/openshift.yaml
We tested both options:
Upgrade from 0.5.1 and fresh install from scratch with 0.7.2. Both cases are not successful.
BTW: The Version 0.7.1 is also not running in Openshift ENV because of SCC isues. (The basic Dynatrace Operator Pod can't be created because of scc vioalation... (anyUID..)

Please use the following feature flag to set the oneagent to run as privileged:
kubectl annotate dynakube dynakube-ea feature.dynatrace.om/oneagent-privileged=true

This should allow the correct SCC to be matched so the pods can be created and have the required permissions.

works like a charm after finding the typo ".om".
Shouldn't that be included in the cr.yaml (e.g. classicFullStack.yaml) by default?


feat(logging): always log exceptions by default

Description
Better to always show exceptions by default.
In development, it will be easier for new users, even if using the debug mode will shows the exceptions.
In production, it's better to show the error if something goes wrong.
This would increase the log size.
While I think it's better to always log server errors (5xx errors), there is no real value to log expcetions for (some) client errors, at least for 404 errors.
So we might want to update the default exception_logging_handler at the same time to avoid that.
Happy to discuss more before merging this.

Closes

this would be considering breaking i think?

It doesn't break the API per se, but it's a major behavior change.
That's said, I don't believe people live without logging exceptions even in production? I might be naive. 👀
In all cases, it would be fine to reduce the log size.

Print exceptions on 5xx errors
For 4xx error:

Do not print the exception, an error log would be enough, or
Reduce stacktrace size

About reducting the stacktrace size
For exampe, the exception is raised twice for a 404 error. It could be raised once, it would be less verbose. But even doing so, the stacktrace is not very interesting for the debug):
INFO:     <IP_ADDRESS>:55018 - "GET /notfound HTTP/1.1" 404 Not Found
ERROR - 2024-06-03 17:04:39,776 - litestar - config - Uncaught exception (connection_type=http, path=/notfound):
Traceback (most recent call last):
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/_asgi/routing_trie/traversal.py", line 157, in parse_path_to_route
    node, path_parameters, path = traverse_route_map(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/_asgi/routing_trie/traversal.py", line 54, in traverse_route_map
    raise NotFoundException()
litestar.exceptions.http_exceptions.NotFoundException: 404: Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/middleware/_internal/exceptions/middleware.py", line 158, in __call__
    await self.app(scope, receive, capture_response_started)
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/_asgi/asgi_router.py", line 90, in __call__
    asgi_app, route_handler, scope["path"], scope["path_params"] = self.handle_routing(
                                                                   ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/_asgi/asgi_router.py", line 112, in handle_routing
    return parse_path_to_route(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jderrien/projects/litestar-minimal/.venv/lib/python3.12/site-packages/litestar/_asgi/routing_trie/traversal.py", line 174, in parse_path_to_route
    raise NotFoundException() from e
litestar.exceptions.http_exceptions.NotFoundException: 404: Not Found

this would be considering breaking i think? @litestar-org/maintainers

I would consider this a breaking change. @jderrien you're right that it doesn't break an API, but behavioural changes can be breaking as well. I think we should shelve this for 3.0

@provinzkraut I can understand.
Instead, I'll send another PR for 2.0 to update the doc (to explain the current behavior and advise to set log_exceptions=True). This should be fine?

@provinzkraut I can understand.
Instead, I'll send another PR for 2.0 to update the doc (to explain the current behavior and advise to set log_exceptions=True). This should be fine?

Sounds good!

Do you want me to close this PR? Re-open one on the v3.0 branch?

Feel free to reopen on v3.0. But seeing how we might drop a lot of the logging functionality for that, it's possible the changes won't even make it to the final release.


Revert #237 and deprecate extending a protected closure

Reverts #237 and throws a deprecation notice instead of an exception when attempting to extend a protected closure.
As I said in silexphp/Silex#1538, I believe there are two ways to correctly handle the issue:

We throw an exception, considering that protected closures are parameters values and that extending them should be forbidden.

We allow users to extend anything. In that case, one should expect both these tests to work IMHO:

$pimple['foo'] = 'Hello';

$pimple['bar'] = $pimple->protect(function () {
    return 'Hello';
});

$pimple->extend('foo', function ($helloString) {
    return $helloString.' World';
});

$pimple->extend('bar', function ($helloClosure) {
    return $helloClosure().' World';
});

$this->assertSame('Hello World', $pimple['foo']);
$this->assertSame('Hello World', $pimple['bar']);

Now, considering that Pimple invokes protected closures when the service is initialized, the current behavior forces you to do this instead (it basically ignores that the closure is protected and treats it as a regular, unprotected entry):
$pimple['bar'] = $pimple->protect(function () {
    return 'Hello';
});

$pimple->extend('bar', function ($helloString) {
    return $helloString.' World';
});

That means both fixes would break BC, so I reverted #237 and added a deprecation notice instead.
(PHP implementation is fixed, C is coming up)

C extension fixed too.

Thank you @skalpa.


I want resolver specific caching

Currently the PolywrapClient has it's own cache that it uses for all wrappers and plugins.
Caching should be implemented on a resolver basis, so that URI resolvers (and URI resolver extensions) can use their own caches.
This is blocked by #1102 URI resolution refactor

Closed by: https://github.com/polywrap/toolchain/pull/1169


Support for extracting fragment from response

Add basic async test
Extract function hinclude.show_content

I think fragment should identify a container-node for the content, not the root-node of the content.
So the extraction will be child-nodes of the fragment, not the fragment itself.
My justification is that if the fragment was "main" or "body" then you probably wouldn't want  and definitely wouldn't want  to be included.
FWIW jquery-pjax also takes this approach.
Extending this idea...
You can use the HTML-document parser to handle partial HTML.
For instance
Hi

will result in a DOM tree like

    Hi

One would assume that if there is no @fragment specified then "body" is the default.
Thus there doesn't need to be separate code-paths for partial-HTML documents or for the fragment option. The steps would be:

if there is no @fragment then fragment is "body".
parse a HTML-document from the XHR response
locate the fragmentElement
if it doesn't exist or isn't the  or a descendant then abort
<h-include>.innerHTML = fragmentElement.innerHTML

What do you think?

:+1:


get_timeseries_essentials method needs to return proxy string rather than ontology URL

For instance,

The proxy should be fixed now

Maybe duplicate of https://github.com/LinkedEarth/pylipd/issues/57


Hassbian disk size

I've installed hassbian twice now, once on a 8Gb sd-card, once on a 32Gb sd-card. Issue is the same: it starts complaining about disk size.
hardware is a raspberry pi, I updated to the lastest hassbian and homeassistant versions with the hassbian-scripts.
The output of:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.8G  1.7G     0 100% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M  6.4M  458M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   22M   22M  51% /boot
tmpfs            93M     0   93M   0% /run/user/1000

I try to use the option "expand filesystem" in raspi-config, tried the same with stopping homeassistant from running but doesn't seem to work.
Are there any homeassistant limitations, or shall I just start messing with manually resizing partitions.

pi@hassbian:~ $ sudo raspi-config
Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): Disk /dev/mmcblk0: 29.7 GiB,<PHONE_NUMBER>4 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9050deec
Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192    97459    89268 43.6M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      98304 62333951 62235648 29.7G 83 Linux
Command (m for help): Partition number (1,2, default 2):
Partition 2 has been deleted.
Command (m for help): Partition type
p   primary (1 primary, 0 extended, 3 free)
e   extended (container for logical partitions)
Select (default p): Partition number (2-4, default 2): First sector (2048-62333951, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (98304-62333951, default 62333951):
Created a new partition 2 of type 'Linux' and of size 29.7 GiB.
Partition #2 contains a ext4 signature.
Command (m for help):
Disk /dev/mmcblk0: 29.7 GiB,<PHONE_NUMBER>4 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9050deec
Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192    97459    89268 43.6M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      98304 62333951 62235648 29.7G 83 Linux
Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_TIME = "de_BE.UTF-8",
LC_MONETARY = "de_BE.UTF-8",
LC_ADDRESS = "de_BE.UTF-8",
LC_TELEPHONE = "de_BE.UTF-8",
LC_NAME = "de_BE.UTF-8",
LC_MEASUREMENT = "de_BE.UTF-8",
LC_IDENTIFICATION = "de_BE.UTF-8",
LC_NUMERIC = "de_BE.UTF-8",
LC_PAPER = "de_BE.UTF-8",
LANG = "en_GB.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").

/move to home-assistant/hassbian-scripts


Feature/253a

Uses SearchConfig from #252
Added checkbox to Search Results compnent

Case sensitive

Case Insensitive

/cc @Shaihuludus

We should probably get the open's reviewed and merged as there is dependency chain slowly starting to build.


Connect with Google (Books)

Recommendation 25. In addition to understanding Amazon’s algorithm, COPIM needs to understand more about SEO in Google and Google Books.

Contact for Google Books provided by UCL Press:
Google Playbook Publisher Summit: https://events.withgoogle.com/google-play-books-publisher-summit/. Head of Publishing Partnerships, EMEA: Ani Attamian<EMAIL_ADDRESS>

Specifications: https://play.google.com/books/publish/u/1/a/11587728181440237556#list

From the Google Books team how to deal with zero pricing:

You can also add the tag  with a value of "01".
As per our official documentation:
ONIX code specifying products that are free of charge, products where a price has not yet been announced, or products where no price is applicable. In particular, for free of charge products,  01 should always be used in place of a  composite where the Price amount is zero.