[SPARK-34144][SQL] Exception thrown when trying to write LocalDate and Instant values to a table

What changes were proposed in this pull request?

When writing rows to a table only the old date time API types are handled in org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#makeSetter. If the new API is used (spark.sql.datetime.java8API.enabled=true) casting Instant and LocalDate to Timestamp and Date respectively fails. The proposed change is to handle Instant and LocalDate values and transform them to Timestamp and Date.
Why are the changes needed?

In the current state writing Instant or LocalDate values to a table fails with something like:
Caused by: java.lang.ClassCastException: class java.time.LocalDate cannot be cast to class java.sql.Date (java.time.LocalDate is in module java.base of loader 'bootstrap'; java.sql.Date is in module java.sql of loader 'platform') at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeSetter$11(JdbcUtils.scala:573) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$makeSetter$11$adapted(JdbcUtils.scala:572) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:678) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:858) at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:856) at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:994) at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:994) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2139) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449) ... 3 more
Does this PR introduce any user-facing change?

No
How was this patch tested?

Added tests

@MaxGekk , @maropu Can you please have a look at this as well or point me to the right person for the review? Thanks!

@MaxGekk , @maropu Can you please have a look at this as well or point me to the right person for the review? Thanks!

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Thanks for your help, @MaxGekk! Please let me know if there's anything else I can do to get this approved.

Thanks for your help, @MaxGekk! Please let me know if there's anything else I can do to get this approved.

@MaxGekk @maropu Please let me know if there's anything else I need to do to get this moving.

@MaxGekk @maropu Please let me know if there's anything else I need to do to get this moving.

ok to test

ok to test

Thank you for your help, @HyukjinKwon

Thank you for your help, @HyukjinKwon

Test build #134550 has started for PR 31264 at commit 307d819.

Test build #134550 has started for PR 31264 at commit 307d819.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39136/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39136/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39136/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39136/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39136/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39136/

Test build #134550 has finished for PR 31264 at commit 307d819.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134550 has finished for PR 31264 at commit 307d819.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134550/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134550/

added PR for 3.0 fix as well: https://github.com/apache/spark/pull/31382

added PR for 3.0 fix as well: https://github.com/apache/spark/pull/31382

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134643/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134643/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39231/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39231/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39231/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39231/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39231/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39231/

Merged to master and branch-3.1.

Merged to master and branch-3.1.

thanks for your help!

thanks for your help!


What's the difference between this and idris-vscode

I'm just starting with Idris and it's unclear to me if I should use this extension or https://github.com/meraymond2/idris-vscode. Perhaps the README could highlight the differences?

https://github.com/meraymond2/idris-vscode/issues/75

https://github.com/meraymond2/idris-vscode - is using idris1/2 (?) ide mode (wrapper is https://github.com/meraymond2/idris-ide-client )
https://github.com/bamboo/idris2-lsp-vscode and https://github.com/ShinKage/idris2-nvim use https://github.com/idris-community/idris2-lsp
many commands are missing in https://github.com/bamboo/idris2-lsp-vscode but present in https://github.com/meraymond2/idris-vscode, e.g. show list of all metavariables in a current file


Better README.  Shows transactions.  Bug fix to make it run :D

Waiting for islandora packagist account to get settled before putting composer install instructions.  Also waiting for licensing issue to get sorted out before putting that blurb in too.  Better than what we had for the time being, at least.

:+1:


ServiceMonitor target port fixed

ServiceMonitor target port fixed (metrics -> internal)

@srfaytkn hello, could you please bump version to next patch - i.e. 6.0.3? https://github.com/centrifugal/helm-charts/blob/master/charts/centrifugo/Chart.yaml#L4

@srfaytkn hello, could you please bump version to next patch - i.e. 6.0.3? https://github.com/centrifugal/helm-charts/blob/master/charts/centrifugo/Chart.yaml#L4

I fixed it @FZambia

I fixed it @FZambia

Thanks a lot!

Thanks a lot!


/xinfo and /info do not send responses

sending a OSC message without arguments a information object is expected as response and not received. If Songbeamer is following the XAir OSC protocoll the following messages are expected
/info
<string server_version>
<string server_name>
<string console_model>
<string console_version>

/xinfo
<string network address>
<string network name>
<string console_model>
<string console_version>

/xinfo response
debug: Received OSC message, Remote info is: {"address":"<IP_ADDRESS>","family":"IPv4","port":10023,"size":64}
/info response
debug: OSC Content is: {"address":"/info","args":[{"type":"s","value":"V2.08"},{"type":"s","value":"osc-server"},{"type":"s","value":"SongBeamer"},{"type":"s","value":"6.00"}]}
-> not a bug - added logging to warn in case of not implemented actions

style of response changed to include types
debug: Sent OSC to <IP_ADDRESS>:10023 with /xinfo and []
debug: Received OSC message from: {"address":"<IP_ADDRESS>","family":"IPv4","port":10023,"size":64}
debug: OSC Content is: {"address":"/xinfo","args":[{"type":"s","value":"<IP_ADDRESS>"},{"type":"s","value":"osc-server"},{"type":"s","value":"SongBeamer"},{"type":"s","value":"6.04"}]}
warn: received a message with an unknown address - not implemented

in fact list of

{"type":"s","value":"<IP_ADDRESS>"}
{"type":"s","value":"osc-server"}
{"type":"s","value":"SongBeamer"}
{"type":"s","value":"6.04"}]

debug: Sent OSC to <IP_ADDRESS>:10023 with /info and []
debug: Received OSC message from: {"address":"<IP_ADDRESS>","family":"IPv4","port":10023,"size":56}
debug: OSC Content is: {"address":"/info","args":[{"type":"s","value":"V2.08"},{"type":"s","value":"osc-server"},{"type":"s","value":"SongBeamer"},{"type":"s","value":"6.04"}]}
warn: received a message with an unknown address - not implemented

list like

{"type":"s","value":"V2.08"}
{"type":"s","value":"osc-server"}
{"type":"s","value":"SongBeamer"}
{"type":"s","value":"6.04"}]

No Feedback actions required because this is information is configured it the connection properties and only used upon init - there is no use case of e.g. a user variable referencing these values


add support namespaces SVG attrs

This PR:

converts SVG namespaced attrs to React friendly ones
removes stray <style /> attrs

My motivation for writing this was because I was getting errors with the SVGs I was given, which all have xlink:href type attrs in them.

Would love to see this merged soon, the namespace thing is really causing trouble for me.

Hey there, this has been open for a while, is there anything blocking?

Hey @JohnPaulHarold,
thanks a lot for your contribution! Could you please provide us an example SVG file where I can check your changes because the replace for <style> tag is not working for me... Thanks 🙏

@incik This should be done


What version of react-native is required?

What is the earliest version of react-native compatible with this component?
I'm wondering because I'm getting cannot read property _style_ of undefined on ViewPropTypes.style in react-native 0.39.2.

I believe 1.8.0 should be good for you

ViewPropTypes is support in<EMAIL_ADDRESS>

Sorry, I've unpublished 1.8.1 (which should've been 2.0.0), so version < 2.0 is compatibly with old RN and >= 2.0 with 0.44+

@Kureev I'll send in a PR that allows both versions to work

@gtzogana I've sent the fix and it's been merged. You can use the latest version of react-native-navbar (2.0.1) with your current version of react-native


Go SQLC Generate is outputting a DBTX interface that is not implemented by *sql.DB golang internal library

Version
1.27.0
What happened?
Not sure if I'm doing something as I am new to sqlc, but it seems to be outputting an interface that is not compliant with the sql.DB type in go.
Here is my go version: go version go1.23.4 linux/amd64
I am using sqlc v1.27.0.
Here is my sqlc.yaml file:
version: "2"
plugins:
- name: golang
  wasm:
    url: https://downloads.sqlc.dev/plugin/sqlc-gen-go_1.4.0.wasm
    sha256: e8206081686f95b461daf91a307e108a761526c6768d6f3eca9781b0726b7ec8
sql:
- schema: "./db/migration"
  queries: "./db/query"
  engine: "postgresql"
  codegen:
  - plugin: golang
    out: "./db/sqlc"
    options:
      package: "db"
      sql_package: pgx/v5
      emit_json_tags: true
      emit_prepared_queries: false
      emit_interface: false
      emit_exact_table_names: false
      emit_pointers_for_null_types: true
      overrides:
        - db_type: "timestamptz"
          go_type:
            import: "time"
            type: "Time"

And here is the DBTX interface that is outputted:
type DBTX interface {
	Exec(context.Context, string, ...interface{}) (pgconn.CommandTag, error)
	Query(context.Context, string, ...interface{}) (pgx.Rows, error)
	QueryRow(context.Context, string, ...interface{}) pgx.Row
}

I believe these methods should all have the suffix: *Context for sql.DB to implement the interface. I can't find anything wrong with the yaml file. Perhaps I have some incompatibility between the versions of go and sqlc I am using, but I don't believe so.
Relevant log output
No response
Database schema
No response
SQL queries
No response
Configuration
No response
Playground URL
No response
What operating system are you using?
Linux
What database engines are you using?
PostgreSQL
What type of code are you generating?
Go

You've set the sql_package to be pgx/v5, which generates code to be used directly with the pgx/v5 package, not database/sql. You can see how to use the generated code in the Getting Started guide.
If you want to use the database/sql package, you need to set sql_package to database/sql. Here's an example of doing that.


Fix crates badge

Rendered

I also noticed our license badge wasn't correct, so I fixed that.

bors r+


Ensure there's at most one Person per domain username

Some of the user accounts in the production environment are based on position instead of the person.
As a result,following a HOTO, a new system user is being authenticated and associated with an old account when a new user (albeight probably in the same position) should have been created.
ANET should be modified to allow associating several Persons to the same domain username. At any given time, at most one Person should be active per domain username, the remaining ones being there for archiving reasons.
When there are no active Persons associated with a username, ANET should prompt for new account creation upon user access. The resulting Person should become the new active Person for that username.
Superusers and administrators should be able to:

deactivate active Persons
activate Persons when there are no other active Persons
switch the active Person when a Person needs to be activated and there is another active Person with the same username

The current authentication filter strategy is (both in AnetDevAuthenticator and AnetAuthenticationFilter):

get the list of Persons with the domainUsername used to authenticate
if the list is empty, create a new Person for that domainUsername
otherwise get the first Person from the list
if the Person is inactive, activate that Person
use this Person as the Principal for this session

This strategy clearly needs to change. We need to agree what it should be.
Simply changing the first step to:

get the list of active Persons with the domainUsername used to authenticate

means that step 4 is no longer possible. It could result in creating a new Person which in real life should be a reactivation of an existing inactive Person, leading to an admin having to merge these two Persons later on.
It rather depends on how common this case is. If the case is very rare compared to a new Person joining, the overhead of the merge action is negligible.
Perhaps @dspCarlson can add some insights from the field?

Superusers and administrators should be able to:

deactivate active Persons

This is already possible, and requires no change.

activate Persons when there are no other active Persons
switch the active Person when a Person needs to be activated and there is another active Person with the same username

Activating a Person should check whether there's already another Person with the same domainUsername that is active. Could possibly ask the admin:
"WARNING: activating this Person will deactivate Person JohnDoe, do you want to continue?"
and if yes, deactivate the other Person and activate the current Person.

When an outgoing person is inactivated (people.status = 1) by a super user, the domain user name for the outgoing person must be set to NULL.  This will allow the incoming person to logon and create a new profile/account.  However, a warning message must be displayed so the super user knows the consequences of inactivating the outgoing person.  Message: "Inactivating JohnDoe means the next person to logon with  will have to create a new profile.  Do you want the next person to login with this user name to create a new profile?"  Does this help?

Message: "Inactivating JohnDoe means the next person to logon with the  user name will have to create a new profile. Do you want the next person to login with this user name to create a new profile?"

Along with deactivating the person, when the end of tour date is expired, the super user must be forced to decide to deactivate the person or change the end of tour date to a future date.  There are too many persons with an expired end of tour date but their status is still active.

@dspCarlson Thanks for the insights. Should we define a (one-time or scheduled) cleanup action as well, e.g. to deactivate all persons whose end of tour date is in the past?

Super users must be forced to acknowledge (maybe with a display message when they login) they have people who are active and yet their end of tour date is expired.  Can we force super users to inactivate people before they can use other functions?  As for a one-time clean up, I can do that from here.  But I want to give super users a chance to clean up their data first.  After a few weeks of letting the super user users clean up their data, I will run an update query.  There will be users/stragglers who are no longer assigned to a position and therefore won't be under a super users purview.

@dspCarlson In general (from a usability standpoint) it is not wise to force users to do something before they can continue their work. They may not be able to decide on the spot what to do about it, and the reason they logged in in the first place is because they want to accomplish something else. 😉
So adding a warning banner seems like a more user-friendly approach.
As for the cleanup action: so yes, we have indeed defined one (just now, ad hoc), which you will be able to perform whenever you feel it's time. 👍
We could even decide on a deadline for the cleanup action, and put that date in the warning banner.

I agree 100% with your last comments.  If we could have a warning banner that would be appropriate.  I believe super users will do their job if the program is able to "coach"/show them issues with their data.
There is another item that refers to having a warning banner for planned outages (like last night).  With a warning banner, the business owner wouldn't have to notify super users and users ahead of time.

FYI: the (generic) banner issue is tracked here: #27

I will refine this story.

From querying the database, we see that the problematic duplicate users account for only about 1-2% of the active users. We should fix those by hand, i.e. set the status of the spurious duplicates to 1 (inactive).
Then we should manually set the domainUsername to NULL for all inactive accounts.
This keeps the proposed implementation simple:

when a super-user inactivates an account, a warning is shown:

Setting this person to inactive means the next person to logon with the user name {person.domainUsername} will have to create a new profile. Do you want the next person to login with this user name to create a new profile?

when an account is inactivated, also the domainUsername field is cleared
when a user authenticates, there should be an active account with that domainUsername, which will then be associated with that user for the duration of the session
if a user authenticates and there is no active account with that domainUsername, a new account is created and the user should fill out the profile

For new profiles, a super-user needs to assign a position. This also gives the super-user the ability to merge the new profile with an old existing profile in the rare case that one should be re-used/re-activated.

One more thought on this issue.  When a user is made inactive, the adviser's position should have an unassigned status.  The position should not have anybody assigned to it.  This affects positions.currentPersonId and peoplePositions.  The super user should also be notified of this via the displayed message.

@dspCarlson: Re: unassigning position:
Code was already in place to clear the position if a person was set to inactive. Nothing needed to change there.

@dspCarlson: Re: people coming back after a while:
When someone authenticates sucessfully, the only thing we get is the domainUsername. We can look up if there's a person linked to it, but if not, we create a new profile (there's no other choice). But as stated in https://github.com/NCI-Agency/anet/issues/65#issuecomment-317777315 :

For new profiles, a super-user needs to assign a position. This also gives the super-user the ability to merge the new profile with an old existing profile in the rare case that one should be re-used/re-activated.

I suggest that when a person fills their details for a new acount, the system finds inactive users with similar data and suggests to reactive these accounts. The same kind of feature could apply to creating new people and places

@vassil That should be a new story. "Similarity" should be defined etc.

Reconciled 30 domainUserNames that were duplicated.  There are no more records with duplicate domainUserNames.

Excellent!

Following test case fails:

Log in as admin,
find one active user with a domain name.
edit, switch status to inactive
-> ANET displays warning that user will not be able to logon anymore using its domain name
save user
inspect domainUsername field in MS SQL
-> domainUsername is not NULL

@gjvoosten - looks like PersonDao.update does not handle domainUsername, hence the change never makes it to the db

Pfrt...


⚠ PrivateHD has degraded performance

In 3652e5a, PrivateHD ($PHD) experienced degraded performance:

HTTP code: 200
Response time: 2004 ms

Resolved: PrivateHD performance has improved in df7cbf6.


Menu by hover on row

mantine-react-table version
1
react & react-dom versions
17
Describe the bug and the steps to reproduce it
Hi, I want to open a menu from the side of a row when I hover over it.  I would appreciate it if you could help me with this.
Minimal, Reproducible Example - (Optional, but Recommended)
.
Screenshots or Videos (Optional)
No response
Do you intend to try to help solve this bug with your own PR?
None
Terms

[X] I understand that if my bug cannot be reliably reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.

Closing as this is a question, please try the Discord server.


Invalid equals assertion

On line 220 of StandardOPCUAService.java an invalid equals assertion is performed. In java == is not equal to .equals when strings are involved. This needs to be refactored into .equals

Refactored assertion to use .equals to compare string value.  See pull request in #28


Fix elasticsearch::template HTTPS usage

Pull request acceptance prerequisites:

[x] Signed the CLA (if not already signed)
[x] Rebased/up-to-date with base branch
[ ] Tests pass
[x] Updated CHANGELOG.md with patch notes (if necessary)
[x] Updated CONTRIBUTORS (if attribution is requested)

Per #688, the ruby provider actually never sets the SSL flag for the Net::HTTP object it uses (whoops!) This change sets the flag appropriately and also adds an explicit test to ensure we're validating it does so.

Thank you @tylerjl, just wanted to let you know that the template was successfully created once I applied this PR.


Fixing VERSION in Makefile

Fix #1706
How to test
$ ampmake clean build
$ docker image ls
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
appcelerator/ampagent         0.16.0-dev          c1af2c41e934        4 minutes ago       26.4MB
appcelerator/amp-aws          0.16.0-dev          a8dcef6bb775        5 minutes ago       16.5MB
appcelerator/amp-local        0.16.0-dev          f6e68bc0c352        5 minutes ago       14.7MB
appcelerator/amp-prometheus   0.16.0-dev          4df0850283ff        5 minutes ago       125MB
appcelerator/agent            0.16.0-dev<PHONE_NUMBER>45        5 minutes ago       31.8MB
appcelerator/ampbeat          0.16.0-dev          2dfad49d85af        5 minutes ago       23.3MB
appcelerator/gateway          0.16.0-dev          e4d90710a442        6 minutes ago       28.8MB
appcelerator/amplifier        0.16.0-dev          dfd25e5b4033        6 minutes ago       38.7MB
amptools                      latest              bbba2e09044e        6 minutes ago       992MB
appcelerator/amptools         1.9                 0d8f164f60ea        4 hours ago         992MB
alpine                        latest              76da55c8019d        4 weeks ago         3.97MB
appcelerator/prometheus       1.7.1               6f8a89872aa1        3 months ago        104MB

but then, we can't do the make VERSION=latest build-ampagent anymore.
the conditional variable assignment is supposed to work, unless the new version of the amptools image doesn't use gnu make.

I don't think anything related to make has been changed in the latest version of amptools.

for a reason I ignore, ampmake sets version=0.0.0 and uses it as an env var when it runs docker to set VERSION. I think it's safe to remove this -e line.

Good catch! I'll update accordingly


Python: Make "Modification of parameter with default" flow-sensitive.

Extends the analysis in py/modification-of-default-value to use taint tracking. This enables us to detect modifications that happen inside other function calls.
@felicity-semmle for the change note. (And also possibly the alert message?)
Open question:
I have encountered the following idiom in some places that are flagged by the current query
def foo(x=[]):
    if not x:
        x = [1]
    # ... more statements, none of which modify x

Should we treat if not x as a sanitiser for x? As far as I can tell, there are no hidden surprises in the above code.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the change. Text LGTM.


test file

Signed-off-by: Daniel Pono Takamori<EMAIL_ADDRESS>

Abandoning for now. Will come up with a new test plan in the future


OIT step 1: Refactor renderer

This refactors the wgpu render code to prepare for implementing OIT, as planned in #196.
Todo:

[x] Refactor so all blending stuff is defined in a single place.
[x] More refactoring to support 1 or 2 render passes.
[x] Implement render mode that we had before this.
[x] Implement simple 2-pass render mode.
[x] Add renderer.blend_mode.
[x] Decide how to implement the two fragment entry points with the shader (templates). Depends on https://github.com/gfx-rs/naga/issues/1509 -> see #200
[x] Implement for all materials.
[x] Fix example to use PySide6.
[x] Remove the with open("c:/dev/py/tmp.wgsl") thing.
[x] Self-review + check for todo's.
[x] Relief limitation that varying assignment must be a oneliner
[x] Cleanup

Well, you could at least call it FragmentBlender :)

Still refactoring and cleaning up, but here are some results for three initial blend modes.
The "solid" blend mode pretends that there is no such thing as transparency. Only included as a stub. It has one benefit: it's order independent :P

The "minimal" blend mode uses one pass, processing opaque and transparent fragments at the same time, while depth-testing. This is basically what we had. You can see how the red plane (the first-drawn) hides the other planes behind it because of the depth test. Also the blue plane (drawn last) disappears behind the green plane. Oh, and the sphere is partially hidden too.

The "simple" blend mode uses two passes. One for opaque and one for transparent fragments. The looks result is much better. However, it's not order-independent, as can be seen in the second image.

)

Just wondering if we are still sorting objects along their depth in camera space prior to rendering? That way at least the result will be deterministic, though still order-dependent.

I think we are.

I'm now happy with the approach to deal with the shaders that run (slightly changed) in multiple render passes. See #200 for details. Other than that, we just create two wgsl samples, and two wgpu shader modules: one for each render pass. I initially wanted to include all entrypoints into the same shader, but that quickly becomes complicated to implement and explain.

This is ready from my end.
Some updates:

I implemented the new approach for all material-renderers! Ran into a few hurdles though:
Since the blend mode affects the pipeline of the wobjects, the pipeline data could no longer be stored on the wobjects themselves (it would go wrong if using one object in multiple renders with different blend modes). Therefore the pipeline-data is now associated via weak-ref dicts. For details see the extensive comment in _renderer.py.
The blender defines the FragmentOutput, but it does not know beforehand whether the eventual shader wants to also write depth. I added another (much simpler) wgsl post processing pass to resolve this.
The varyings resolve post-processing now deals with multiline assignments.

Overall, this refactoring is mainly a preparation for more advanced blend modes, adding support for multi-stage rendering. Because we now have a 2 pass blend mode, even though its still simple/wrong, it already yields much better results for scenes that contain transparent objects. It also deals much better with e.g. the semitransparent edges of our lines (for aa); our lines look much better now!

Since the blend mode affects the pipeline of the wobjects, the pipeline data could no longer be stored on the wobjects themselves (it would go wrong if using one object in multiple renders with different blend modes). Therefore the pipeline-data is now associated via weak-ref dicts. For details see the extensive comment in _renderer.py.

Sounds like a big improvement to me!

The blender defines the FragmentOutput, but it does not know beforehand whether the eventual shader wants to also write depth. I added another (much simpler) wgsl post processing pass to resolve this.

How should I feel about this? :)

The varyings resolve post-processing now deals with multiline assignments.

Awesome!

Overall, this refactoring is mainly a preparation for more advanced blend modes, adding support for multi-stage rendering. Because we now have a 2 pass blend mode, even though its still simple/wrong, it already yields much better results for scenes that contain transparent objects. It also deals much better with e.g. the semitransparent edges of our lines (for aa); our lines look much better now!

Let's merge and continue in another PR!

How should I feel about this? :)

It means we can write:
var out = finalize_fragment();
out.depth = ndc_pos.z / ndc_pos.w;

and it Just Works.
We could make it more explicit, e.g.:
var out = finalize_fragment_at_depth(ndc_pos.z / ndc_pos.w;);

BTW, I think I may replace add_fragment and finalize_fragment with a single function. Let's revisit then. I'll add a note in the TODO of #196

As far as I am concerned we can merge :)

Wasn't sure whether @berendkleinhaneveld wanted to have a look.

Please go ahead and merge: I'll catch up later!


why is the -p flag required?

On https://github.com/stich86/lpac-libmbim-wrapper/blob/4641f27e43478c79d37ab18e485bf75a9b5f7ffd/wrapper.py#L23 we have a -p flag that is passed to mbimcli. What is this for? It's not working for me with that. Getting the following:
$ sudo mbimcli -p -d /dev/cdc-wdm0 --ms-set-uicc-open-channel="application-id=a0000005591010ffffffff8900000100"
error: operation failed: MBIM protocol error: NotOpened
$

However, if I remove -p, I get:
$ sudo mbimcli -d /dev/cdc-wdm0 --ms-set-uicc-open-channel="application-id=a0000005591010ffffffff8900000100"
Succesfully retrieved open channel info:
	  status: 144
	 channel: 3
	response: 6F:1F:84:10:A0:00:00:05:59:10:10:FF:FF:FF:FF:89:00:00:01:00:A5:04:9F:65:01:FF:E0:05:82:03:02:02:02
$

-p is for Proxy call, otherwise each time you got error of not open channel

okay, but I'm getting an error with it that says NotOpened

okay, but I'm getting an error with it that says NotOpened

Which modem and which libmbim version? I've tested it under Ubuntu 23.10 and OpenWrt SNAPSHOT without any issue using proxy mode

$ mbimcli -V
mbimcli 1.28.0
Copyright (C) 2013-2021 Aleksander Morgado
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$

$ cat /etc/issue
Ubuntu 22.04.4 LTS \n \l

  Hardware |            manufacturer: Quectel
           |                   model: EM120R-GL

$ uname -a
Linux rd 5.15.0-1053-raspi #56-Ubuntu SMP PREEMPT Mon Apr 15 18:50:10 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

I can use lpac in AT mode fine by the way. That is undesirable though because ModemManager locks the tty up and I have to stop the ModemManager service to use lpac every time, which also disconnects my internet connection. I'm hoping MBIM mode won't have that problem.

Where is the application-id coming from? Is that something that lpac makes up?

Yes all application ids are generated by LPAC, wrapper just get it from stdin

Is ModemManager supposed to be started or stopped when using this wrapper?

If it's using MBIM port, I think it's better to close it

Okay, I'm getting a little farther. Here's the new outcome:

shut down
remove power
restore power and boot
stop ModemManager
run wrapper.py chip info
it yields the chip info output expected, but then it won't close the channel and I get

INFO: Close channel 01
close channel_id: 1
Error executing command: error: operation failed: Unknown status 0x87430003

after that, nothing works again without starting back with (1).

Okay, I'm getting a little farther. Here's the new outcome:

shut down

remove power

restore power and boot

stop ModemManager

run wrapper.py chip info

it yields the chip info output expected, but then it won't close the channel and I get

INFO: Close channel 01

close channel_id: 1

Error executing command: error: operation failed: Unknown status 0x87430003

after that, nothing works again without starting back with (1).

What's happen if you remove -p form mbimcli?

https://aleksander.es/data/20210421 - Introduction to ModemManager.pdf page 22 talks about what the mbim-proxy proxy does. Apparently it allows you to keep ModemManager running if you use it.

https://github.com/linux-mobile-broadband/libmbim/blob/214bacbc1ce72121dd89ba09e0598a6f05078f8e/NEWS#L306 might be related.

Also related: https://github.com/linux-mobile-broadband/libmbim/blob/214bacbc1ce72121dd89ba09e0598a6f05078f8e/NEWS#L337 .

Okay, I'm getting a little farther. Here's the new outcome:

shut down
remove power
restore power and boot
stop ModemManager
run wrapper.py chip info
it yields the chip info output expected, but then it won't close the channel and I get

INFO: Close channel 01

close channel_id: 1

Error executing command: error: operation failed: Unknown status 0x87430003

after that, nothing works again without starting back with (1).

What's happen if you remove -p form mbimcli?

Seems like removing -p doesn't work if mbim-proxy is running, just times out. If you manually kill mbim-proxy, seems you can run without -p because the proxy isn't started. If you kill mbim-proxy and run with -p, it automatically restarts mbim-proxy. I am testing all this with sudo mbimcli -d /dev/cdc-wdm1 --ms-query-uicc-application-list, which doesn't seem to be having problems.

See also, https://modemmanager.org/docs/modemmanager/debugging/#qmi-proxy-or-mbim-proxy-debug-logs .

Now I'm getting:
$ sudo mbimcli -p -d /dev/cdc-wdm1 --ms-set-uicc-open-channel="application-id=a0000005591010ffffffff8900000101"
error: operation failed: Unknown status 0x87430001
$

This seems to be the same with and without mbim-proxy. Not sure why the error is different now.

Okay, I'm getting a little farther. Here's the new outcome:
1. shut down

2. remove power

3. restore power and boot

4. stop ModemManager

5. run `wrapper.py chip info`

6. it yields the chip info output expected, but then it won't close the channel and I get

INFO: Close channel 01
close channel_id: 1
Error executing command: error: operation failed: Unknown status 0x87430003

after that, nothing works again without starting back with (1).

Correction, running sudo mmcli -m 2 -r will allow it to reset without requiring the power cycle. If I do that, I can run
sudo mbimcli -p -d /dev/cdc-wdm1 --ms-set-uicc-open-channel="application-id=a0000005591010ffffffff8900000100"

successfully (2) times before it errors out and requires sudo mmcli -m 3 -r to be run to reset it.

Wondering what eSIM and modem you are using?
How does this command select which SIM it is working on with a dual SIM modem?

Okay, I figured out the bug, your script wasn't closing the channel. Here is the fix.

@@ -76,8 +81,8 @@ def handle_type_apdu(func: str, param: str):

     if func == "logic_channel_close":
         try:
-            print(f"INFO: Close channel {param}")
-            close_channel(int(param))
+            print(f"INFO: Close channel {CHANNEL_ID}")
+            close_channel(int(CHANNEL_ID))
             return {"ecode": 0}
         except subprocess.CalledProcessError as e:
             print(e)

Possibly you are using another SIM or modem that allow more active channels?

Okay, so I'm able to use
sudo wrapper.py profile enable<PHONE_NUMBER>3580233317

followed by
sudo mmcli -m 2 -r

to change the profile on the eSIM and then re-load it.
Not sure if there is a better way to re-load the eSIM without resetting the entire modem? However, the point is, I don't need to stop ModemManager to do this. The mbim-proxy allows both to communicate with the modem at the same time.

@z3ntu, I think lpac-libqmi-wrapper might have the same bug here https://github.com/z3ntu/lpac-libqmi-wrapper/blob/0ff0bb62273bba2a2d530f5eb7863dfda4105528/lpac-libqmi-wrapper.py#L96 .

Wondering what eSIM and modem you are using?
How does this command select which SIM it is working on with a dual SIM modem?

I've tested with Foxconn T99W175 using embedded eSIM (on slot 2) and physical eSIM (Levovo Thales) on slot 1
And in a modem without eSIM (using physical one) on Quectel RM502GQ-GL

fixed by https://github.com/stich86/lpac-libmbim-wrapper/pull/6

Could we reopen this?
The current code still requires -p, and the reason was just found:
https://github.com/stich86/lpac-libmbim-wrapper/issues/8#issuecomment-2166837603
So the real fix is yet pending.

Not sure that these are related. I also don't understand the comment you link to.
I think my original issue with the -p flag might have been something jammed up with the mbim-proxy after lots of testing. After I understood what the proxy was for and restarted everything it exposed a new issue, which was then fixed by https://github.com/stich86/lpac-libmbim-wrapper/pull/6 .

The issue is, -p flag is currently hardcoded and won't work if you don't use mbim-proxy and/or MM.
To make it work a such way, the linked fix is needed, then the flag could be made optional.

@AndySchroder does this make sense?
If you disagree to reopen this issue, I'll probably open a new one.


odo requires the 'master' branch

/kind bug
What versions of software are you using?
Operating System: Fedora 32
Output of odo version: odo v2.0.1 (a9fff7da4)
How did you run odo exactly?
odo create --devfile https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml --starter -v 5
Actual behavior
I1125 11:48:09.678897  303618 util.go:730] HTTPGetRequest: https://raw.githubusercontent.com/openshift/odo/master/build/VERSION
I1125 11:48:09.678974  303618 util.go:751] Response will be cached in /tmp/odohttpcache for 1h0m0s
I1125 11:48:09.679037  303618 util.go:764] Cached response used.
I1125 11:48:09.821245  303618 util.go:425] path .odo/config.yaml doesn't exist, skipping it
I1125 11:48:09.821305  303618 util.go:425] path .odo/env/env.yaml doesn't exist, skipping it
I1125 11:48:09.821343  303618 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:48:09.821462  303618 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:48:09.821496  303618 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
I1125 11:48:09.838847  303618 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:48:09.838934  303618 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
I1125 11:48:09.851258  303618 util.go:425] path devfile.yaml doesn't exist, skipping it
Validation
I1125 11:48:09.851294  303618 util.go:425] path devfile.yaml doesn't exist, skipping it
 •  Creating a devfile component from devfile path: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml  ...
 ✓  Creating a devfile component from devfile path: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml [45712ns]
 •  Validating devfile component  ...
I1125 11:48:09.851378  303618 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
 ✓  Validating devfile component [167059ns]
I1125 11:48:09.851549  303618 util.go:730] HTTPGetRequest: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml
I1125 11:48:09.978051  303618 content.go:34] converted devfile YAML to JSON
I1125 11:48:09.978107  303618 apiVersion.go:50] devfile apiVersion: '2.0.0'
I1125 11:48:09.978115  303618 helper.go:36] devfile apiVersion '2.0.0' is supported in odo
I1125 11:48:09.983099  303618 schema.go:47] validated devfile schema
I1125 11:48:09.983466  303618 validate.go:43] Successfully validated devfile sections
I1125 11:48:09.983476  303618 validate.go:43] Successfully validated devfile sections
I1125 11:48:09.983483  303618 validate.go:43] Successfully validated devfile sections
I1125 11:48:09.983486  303618 validate.go:43] Successfully validated devfile sections
I1125 11:48:09.983529  303618 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml

Starter Project
 •  Downloading starter project thoth-odo-example from https://github.com/goern/thoth-odo-example  ...
 ✗  Downloading starter project thoth-odo-example from https://github.com/goern/thoth-odo-example [382ms]
 ✗  couldn't find remote ref "refs/heads/master"

Expected behavior
I1125 11:49:19.892935  303743 util.go:730] HTTPGetRequest: https://raw.githubusercontent.com/openshift/odo/master/build/VERSION
I1125 11:49:19.893515  303743 util.go:751] Response will be cached in /tmp/odohttpcache for 1h0m0s
I1125 11:49:19.894018  303743 util.go:764] Cached response used.
I1125 11:49:20.037106  303743 util.go:425] path .odo/config.yaml doesn't exist, skipping it
I1125 11:49:20.037127  303743 util.go:425] path .odo/env/env.yaml doesn't exist, skipping it
I1125 11:49:20.037139  303743 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:49:20.037179  303743 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:49:20.037190  303743 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
I1125 11:49:20.043378  303743 util.go:425] path devfile.yaml doesn't exist, skipping it
I1125 11:49:20.043403  303743 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
I1125 11:49:20.050274  303743 util.go:425] path devfile.yaml doesn't exist, skipping it
Validation
I1125 11:49:20.050300  303743 util.go:425] path devfile.yaml doesn't exist, skipping it
 •  Creating a devfile component from devfile path: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml  ...
 ✓  Creating a devfile component from devfile path: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml [37959ns]
 •  Validating devfile component  ...
I1125 11:49:20.050369  303743 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml
 ✓  Validating devfile component [118740ns]
I1125 11:49:20.050492  303743 util.go:730] HTTPGetRequest: https://raw.githubusercontent.com/goern/thoth-odo-example/main/devfile.yaml
I1125 11:49:20.246928  303743 content.go:34] converted devfile YAML to JSON
I1125 11:49:20.247032  303743 apiVersion.go:50] devfile apiVersion: '2.0.0'
I1125 11:49:20.247046  303743 helper.go:36] devfile apiVersion '2.0.0' is supported in odo
I1125 11:49:20.259034  303743 schema.go:47] validated devfile schema
I1125 11:49:20.259955  303743 validate.go:43] Successfully validated devfile sections
I1125 11:49:20.259979  303743 validate.go:43] Successfully validated devfile sections
I1125 11:49:20.259991  303743 validate.go:43] Successfully validated devfile sections
I1125 11:49:20.259998  303743 validate.go:43] Successfully validated devfile sections
I1125 11:49:20.260084  303743 preference.go:209] The path for preference file is /home/goern/.odo/preference.yaml

Starter Project
 •  Downloading starter project thoth-odo-example from https://github.com/goern/thoth-odo-example  ...
 ✓  Downloading starter project thoth-odo-example from https://github.com/goern/thoth-odo-example [861ms]

Please use `odo push` command to create the component with source deployed
I1125 11:49:21.122317  303743 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)

Any logs, error output, etc?

odo takes default branch as master, currently specifying branch name is not supported by odo.
however it should support main as default branch too as many projects are now using main as default branch

actually odo should be able to checkout out any branch, isn't the devfile format supporting that?

yes, it should support checkout from any branch.


variables & refactoring

Currently 1-Start_WiFi.sh probably needs to export the variables:

Network_interface
WiFi_name

Could do something like the following:
function choose_network_interface {
...
         read Network_interface; echo "${Network_interface}"
}

Network_interface=$(choose_network_interface)

However that would probably require redirection of everything except wanted output to stderr, which isn't something I feel comfortable with either

Ya, I tried this and it was a complete mess. Will maybe upload how it looked like.

Instead, what I ended up doing is putting all variables in one file Settings.txt which is source by all scripts. That way, if user has already configured something we can skip those steps.
For example, if you've got a wpa_supplicant.conf file, no need to (re)generate one.


[Bug]: Metadata filters not working with VectorIndexRetriever for Milvus

Bug Description
Here is my code that creates and adds filters
            # configure filters
            filters = MetadataFilters(filters=[MetadataFilter(key="file_name", value=filename),
                                               MetadataFilter(key="topic_id", value=topic.topic)],
                                      condition=FilterCondition.AND)
            # configure retriever
            retriever = VectorIndexRetriever(
                service_context=self.service_context,
                index=index,
                similarity_top_k=6,
                filters=filters
            )

            nodes = retriever.retrieve(query)

When run, it throws the following error:
MilvusException: (code=1100, message=failed to create query plan: cannot parse expression: file_name == "Lohmod 2024-03-1.pdf"topic_id in [\'XYZ\'] and and topic_id == "XYZ", error: invalid expression: file_name == "Lohmod 2024-03-1.pdf"topic_id in [\'XYZ\'] and and topic_id == "XYZ": invalid parameter)>

Version
0.10.37
Steps to Reproduce

Create a vector index using Milvus as vector store
Try to filter nodes using VectorIndexRetriever
Use multiple metadata filters "and"-ed. (Single filter works fine)

Relevant Logs/Tracbacks
No response

Hi @mballav, it seems that you are not using the latest version of llama-index-vector-stores-milvus, try installing the latest one and see how it goes:
pip install llama-index-vector-stores-milvus==0.1.14

Additionally, what's the value of topic.topic? Is it a list of string? It will be more helpful if you could provide a minimal and runnable code snippet :)

@RussellLuo Thanks for the tip. I did that, though the issue really had to do with the implementation of _to_milvus_filter().
The bot really pointed out what needed to be done!
Now I have my own version of MilvusVectorStore and it works beautifully.
Thanks alot!


Font face changes while dragging in the demo

Not sure if that's an OS, browser, or library limitation. Most of my DnD apps drag images, so I've never noticed it before if it's OS or browser.

@mrozbarry Can you provide your OS and browser to help me debug?

Alright, I find the bug at safari. I'll fix it soon. Thanks!


image-tests: skip azure-rhui test on rhel-86

This pull request includes:

[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

I'll test it in osbuild repo this time first

Should be good, I can see azure_rhui test is ran in 8.7 and 9.1 jobs but not in 8.6 https://gitlab.com/redhat/services/products/image-builder/ci/osbuild/-/pipelines/575871232


cli switches for parallel workers and hash verify before closing source and dest files

Your file copy is sound, but I didn't see any switches to adjust the number of threads you copy files with in parallel.
My change request is that you provide a cli clap arg for cpu cores to use . i.e. --cpucorepercenttouse 100
In this use case if your system has 8 cores, then it would use 8 workers to copy 8 files in parallel.
If we want to adjust it to a lower level to all the computer to do other things, that would be useful.
--verifybeforeclosefiles [hashcheckalgo]
In this use case you would want to hashcheck the source against the hashche of the destination after copying the file.  There could be a way to hash check while you have all bytes in memory before the write, that would be cheating since it's all about verify after the write and not before.
Technically after all the bytes are written, but before you close the file, you may ftell the position back to the beginning of the file, read all the bytes on the destination device with the bytecount before you close the destination file after writing it.  That could save precious time since I believe closefile is an expensive os operation.

First, I dont think you will get any speedup from parallelization, more likely the speed would degrade, for example on HDD it means additional head repositioning instead of linear reading.
Next, the topmost progress bar shows the progress of currently copied file, and a line above it shows current filename. What is your idea for multiple files? Show a progress bar and filename for each?

On 4/14/19 7:13 PM, acidnik wrote:

First, I dont think you will get any speedup from parallelization, more likely the speed would degrade, for example on HDD it means additional head repositioning instead of linear reading.
Next, the topmost progress bar shows the progress of currently copied file, and a line above it shows current filename. What is your idea for multiple files? Show a progress bar and filename for each?

I would like to address your misunderstanding about speed degradation
resulting from parallelizing file operations:
1)It depends on the underlying operating system and hardware.  OS can
handle the parallelism and at the lowest-level put it in sequence for
the hard-drive to take it hopefully without disk-chatter so I would
contest that speed-degradation perspective, but the hard-drive's health
and ETA to bad blocks/drive failure will arrive more quickly.  So
eventually you'll have to buy another hard-drive sooner as a result of
parallelizing file operations.  It's expected but a problem since the
data is not necessarily redundant on only one hard-drive.
2)If you are running legacy hard-drives with a more state-of-the-art
configuration such as a parallel file system with many hard-drives(i.e.
orangefs, wekaio, lizardfs), then what you are saying is not necessarily
true.  As stated before bad blocks/drive failure will arrive more
quickly, but it's expected and not a problem since the data is redundant
and drives can be replaced.
3)If you are using a single state-of-the-art NON-MECHANICAL nvm-e drive
then what you are saying is entirely false. nvm-e drive circuitry can
handle parallelism gracefully since they are not mechanical after all.
The more expensive ENTERPRISE LEVEL nvm-e drives especially are designed
for this in mind. NO DISK CHATTER.  NO HEAD POSITIONING WEAR.
4)If you are using many nvm-e drives and with state of the art parallel
file systems with many nvm-e drives, then you will achieve the optimal
performance to parallelize file operations without affecting the life
expectancy of the nvm-e drive since they do not suffer from disk chatter
or head positioning wear.
BOTTOM LINE:  don't just design it for a single mechanical drive.
design it for nvm-e drives....lots of them :)  I do appreciate your
comment about sshfs for remote drive mounts BTW.  Not sure it is the
most optimal way for remote file systems though.  There are many tools
out there, but currently the tool of choice has been
https://en.wikipedia.org/wiki/Rsync
for remote file operations and does have threads control within it.  It
is written in C, but the code is mature(hasn't changed much for a
while).  It could be interesting to see ppcp morph into an rsync like
written in rust.
Now going back to your interface for displaying progress on parallel
file operations.  You have already seen something that does parallelism
with rust.  The cargo build does crazy parallelism with all the cores
you have.  It tells you which packages it is downloading in parallel,
but on one line and flips between them.  The trick isn't in displaying
their progress in parallel, defining what they complete(i.e. like a
defined buffer size of bytes) and then sending a message to a receiving
channel to show progress as you have already done is the trick.
You've got one worker, but flip that into many workers but sending to
the same receiving channel should be good for display progressing for
all of them.
Suggestion for another switch:
"--tui on"
This displays tui progress. If not mentioned on the cli then
defaults on.
"--tui off" just send to stdout as legacy cp/rsync tools do.
I hope this helps to make your tool even better.

Thanks for the clarifications, that was quite interesting read. But I'd really like to see some benchmarks showing that parallel copy (assuming source / destination are the same, as it would be the most common case with ppcp) makes noticeable positive difference.
--tui off will not be implemented. ppcp designed to be an interactive tool, without tui it would be just a poor clone of existing tools

You are asking for a test case to reveal a parallel copy would outperform a sequential copy.
You are also asking whether your tool would be a poor clone of existing non-tui tools if there was non-tui option.
I will address both.
Try creating:
1)5 million 25KiB-size files spread across unique directories holding 4 thousand files.
2)5 million 25KiB-size files spread across unique directories holding 2 thousand files.
3)5 million 25KiB-size files spread across unique directories holding 500 files.
4)1 thousand 1 GiB-size files spread across  10 unique directories holding 100 files.
it sequentially, then do a parallel file copy on 1)one mechanical hard-drive, 2)nvme 3)on the cloud.
Then come back to me that your sequential copy accomplishes the task in the same amount of time.
You're core code can be tweaked to use a parallel iterator with crossbeam sending the results from the parallel workers to tui/non-tui progress receiving channel.
I can appreciate you have a preference for tui, but why would you not offer it to those legacy tool users as a non-tui if they deem it better?

I know how to do benchmarking, you don't have to explain this to me. I was hoping that you already have some benchmarking done (or read somewhere), otherwise it sounds like your statements are pulled out of the thin air.
I just did a quick test:
https://gist.github.com/acidnik/1d418c7851e144ddf5ee89cba279780f
On my machine (macbook with SSD) the numbers are the same (within the margin of error) with any $num_jobs. You can run it yourself and post your results here, maybe that will convince me
Non-tui: I firmly believe that legacy tool users can and should use cp and/or rsync for their legacy tasks. Adding this option to ppcp just not worth the effort


INSTALL.md updated

I have added steps to build prana-osto make the user work easier :).

Thanks for adding this stuff


Refactor SetExplicitMockBehaviorAnalyzer to use KnownSymbols and add CodeFix

This PR accomplishes two goals:

It refactors the SetExplicitMockBehaviorAnalyzer to use KnownSymbols to simplify the analyzer and use the IOperation-based analysis
It adds a corresponding code fixer that provides two entries under the "lightbulb"

Set MockBehavior (Loose)
Set MockBehavior (Strict)

I admit this is bigger than I'd like for a single change. If it's too big to review effectively, let me know and I'll try to break it up.


Increase stopping event test timeouts

Summary
The currently stopping event timeout (10 seconds) isn't always enough in our CI pipelines. Switch to using the general timeout (3 minutes) for all stopping event tests that expect to be stopped early.
For tests that expect to not be stopped by an event (and thus take the full timeout), use a timeout of 15 seconds to avoid slowing down successful test runs.

Release Notes Entry

I believe stopping event support was added in 7.1, so please backport to release/7.x

/backport to release/7.x


Updates and support

Is the library still maintained? What about migrating to the latest versions of flutter/dart? Is the roadmap up to date?

I'm currently working on a completely new implementation without all the janky jsons and enums and stuff, but I also have a slightly upgraded fork on my profile, some things won't work as expected tho, for example if I recall correctly, all images have been swapped to work as URLs.

There is no update on this package for a long time .


Modularize SFX-Standalone

Make SFX-Standalone modularized including:

Make most of the feature of SFX-Standalone be a module.
Every module can expose different components.
Every components can be requested via Dependency Injection.
Have Dependency Injection engine to drive the modularization model.
Be prepared to support SFX-Standalone plugin feature.

#25 Modularization fundamental.
#26 Enhance the logging and fix some bugs.

Checked in with PR #58 #61


Quick question

I've been through the Laracasts' videos last week, and I really liked the work, so thanks for packaging this.
Quick question though, as I don't see this in the usage section: where exactly would you require the package exactly? My initial thought was that you'd probably make a reusable component out of this, but it seems like you didn't, and Jeffrey used the classes he made directly in a "master" js file.
So, say in a more traditional Laravel project, how do you envision this being used?
Would yourequire this package in app.js or bootstrap.js? Would you use the code in all of your component.vue files?  Something else?

Hi,
you're a quick one 😄
This package is currently in development, the API isn't stable yet, documentation still has to be written.
So have a little more patience while we finish up our work. Prudent ETA: mid January 2017.

I've added a little not in the readme explaining that this package isn't finished yet.

HAHA, yeah, I follow you and I was kind of currently playing with Jeffrey's class at the time I was that you created the repo! 🤣


Using strings.Builder in golang v1.10 to reduce memory allocations

According issue #240 , there is no reason why we don't, so I've tried.
And we are facing the real problems, the json.Unmarshal has been in our service's hot path. The Iterator.ReadString and Iterator.readStringSlowPath has been called multiple times per second. Any reasonable patch which could reduce the memory allocation will be valuable.
I've tried to avoid to use any incompatible trick of Golang. So if there is any plan to support new features of golang v1.10, this PR should be helpful I think.

I've checked the issue of codeclimate, it's just facing a bug. The codeclimate has counted the nested function return statement. For the code readability, I think the code should be like this.

is there any benchmark to support this is actual a improvement? why ret = strf.NewString(iter.buf[iter.head:i]) is faster?

here is a benchmark in github.com/junchih/stringx package, https://github.com/junchih/stringx/blob/master/factory_test.go#L18 . you can go with go test -gcflags=-N -bench=. -benchmem ./... to get the result.
the reason why it is faster, you could get from the implementation of stringx and this function https://golang.org/src/strings/builder.go?s=1379:1412#L36 .

iter.buf might be reused, so we are decoding from a reader. Cast the buffer to string, will cause the string point to invalid region later. I used the impl in very early days, and fall back to current impl due to the bug introduced.

hi @taowen , sorry for so long delay. I just made the benchmark to compare before this PR and after. the logs has been a little longer, below.
there are two benchmark cost a little logger time comparing to before. Benchmark_jsoniter_nested and Benchmark_jsoniter_skip, I did the pprof, the line where ReadString and Parse both stands made the different time costing. so I just add another benchmark Benchmark_jsoniter_string, so everything make sense, you could figure out by yourself from the code.
and at the latest two commits, I just move the string factory object to the Iterator struct to reduce the time costing.
at last, to answer your question above, I'm not sure I got your point, sorry. but those calling trace might answer your worry.  here junchih/stringx/factory.go and then here strings/builder.go.
json-iterator:master
$ go test -bench=. -benchmem ./...
...
goos: linux
goarch: amd64
pkg: github.com/json-iterator/go/benchmarks
Benchmark_encode_string_with_SetEscapeHTML       2000000               759 ns/op             760 B/op             5 allocs/op
Benchmark_jsoniter_large_file                     100000             12801 ns/op            4888 B/op            79 allocs/op
Benchmark_json_large_file                          50000             36893 ns/op            6912 B/op            18 allocs/op
...
pkg: github.com/json-iterator/go/misc_tests
Benchmark_jsoniter_array        10000000               202 ns/op               0 B/op          0 allocs/op
Benchmark_json_array             1000000              1656 ns/op             480 B/op         12 allocs/op
Benchmark_jsoniter_float        50000000                29.5 ns/op             0 B/op          0 allocs/op
Benchmark_json_float             3000000               518 ns/op             312 B/op          4 allocs/op
Benchmark_jsoniter_encode_int   100000000               19.1 ns/op             0 B/op          0 allocs/op
Benchmark_itoa                  30000000                61.8 ns/op            16 B/op          1 allocs/op
Benchmark_jsoniter_int          100000000               23.3 ns/op             0 B/op          0 allocs/op
Benchmark_json_int               3000000               507 ns/op             312 B/op          4 allocs/op
Benchmark_jsoniter_nested        3000000               527 ns/op             256 B/op          8 allocs/op
Benchmark_jsoniter_string        3000000               480 ns/op              96 B/op          2 allocs/op
Benchmark_json_nested             500000              2671 ns/op             552 B/op         10 allocs/op
...
pkg: github.com/json-iterator/go/skip_tests
Benchmark_jsoniter_skip          1000000              1662 ns/op             224 B/op         16 allocs/op
Benchmark_json_skip               200000              6948 ns/op             480 B/op         10 allocs/op

junchih:master
$ go test -bench=. -benchmem ./...
...
goos: linux
goarch: amd64
pkg: github.com/json-iterator/go/benchmarks
Benchmark_encode_string_with_SetEscapeHTML       2000000               764 ns/op             760 B/op          5 allocs/op
Benchmark_jsoniter_large_file                     100000             12973 ns/op            5432 B/op         14 allocs/op
Benchmark_json_large_file                          50000             34490 ns/op            6912 B/op         18 allocs/op
...
pkg: github.com/json-iterator/go/misc_tests
Benchmark_jsoniter_array        10000000               206 ns/op               0 B/op          0 allocs/op
Benchmark_json_array             1000000              1681 ns/op             480 B/op         12 allocs/op
Benchmark_jsoniter_float        50000000                28.4 ns/op             0 B/op          0 allocs/op
Benchmark_json_float             3000000               516 ns/op             312 B/op          4 allocs/op
Benchmark_jsoniter_encode_int   100000000               19.2 ns/op             0 B/op          0 allocs/op
Benchmark_itoa                  30000000                62.1 ns/op            16 B/op          1 allocs/op
Benchmark_jsoniter_int          100000000               23.1 ns/op             0 B/op          0 allocs/op
Benchmark_json_int               3000000               506 ns/op             312 B/op          4 allocs/op
Benchmark_jsoniter_nested        2000000               708 ns/op             344 B/op          7 allocs/op
Benchmark_jsoniter_string        3000000               454 ns/op              92 B/op          1 allocs/op
Benchmark_json_nested             500000              2643 ns/op             552 B/op         10 allocs/op
...
pkg: github.com/json-iterator/go/skip_tests
Benchmark_jsoniter_skip          1000000              1955 ns/op             440 B/op          7 allocs/op
Benchmark_json_skip               200000              6583 ns/op             480 B/op         10 allocs/op
...

hi @taowen , will this be merged or not?! it's been a week.

ret = iter.strf.NewString(iter.buf[iter.head:i]) the underlying iter.buf will change. it is not safe to use unsafe pointer to cast the byte array to string. It may work for some benchmark, it will cause problem when the input is a reader, and reading data in chunks.

ret = iter.strf.NewString(iter.buf[iter.head:i]) the underlying iter.buf will change. it is not safe to use unsafe pointer to cast the byte array to string. It may work for some benchmark, it will cause problem when the input is a reader, and reading data in chunks.

do you read the implementation of function iter.strf.NewString? you should not give two comments based on your same old illusion in this PR discussion, since I've already give your code line number like below

the reason why it is faster, you could figure out from the implementation of stringx and this function https://golang.org/src/strings/builder.go?s=1379:1412#L36 .

Very sorry, since so long there is no more progress about this discussion, I'll just close this PR for no more responsibilities of me. Happy day!


[Feature Request] Add image preview to specify start position

Existing Component
Yes
Component Name
el-image
Description
When i render the el-image image list with v-for, whenever i click on an image, the preview always starts with the first image, hope to add the index attribute to specify where the preview starts. [https://jsbin.com/hefepuy/2/edit?html,output]

Same for me, this would be a great improvement

Have you solved the problem? I regenerate the image array when I click Preview
------------------ 原始邮件 ------------------
发件人: "Johannes<EMAIL_ADDRESS>发送时间: 2020年1月6日(星期一) 凌晨1:50
收件人<EMAIL_ADDRESS>抄送: "Kaizhong.Zheng"<946578749<EMAIL_ADDRESS>主题: Re: [ElemeFE/element] [Feature Request] Add image preview to specify start position (#18101)
Same for me, this would be a great improvement
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

fixed

@cs1707 how do you specify an index now?


ValueError: No gt or bboxes

I'm using my own data in coco format and running finetuning on a pretrained net (X-101-64x4d-FPN) using load_from option and distributed training. But it made a mistake.I think this is a BUG that I can't solve.

Did you solve the problem？


Chat: Links with Chinese don't work

Keybase GUI Version: 1.0.22-20170502110012+a3c4856
macOS: 10.12.4

Paste a URLs with Chinese characters into a chat
Link making algorithm ignores Chinese portion of the URL.

Example:
Pasting the link https://www.thestandnews.com/politics/再有政府人員到hidden-agenda-放蛇-警車-大批持盾警員到場欲拘捕-ha-負責人/ into a chat results in a link to https://www.thestandnews.com/politics/

This also happens with Spanish.
For example, https://www.yelp.com/biz/el-fogón-de-geña-austin-2 gets cut off at the ó and it becomes https://www.yelp.com/biz/el-fog.


SuperBuild failed on Windows

A bunch of errors like below
.\ospray\Build\OpenVKL\src\openvkl\drivers\ispc\volume\amr\AMRAccel.cpp(118): error C2666: 'ospcommon::math:: vec_t<float,3,false>::operator []': 4 overloads have similar conversions [.\ospray\Build\OpenVKL\build\openvk l\drivers\ispc\openvkl_module_ispc_driver.vcxproj] [.\ospray\Build\OpenVKL.vcxproj]            .\ospray\build\install\ospcommon\include\ospcommon\math\vec.h(199): note: could be 'T &ospcommon::   math::vec_t<T,3,false>::operator [](const size_t)'                    with                    [                        T=float                    ]            .\ospray\build\install\ospcommon\include\ospcommon\math\vec.h(193): note: or       'const T &ospco   mmon::math::vec_t<T,3,false>::operator [](const size_t) const'                    with                    [                        T=float                    ]            .\ospray\Build\OpenVKL\src\openvkl\drivers\ispc\volume\amr\AMRAccel.cpp(118): note: or       'buil   t-in C++ operator[(T, int)'                    with                    [                        T=float                    ]            .\ospray\Build\OpenVKL\src\openvkl\drivers\ispc\volume\amr\AMRAccel.cpp(118): note: or       'buil   t-in C++ operator[(T, int)'                    with                    [                        T=float                    ]            .\ospray\Build\OpenVKL\src\openvkl\drivers\ispc\volume\amr\AMRAccel.cpp(118): note: while trying t   o match the argument list '(ospcommon::math::vec3f, int)'
Looks like in ".\ospcommon\include\ospcommon\math\vec.h", there is ambiguity between
   inline const T &operator[](const size_t idx) const     {       assert(idx < 2);       return (&x)[idx];     }
and the operator*() converter.

Thanks for this report! What MSVC version are you using?

Oh, sorry to miss that. It's MSVC 2017.
There is a stackoverflow post for this kind of issue:
https://stackoverflow.com/questions/1726740/c-error-operator-2-overloads-have-similar-conversions


SQLite - backup/restore error

I'm having problem executing SQLite command.
_, err := database.Exec(".backup database.bak")
...
_, err := database.Exec(".restore database.bak")

Output:
near \".\": syntax error

The .backup and .restore commands are offerings of the SQLite shell, not of its SQL dialect. For backup api, use the dedicated funcs from github.com/mattn/go-sqlite3 package.

The .backup and .restore commands are offerings of the SQLite shell, not of its SQL dialect.

lol so that was it. OK! I had encountered this problem in the college project.

For backup api, use the dedicated funcs from github.com/mattn/go-sqlite3 package.

We're avoiding more dependencies so I think trying to run processes directly helps.

We're avoiding more dependencies so I think trying to run processes directly helps.

If you're using SQLite, chances are you're already using mattn/go-sqlite3 or modernc.org/sqlite. They are the defacto standard SQLite drivers in Go, and sqlx doesn't work with SQLite without either of them.

We're avoiding more dependencies so I think trying to run processes directly helps.

If you're using SQLite, chances are you're already using mattn/go-sqlite3 or modernc.org/sqlite. They are the defacto standard SQLite drivers in Go, and sqlx doesn't work with SQLite without either of them.

Nice tip. thx!


Customization tab in library properties crashes files

Description
The app crashes when trying to customise a library from the properties window
Steps To Reproduce

right click a library
properties
customization

Expected behavior
This should open a panel to change the library icon or add folders
Files Version
<IP_ADDRESS>
Windows Version
Windows 11 22000.318
Relevant Assets
No response

Can you attach log files?

#6914 should be the same issue and has the log files. Would still be useful to have  @currall's log files too for confirmation.

This issue should be resolved with #7004.


Mohist

What is your Quests version / build number (do not say "latest")?
Quests 5.0.2-b469
Describe the problem and how to replicate it.
using core Mohist 1.20.1-596 , when I tested this in another version of mohist everything worked as I described.
When i try to create quest or edit no one command like a input "1- Create new quest" or "2 - Edit a quest", not working.
When i try to input 1, 2 ,3  to chat nothing happens, but when i click on them it works, and when i input a quest name its too not working.
Any console errors to share?
no errors

If applicable, can you provide an example quest from quests.yml?
No response

Mohist is not officially supported, but I'll try to find time to replicate this.

Mohist is not officially supported, but I'll try to find time to replicate this.

Thank you!

I've submitted a bug report to Mohist's issue tracker, linked above. Please direct further inquiries to that ticket.


Failing "Bullshit" does not give an option to "Restart" or Display Correctly on Raid End

Failing "Bullshit" does not give an option to "Restart" or Display Correctly on Raid End. Still says "Task Failed" and where it should say "restart" it says "complete".

Accept Quest "Bullshit"
Kill scav on customs to fail quest
On raid end, look at quest while in stash

DOES NOT CHANGE ON CLIENT RESTART
Screenshots look the exact same but were taken before and after restart of server and client

Attachments

66d5e7f4000507b57b24745c.json (1597 KiB)
LogOutput.log (7 KiB)
2024.09.10_13-18-51_<IP_ADDRESS>.32291_backend.log (18 KiB)
server-2024-09-10.log (123 KiB)
2024-09-10[09-23] (0).png (1995 KiB)

Originally written by lukeondiscord

Fixed with: https://github.com/sp-tarkov/server/commit/e650271d3a903f890983ecafcfa289526663570e

Originally written by chomp


Make MessageAdapter.send_message thread safe

Context:
Fixes https://github.com/redox-os/orbtk/issues/464
@FloVanGH
Contribution checklist:

[ ] Add documentation to all public structs, traits and functions.
[ ] Add unit tests if possible
[ ] Describe the major change(s) in the CHANGELOG.MD
[ ] Run cargo fmt to make the formatting consistent across the codebase
[ ] Run cargo clippy to check with the linter

Thank you


Converting K8s LBs to internal LBs

What happens when your PR merges?
The Kubernetes services for Admin, API, Documentation, and Document Download will be configured to be internal load balancers instead of public.
There is manual intervention required for this release
This release requires ~5 Minutes of downtime
Release Steps

Merge to main/release to prod w/ regular pipeline
On a local machine, set kubernetes context to target environment
Run the script ./scripts/recreateServices.sh passing the folder name of the kustomization for that environment.
example: ./recreateServices.sh staging
The services will be deleted and then recreated.

What are you changing?

[X] Changing kubernetes configuration

Provide some background on the changes
This is a security fix to eliminate endpoints that are not behind the firewall.
Checklist if making changes to Kubernetes:

[X] I know how to get kubectl credentials in case it catches on fire

Is there a way we can avoid downtime for this? What would be required effort versus no downtime?

Is there a way we can avoid downtime for this? What would be required effort versus no downtime?

I've done a PoC on how to do this without downtime in my scratch. The flow would go as follows:

Merge a PR to notification-terraform doing the following:

Create 4 new target groups, one for each service
Modify the default action of the aws_lb_listener.notification-canada-ca as follows:

    type             = "forward"
    forward {
      target_group {
        arn =  aws_alb_target_group.notification-canada-ca-admin.arn
      }
      target_group {
        arn =  aws_alb_target_group.notify-admin-bak.arn
      }
    }
#    target_group_arn = aws_alb_target_group.notification-canada-ca-admin.arn
  }

Modify the aws_lb_listener_rule for the other 3 services to include the backup target group

Merge a PR to notification-manifests that does the following:

Creates 4 new "backup" services - one per app... ex "admin-backup"
Creates 4 new targetGroupBindings - to bind the backup services to the new targetGroups
Adds the internal alb annotations to the existing services

Run the recreateServices.sh script in staging
Revert notification-terraform PR
Create a new PR to remove the backup services and targetgroupbindings in notifications-manifests

Closing this PR in favour of https://github.com/cds-snc/notification-manifests/pull/1951


Make final updates to install guide

make sure to include info about possible issue creating initial portal administrator account.

Assume this is for this upcoming release. Setting milestone.

@conklinbd yes, just forgot to set the milestone.

Fixed in Esri\ops-server-config v.next. Also included in TEM 2015 build 20151011_0757. Closing issue.


Make wallet address entry mandatory when setting up a farm

Many farmers forget to enter their wallet address while setting up their farm. please make the wallet address field mandatory:

@andhartl Can you try on staging website, does this workflow is good enough for you ?


🌱  Tunnel: Validate namespace/pod at the syncer side

Summary
This PR adds an extra layer of security at the syncer side of the tunneler to ensure that the proxied requests are against a namespace owned by the syncer and a pod in upsync state.
Related issue(s)
Fixes https://github.com/kcp-dev/kcp/issues/1975

/retest

/retest

/retest

/retest

/retest

Could we complete unit tests to cover the new cases ?

Done, PTAL

/lgtm

/retest

/lgtm

/approve


Remove Terraform Integration test script

Describe the bug
The Integration-tests scripts that exists in this repo are currently not used. but these scripts are used in in the aws-otel-lambda distribution for the integration testing, the aws-otel-lambda use this whole repo as a sub module. This issue is created to bring this discussion into light whether to port over this script to aws-otel-lambda repo completely or determine if these scripts can be used for integ tests for this repo.

Until there's a need for tf files in this repo, my vote is to remove them here.


Revert "Drop s3 read rate limit to 768"

Reverts attic-labs/noms#3196
Based on further logging, I'm not sure this will help. So, revert for the time being.

TBR


Wordpress now on 6.6

Link to product page on endoflife.date
Let us know the URL of the product page on endoflife.date you found the incorrect details.
https://endoflife.date/wordpress
Details of incorrect and correct details you have found
https://wordpress.org/download/releases/
says 6.6 is the only supported page
as of Jul-16-2024
For example, the active support date on endoflife.date is not the same as in the product documentation.
endoflife.date: 2021-02-02
Product documentation: 2022-02-02
What is the source website for the product and for its version information?
For example: https://github.com/PowerShell/PowerShell/tags
Additional context
Add any other context or screenshots about the incorrect details.

Thanks for letting us know, I just filed PR #5466 to update the entries.


chore(wasm): add CI workflow

References #3475

Attempting to execute the workflow in headless mode seems to fail. Alternatively, the interactive mode (without the headless flag) stalls indefinitely since the CI environment can't interact with it. We may need to rethink our approach to testing the WASM crate.

Nice start! Looks like the new test isn't passing yet, so I'll check back when it's green. N.B. we already have a "wasm" job inside the rust workflow—perhaps this new test should be added around here: https://github.com/penumbra-zone/penumbra/blob/b56182336aff03e5d68dd4cfccc60f886b11d0ee/.github/workflows/rust.yml#L105 (I realize the parent ticket mentioned a new workflow, but I'm being persnickety. <3) Currently, the only wasm-related CI we run on this repo is "make sure the wasm builds OK." Is that check still useful if we're actually running tests? That is, doesn't running the tests involve building before running?
Holler if you want more eyes on this.

Tagging @conorsch if you have any suggestions?

Ha, I hadn't seen your comment when I wrote my message. Thanks for pinging.

Attempting to execute the workflow in headless mode seems to fail. Alternatively, the interactive mode (without the headless flag) compiles successfully, but stalls indefinitely since the CI environment can't interact with it.

Can you teach me how to run this test locally? I see the new command added to both the wasm crate's README and the CI job: wasm-pack test --chrome -- --test test_build --target wasm32-unknown-unknown --release --features "mock-database" but when I run that locally, I get stuck on:
Set timeout to 20 seconds...
Interactive browsers tests are now available at http://<IP_ADDRESS>:8000

Note that interactive mode is enabled because `NO_HEADLESS`
is specified in the environment of this process. Once you're
done with testing you'll need to kill this server with

which is odd, because I definitely don't have NO_HEADLESS exported in my environment. Are you able to run these tests locally? If so, and you can teach me to do the same, I'm sure we can make CI happy.

You're running it properly! Since it's an interactive browser, you need to open http://<IP_ADDRESS>:8000 to start executing the test. Alternatively running it in headless mode locally fails.

... fails locally.

Yup, same here, that command fails. Likely we'll have to prepare a webdriver.json config or something. But my recommendation is to forget about the CI workflows until we have a testing command that executes reliably locally. For instance, this command does pass for me locally:
❯ wasm-pack test --headless --firefox -- --test test_build --target wasm32-unknown-unknown --release --features "mock-database"
[...snip...]
Set timeout to 20 seconds...
Running headless tests in Firefox on `http://<IP_ADDRESS>:42199/`
Try find `webdriver.json` for configure browser's capabilities:
Not found
running 1 test

test test_build::tests::mock_build_serial_and_parallel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored

That's a start, at least!

This was backlogged in favor of more higher priority web related work. @conorsch Strangely wasm-pack test --headless --firefox -- --test test_build --target wasm32-unknown-unknown --release --features "mock-database" passes locally in the wasm-ci-cd branch, but fails in the main branch. What would the reason for that be since this branch doesn't have any changes that should affect that?

tagging @grod220

Strangely wasm-pack test --headless --firefox -- --test test_build --target wasm32-unknown-unknown --release --features "mock-database" passes locally in the wasm-ci-cd branch, but fails in the main branch. What would the reason for that be since this branch doesn't have any changes that should affect that?

That's not clear to me, either, but FWIW I can definitely reproduce the behavior you describe. There are changes to the wasm tests in this diff, but even copying crates/wasm/tests/test_build.rs on main and rerunning the tests still fails.

Sadly, won't be able to dive into this before my holiday given other high-priority rpc changes before we freeze those. Think we can de-prioritize this and revisit later.

fails in the main branch

As of merge of #3586, I'm able to run wasm-pack test --headless --firefox -- --test test_build --target wasm32-unknown-unknown --release --features "mock-database" on main and it passes! I'm going to rebase this branch on top of latest main and poke at the new test.

@conorsch might need webdriver.json config to satisfy CI?

I've made sure I can run the wasm-pack headless test command on both my local workstation, and also on a truly headless setup (rented cloud box), and no problems in either place. So there's something different about the GHA context that we're overlooking. I'll dig into this more.

Finally sorted out the weirdness between local env and CI env: the WIP CI job didn't have lfs=true, so the proving keys being included in the tests weren't real. The error was occurring on https://github.com/penumbra-zone/penumbra/blob/9a5bb7fc18a03137938d5be3def146bf55033110/crates/wasm/tests/test_build.rs#L57 which gets data from https://github.com/penumbra-zone/penumbra/blob/9a5bb7fc18a03137938d5be3def146bf55033110/crates/wasm/tests/test_build.rs#L45
Then simply adding lfs=true to the checkout task worked. I've since reorganized a bit, adding the headless browser tests to the existing WASM workflow. When we next discuss breaking the wasm crate out of the workspace, we can rethink how best to map the tests to the testing coverage we want. For now, the headless browser tests are running just the test_build setup, against headless Firefox, which I declare Good Enough™ for merge.

cc @grod220

wow! amazing work 🎊


fix typos

Thanks - all the files in api/class* are auto-generated from the C++ source in avogadrolibs - I suspect the typo is somewhere in avogadro/io/ since it's showing up in all those cases.
The main thing is to fix the source.


[Fix]: yaml for Daint introduced by clean-up

launch jenkins spack install -v --test==root int2lm@c2sm_master%pgi

launch jenkins spack install -v --test=root int2lm@c2sm_master%pgi

@elsagermann @lxavier The problem is the precedence of the different compiiler versions.
I think we need to find a strategy how to deal with new compilers and defaults on the package level, otherwise we will run into such problems all the time.

launch jenkins spack installcosmo --test==root cosmo@master%pgi cosmo_target=gpu

launch jenkins spack installcosmo --test=root cosmo@master%pgi cosmo_target=gpu

This PR solves two problems:

Fix compilation of COSMO-GPU on Daint
Fix int2lm compilation with PGI on Daint
@elsagermann This will probably affect the packages that need the nvidia-compiler. Could you check that it does not break anything there?

@elsagermann Can you check this changes for Icon?
I would like to merge this fix tomorrow.


United location for API credentials

Each widget instance shouldn't need to provide the creds.
Solo options page?
Tag onto existing settings page?

If dedicated page, make it a submenu item of "Settings". It doesn't need a top level item.

/wp-admin/options-general.php?page=mb_untappd_settings
Under "Settings" menu item.


Update README to container-native development

Purpose
The README file should be in line with FUN's approach to container-native development.
Proposal

[ ] Move "standard" installation procedure to /docs
[ ] How to build/run/lint/test Richie using Docker
[ ] Code formatting expectations

I guess this issue has been taken care of?


Rearrange Code to fix Circular Imports

Closes #3616
Code Changes

[x] move scope_registry to fides.common.api
[x] move config up one level to be a top-level module (fides.config)
[x] remove echo_green and echo_red from fides.core.utils

Steps to Confirm

[ ] list any manual steps for reviewers to confirm the changes

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated

Description Of Changes
Write some things here about the changes and any potential caveats

@adamsachs I think I got the circular dependencies all fixed, but now I'm hitting this error when running the migration command
 from cryptography.hazmat.primitives.ciphers.aead import AESGCM
ModuleNotFoundError: No module named 'cryptography.hazmat'

Thanks for your work so far @ThomasLaPiana.  Just flagging this will cause corresponding fidesplus changes to be made too since we've changed the locations of some pieces that are imported there.

hmm, seems like our own src/fides/api/cryptography/__init__.py module is what the cryptography import is resolving to within the dev docker image, rather than cryptography resolving to the 3rd party package. trying to track down what could have changed recently in our local setup to make that be the case...

hmm, seems like our own src/fides/api/cryptography/__init__.py module is what the cryptography import is resolving to when at /fides/src/fides/api within the docker image, rather than cryptography resolving to the 3rd party package/library we expect. i think this is just a product of code reorg bringing up a naming conflict, since before we didn't have a cryptography in the same dir as the alembic.ini where we run our alembic commands from.
could we just move the alembic.ini (and perhaps the /migrations subdir it relies upon) into a dedicated alembic or something similar subdir to prevent any namespace conflicts?

This makes a ton of sense, might as well!
from The Zen of Python:

Flat is better than nested

@adamsachs @pattisdr good to go for another review! Looks good to me :)

hooray @ThomasLaPiana and @adamsachs thanks again


Cache dependency crates

This PR picks up from #1086.
Credit to @justinmoon for inspiring me to look into github caching and the workings of cargo. Learned a lot :grin:
This adds caching of the dependency crates and crates.io index that are currently downloaded each time a job starts.
A separate update-deps job have been added where the crates will be cached and shared between the rest of the jobs.
If the dependencies change then the cache will be updated.
This has the extra benefit that we ensure each job in a workflow use the exact same dependency versions even if an update got pushed while the CI was setting up the different jobs.
The cache hit might miss on windows currently since it uses a different compression algorithm due to a bug. Will hopefully get fixed on it's own.
Since we currently don't cache the build target we can also get a small compile speed boost by setting CARGO_INCREMENTAL to 0.
Caching of the build target will be a separate PR since it is a bit more complicated to do properly with the large build sizes and the need to clean old build artifacts.
The code is heavily inspired from the Testing a library example in this PR.  So also credit to @mzabaluev
Tho i have decided against having a separate cache for the crates.io index as shown in the example.
I don't think the double digit savings in seconds by not downloading the full index each workflow is worth the extra lines of code it would add.

Investigating why clippy is failng. Didn't during testing.

I found during some testing that it might still download some crates in the main job. But it seems to only happen on windows.
Not sure what is happening since the fetch command in update-deps should fetch for all targets.
Investigating.

Having second thoughts on caching only the download dependency crates this way.
Looking more into the numbers i think this is not worth it alone currently, because the savings are relatively small.
Combined with the fact that Github cache restore times seems to fluctuate quite a lot as of this moment.
Sometimes it takes seconds to restore a cache and other times it can take minutes even for the same cache files.
I do think the approach of generating Cargo.lock in one job and then sharing that in the other jobs has some value tho.
I might open a new PR for caching the build target since that is where the CI spends the most of it's time. Some real gains in time can be achieved there.
But i'm currently having problems satisfying both the Github cache size limits while still being able successfully build from the cache on all targets. At least on stable Rust
Looking into it still tho so will try to return with something better :grin:


remove -R flag gcc does not understand

On Ubuntu 16.04 LTS
I used cmake to build shared libraries for SCIP and installed them into /usr/local,
which SCIP seems to hold as the default install location.
I was trying to install PySCIPOpt via:
$ git clone https://github.com/SCIP-Interfaces/PySCIPOpt
$ cd PySCIPOpt/
$ python setup.py build

with Anaconda python 2.7 when I got this error:
gcc -pthread -B /mnt/data/anaconda2/compiler_compat -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/darrenrhea/PySCIPOpt/include -I/mnt/data/anaconda2/include/python2.7 -c src/pyscipopt/scip.c -o build/temp.linux-x86_64-2.7/src/pyscipopt/scip.o

gcc -pthread -shared -B /mnt/data/anaconda2/compiler_compat -L/mnt/data/anaconda2/lib -Wl,-rpath=/mnt/data/anaconda2/lib,--no-as-needed build/temp.linux-x86_64-2.7/src/pyscipopt/scip.o -L/home/darrenrhea/PySCIPOpt/lib -L/mnt/data/anaconda2/lib -R/home/darrenrhea/PySCIPOpt/lib -lscip -lpython2.7 -o build/lib.linux-x86_64-2.7/pyscipopt/scip.so

gcc: error: unrecognized command line option ‘-R’
error: command 'gcc' failed with exit status 1

This is my attempt to fix this by doing for Linux what is done for Darwin.  Instead of -R it is now doing -Wl,-rpath,/usr/local/lib instead:
gcc -pthread -B /mnt/data/anaconda2/compiler_compat -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/mnt/data/anaconda2/include/python2.7 -c src/pyscipopt/scip.c -o build/temp.linux-x86_64-2.7/src/pyscipopt/scip.o
gcc -pthread -shared -B /mnt/data/anaconda2/compiler_compat -L/mnt/data/anaconda2/lib -Wl,-rpath=/mnt/data/anaconda2/lib,--no-as-needed build/temp.linux-x86_64-2.7/src/pyscipopt/scip.o -L/usr/local/lib -L/mnt/data/anaconda2/lib -lscip -lpython2.7 -o build/lib.linux-x86_64-2.7/pyscipopt/scip.so -Wl,-rpath,/usr/local/lib

All of the examples found in  the examples/finished directory run after making this change to setup.py and doing
$ python setup.py build
$ python setup.py install

Maybe there is a better way, I don't know.

Codecov Report

Merging #95 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #95   +/-   ##
=======================================
  Coverage   72.14%   72.14%
=======================================
  Files          15       15
  Lines         980      980
=======================================
  Hits          707      707
  Misses        273      273

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74e2819...d80fbc5. Read the comment docs.

I implemented the second change request on the master branch and will close this PR now, because of inactivity.


SLayer's derivative wrt input

Hello, I have a question about using SLayer. Could you please clarify why isn't it possible to differentiate w.r.t. its input?
Thank you

This is possible. Ensure that requires_grad is set to True for input tensors.


CLI Changes Sub-PR: Visualisation subcommand

This forms a part of PR #583 and must not be merged into main alone. It is only the implementation of isolated functionality -- and thus not expected to necessarily pass the CI checks -- to split up the work into logical chunks.

This PR contains a lot of seemingly unnecessary machinery to achieve its goals. While it's not necessary for re-implementing the visualisation subcommand, it will be useful for the formatting subcommand...


Add development instructions for Polymer 3 convertion [skip ci]

Fixes #174

This change is

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

DEVELOPMENT.md, line 84 at r1 (raw file):

In order to use a Vaadin component in Polymer 3, the codebase has to
be converted first. Follow these steps to convert a Vaadin component
in your local repository:

The second sentence is not clear to me, I would simply in the way:
Change to the directory of the Vaadin component you want to convert, and follow these steps:

Comments from Reviewable

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

a discussion (no related file):
Consider also adding some mention that unlike Polymer elements (which have auto generated branches), Polymer 3 versions of Vaadin components are only published to npm, not pushed to GitHub. Any proposed PRs should target master branch of component and P2 codebase.

DEVELOPMENT.md, line 107 at r1 (raw file):

# Run development server
$ polymer serve pols --npm --module-resolution node --open

what is "pols"? was it your personal shortcut?

Comments from Reviewable

Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions.

a discussion (no related file):
Previously, web-padawan (Sergey Kulikov) wrote…
Consider also adding some mention that unlike Polymer elements (which have auto generated branches), Polymer 3 versions of Vaadin components are only published to npm, not pushed to GitHub. Any proposed PRs should target master branch of component and P2 codebase.

Good suggestion, done.

DEVELOPMENT.md, line 84 at r1 (raw file):
Previously, manolo (Manuel Carrasco Moñino) wrote…
The second sentence is not clear to me, I would simply in the way:
Change to the directory of the Vaadin component you want to convert, and follow these steps:

Better now?

DEVELOPMENT.md, line 107 at r1 (raw file):
Previously, web-padawan (Sergey Kulikov) wrote…
what is "pols"? was it your personal shortcut?

oops, yes. Removed.

Comments from Reviewable

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

DEVELOPMENT.md, line 84 at r1 (raw file):
Previously, platosha (Anton Platonov) wrote…
Better now?

yep

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable