[SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC

What changes were proposed in this pull request?

This pull request is to support query execution before reading DataFrame and after reading/writing DataFrame over JDBC.
There are two new options; preActions and postActions in DataFrame's JDBC configuration.
SQL statements specified in preActions will be executed before reading/writing DataFrame via JDBC.
SQL statements specified in postActions will be executed after writing DataFrame via JDBC.
Note: postActions is only supported in JDBC writer, not in JDBC reader. It is because it won't be needed so often and I was not able to find good places to implement this.
Why are the changes needed?

For ETL workload, there is a common requirement to perform SQL statement before/after reading/writing over JDBC.
Here's examples;

Create a view with specific conditions
Delete/Update some records
Truncate a table (it is already possible in truncate option)
Execute stored procedure (it is also requested in SPARK-32014)

See details in the JIRA. https://issues.apache.org/jira/browse/SPARK-32013
Does this PR introduce any user-facing change?

Yes.
With this feature, users can run any SQL statements before/after reading/writing DataFrame over JDBC.
How to use it
Here's two examples which describes how to use the new options.

Read

val df = spark.read.format("jdbc")
      .option("url", "jdbc:someUrl")
      .option("dbtable", "TEST.PEOPLE")
      .option("preActions", "delete from test.people where name = 'fred'")
      .load()

Write

    df.write.format("jdbc")
      .option("url", "jdbc:someUrl")
      .option("dbtable", "TEST.CUSTOMQUERY")
      .option("postActions", "insert into TEST.CUSTOMQUERY values ('fred', 1)")
      .options(properties.asScala)
      .save()

It does not affect any existing behavior. It just adds new options to use this new feature.
How was this patch tested?

I added test cases into JDBCSuite.scala and JDBCWriteSuite.scala, and confirmed all the tests have been passed.

@moomindani Thanks. Could we illustrate the usage of these two options via examples in the PR description ? I think, it will help the reviewers.

@dilipbiswal Sure I will add it in this PR description.

ok to test

For ETL workload, there is a common requirement to perform SQL statement before/after reading/writing over JDBC.
Here's examples;
Create a view with specific conditions
Delete/Update some records
Truncate a table (it is already possible in truncate option)
Execute stored procedure (it is also requested in SPARK-32014)

@gatorsmile WDYT the usecases described above? I'm not sure it's reasonable or not because I've never met users making such a request.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@moomindani Thanks a lot for working on it. Would be nice to wait on answer to @maropu's question.

ok to test

Test build #124703 has started for PR 28953 at commit 47ef483.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29314/
Test PASSed.

Test build #124703 has finished for PR 28953 at commit 47ef483.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124703/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #124713 has started for PR 28953 at commit 47ef483.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29324/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29326/
Test PASSed.

Test build #124715 has started for PR 28953 at commit f2e68a2.

Test build #124713 has finished for PR 28953 at commit 47ef483.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124713/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29357/
Test PASSed.

Test build #124745 has started for PR 28953 at commit 0db0376.

Test build #124745 has finished for PR 28953 at commit 0db0376.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124745/
Test FAILed.

Test build #124715 has finished for PR 28953 at commit f2e68a2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124715/
Test FAILed.

retest this please

Test build #124748 has started for PR 28953 at commit 0db0376.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29360/
Test PASSed.

Test build #124748 has finished for PR 28953 at commit 0db0376.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124748/
Test FAILed.

retest this please

Test build #124864 has started for PR 28953 at commit 0db0376.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29471/
Test PASSed.

Test build #124864 has finished for PR 28953 at commit 0db0376.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124864/
Test FAILed.

retest this please

Test build #124908 has started for PR 28953 at commit 0db0376.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29514/
Test PASSed.

Test build #124908 has finished for PR 28953 at commit 0db0376.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124908/
Test FAILed.

retest this please

Test build #125033 has started for PR 28953 at commit 0db0376.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29640/
Test PASSed.

Test build #125036 has started for PR 28953 at commit a75d1e8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29643/
Test PASSed.

Test build #125036 has finished for PR 28953 at commit a75d1e8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #125033 has finished for PR 28953 at commit 0db0376.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125036/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125033/
Test FAILed.

retest this please

Test build #125310 has started for PR 28953 at commit a75d1e8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29921/
Test PASSed.

Test build #125310 has finished for PR 28953 at commit a75d1e8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125310/
Test FAILed.

retest this please

Test build #125318 has started for PR 28953 at commit a75d1e8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29930/
Test PASSed.

btw, postActions is only for write paths? The title does not say so...

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125318/
Test FAILed.

Test build #125464 has started for PR 28953 at commit 244b325.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30077/
Test PASSed.

Test build #125464 has finished for PR 28953 at commit 244b325.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125464/
Test FAILed.

Test build #125465 has started for PR 28953 at commit 9fb8383.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30078/
Test PASSed.

btw, postActions is only for write paths? The title does not say so...

Right. I modified the title.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125465/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30149/
Test PASSed.

Test build #125535 has started for PR 28953 at commit 9fb8383.

Test build #125535 has finished for PR 28953 at commit 9fb8383.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125535/
Test FAILed.

retest this please

Test build #125711 has started for PR 28953 at commit 9fb8383.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30324/
Test PASSed.

Test build #125711 has finished for PR 28953 at commit 9fb8383.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125711/
Test FAILed.

retest this please

Test build #125716 has started for PR 28953 at commit 9fb8383.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30329/
Test PASSed.

Test build #125716 has finished for PR 28953 at commit 9fb8383.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125716/
Test FAILed.

retest this please

Test build #125741 has started for PR 28953 at commit 9fb8383.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30351/
Test PASSed.

Test build #125741 has finished for PR 28953 at commit 9fb8383.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125741/
Test PASSed.

@moomindani now, the conflict happens. Could you please resolve it?

Test build #126950 has started for PR 28953 at commit 7ca42b1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31574/
Test PASSed.

Test build #126950 has finished for PR 28953 at commit 7ca42b1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126950/
Test FAILed.

Test build #126952 has started for PR 28953 at commit 70d8719.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31576/
Test PASSed.

Test build #126952 has finished for PR 28953 at commit 70d8719.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126952/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31581/
Test PASSed.

Test build #126957 has started for PR 28953 at commit 311a89e.

Test build #126957 has finished for PR 28953 at commit 311a89e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126957/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31589/
Test PASSed.

Test build #126966 has started for PR 28953 at commit ae7afd2.

Test build #126966 has finished for PR 28953 at commit ae7afd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126966/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31674/
Test PASSed.

Test build #127052 has started for PR 28953 at commit fbac1f3.

Test build #127052 has finished for PR 28953 at commit fbac1f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127052/
Test PASSed.

Test build #127620 has started for PR 28953 at commit 13f0dfc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32244/
Test PASSed.

Test build #127620 has finished for PR 28953 at commit 13f0dfc.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127620/
Test FAILed.

Test build #127632 has started for PR 28953 at commit 62d7fc0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32256/
Test PASSed.

Test build #127632 has finished for PR 28953 at commit 62d7fc0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127632/
Test FAILed.

Test build #127649 has started for PR 28953 at commit 979cec5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32272/
Test PASSed.

Test build #127649 has finished for PR 28953 at commit 979cec5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127649/
Test PASSed.

Test build #127681 has started for PR 28953 at commit 3244c30.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32305/
Test PASSed.

Test build #127681 has finished for PR 28953 at commit 3244c30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127681/
Test PASSed.

I resolved recent conflicts.
Folks, can you review this and proceed it? I thought there are no blockers now.

@gatorsmile Thank you for your comment.
I understand that this kind of hooks are not specific to JDBC, it can be general to all the data sources.
However, for JDBC, it should be executed in data source side. This part can be specific to JDBC (or other data sources which include execution engine.).
Of course we can have discussion in DSV2 API design, however, can we proceed this PR separately?

@gatorsmile Just a reminder.. Can you take a look?

This sounds like JDBC specific, as we can't use SQL to define actions for data sources like parquet.
To clarify your use case, do you really need hooks, or just need the ability to run arbitrary SQL against the JDBC server with Spark?

@cloud-fan We need hooks right before/afterJDBC reads/writes especially for ETL purpose. It is also frequently asked request from our customers.

Hello everyone, I know this pr has been closed but it would be really helpful if I get a resolution on the same topic. So here I am trying to call snowflake procedure in the aws glue job using Pyspark. Below is the code:
df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("query", "call SP_UTILIZATION_INSIGHTS_LOOKUP_TABLE()").option("preActions", "call SP_UTILIZATION_INSIGHTS_LOOKUP_TABLE()").load()
While execution I'm getting an error as SQL Compilation error: at 'call'.
In the log files what I found is it basically creates a select statement as 'Select * from (call SP_UTILIZATION_INSIGHTS_LOOKUP_TABLE()) where 1=0' and due to this it failed.
Any help here?

@peter-toth I vaguely remembered that you added a new JDBC option to support this case?

@peter-toth , can you share some examples of sessionInitStatement to call the stored procedure?

Can you please try:
df = spark.read.format(SNOWFLAKE_SOURCE_NAME)
  .option("sessionInitStatement", "call SP_UTILIZATION_INSIGHTS_LOOKUP_TABLE()")
  .option("query", "SELECT * FROM your_table_that_is_populated")
  .load()

@peter-toth , I have tried your example but it doesn't call the SP which populates a table. What I was expecting was SP will populate the table and then that table which is used in the "query" but it didn't happen.

@peter-toth, any update on the above query? It's not executing the SP.

@abhishekboga, sorry I can't test this feature with Snowflake, but I added the following simple unittest using H2 into sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:
  test("SPARK-32013: option sessionInitStatement, call stored function") {
    val initSQL =
      """CREATE ALIAS NEXT_PRIME AS '
        |String nextPrime(String value) {
        |  return new BigInteger(value).nextProbablePrime().toString();
        |}';
        |INSERT INTO TEST.seq VALUES(NEXT_PRIME(1000))
        |""".stripMargin
    val df = spark.read.format("jdbc")
      .option("url", urlWithUserAndPass)
      .option("query", "SELECT * FROM TEST.seq WHERE ID > 1000")
      .option("sessionInitStatement", initSQL)
      .load()
    assert(df.collect() === Array(Row(1009)))
  }

and it does work.


extend explicitArray options as a key array for the xml support named array really

Coverage remained the same at 97.633% when pulling 8aee058952bd05d1b90c21720c784ca87ed1a0c4 on logoran:master into e8540dcafaa115484543d8016df39dca8a18d245 on Leonidas-from-XIV:master.

This is very useful and elegant.
Why is this not being merged?

Because I think explicitArray should be always default to produce an object with an obvious structure, and not be extended to have options so the tree that is output gets even more unpredictable.

IMHO, this does give predictable output (since we can define the keys) and also better control over the output.
Specifically, I'm converting xml\json back and forth in my application and I don't want to pollute my jsons with explicit arrays for every field.

You can easily do such a thing as a postprocessing step (unlike the opposite direction), no need to push such auxillary logic into this library.


Set up Gitter Room

It would be great if there would be a Gitter room for BrightFutures

Thanks for the suggestion!
I have no experience with Gitter myself. Can you provide an example of (a) project(s) where this works well for?
I'll have to admit that I am a bit reserved regarding this idea. There are several ways to get support already, e.g. on Twitter, StackOverflow and issues here on GitHub, none of which are used extensively. I'd be worried that the Gitter room would be empty most of the time. In that case it could do more harm than good, since it will be another place where you could find answers to a hand full of questions.
But, as said, I'd like to hear your motivation and see cases where it works well. Thanks again for the suggestion!

Just chipping in here, I mainy program in Scala (started learning Swift recently), and Gitter is a godsend in regards to open source projects
Its basically meant to be a replacement for IRC (not really for stack overflow). For people that have quick and dirty questions, or are looking for general information its really handy. The main thing is, its a really painless integration with github.

I've just opened a Gitter Room for BrightFutures. Let's try it out. Thanks for the suggestion!


Bandit 1.7.0 still picked up by Python 2

Describe the bug
https://github.com/PyCQA/bandit/issues/663 again but with wheels disabled.
It seems that, indeed, bandit does not include the data-requires-python=">=3.5" on its deliverables:
view-source:https://pypi.org/simple/bandit/

The PKG-INFO does contain: Requires-Python: >=3.5
Which is a result of setting in setup.py:
setuptools.setup(
    python_requires='>=3.5',
    setup_requires=['pbr>=2.0.0'],
    pbr=True)

What else might I be missing?  Could you provide a way to recreate?  Thanks

My best bet is that the PyPI tooling did not look into setup.py but setup.cfg and did not find it (Requires-Python) there, hence skipped.

The failure happen, so pip sees it obviously. Only the index got it wrong.

any update on this issue?

any update on this?

The decorator package had similar issue recently: https://github.com/micheles/decorator/issues/102
Maybe the same actions shall be taken for bandit package?

Reopen if this is still an issue. I double checked and followed the instructions on https://packaging.python.org/en/latest/guides/dropping-older-python-versions/ and I still see that Bandit is requiring minimum of py3.8.
tar xfO ~/Downloads/bandit-1.7.6.tar.gz bandit-1.7.6/PKG-INFO
...
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Security
Requires-Python: >=3.8
Provides-Extra: yaml
Provides-Extra: toml
...

Also the generated wheel has the ".py3" extension.
bandit-1.7.6-py3-none-any.whl (123.5 kB view hashes
Uploaded Dec 8, 2023 py3


Fix to compile with Visual C++ and /Zc:implicitNoexcept-.

Otherwise gets:
Error C2694 'override': overriding virtual function
has less restrictive exception specification than base
class virtual member function 'base'
Similar changes are being made e.g.:
https://github.com/boostorg/json/pull/636
And proposed here by me:
https://github.com/boostorg/iostreams/pull/136
I grant there there could be more of this.
These two are just enough for our codebase.

Why would anyone want destructors to not be noexcept by default?

Our code is really not portable, very Windows-specific, fairly hopelessly so, and likely will remain that way for a very long time.
This is not the biggest problem by far.
If we really wanted portability in this regard, we'd run some large tool over our code and mark all destructors as noexcept(false), and pretend we got everything (i.e. ignoring third party code).

whose exception specification will change from compiler to compiler

Isn't it the other way around? This tries to make the exception specification always the same?

@jaykrell CI is working again, if you rebase it'll run everything now.


OWASP false alert for CVE-2023-36414 ?

Query/Question
We are using the azure SDK for Java and are getting OWASP alerts for azure-identity-1.10.4.jar related to CVE-2023-36414 which looks to me like a vulnerability in the dotnet based SDK I suppose this Java SDK may be talking to.
Maven is not listing any vulnerabilities for this library and the fix from Microsoft that is linked to is a NuGet .Net package.
I am looking for more information, hopefully confirming that this OWASP alert is a false positive ?
Why is this not a Bug or a feature Request?
It would be a bug in the OWASP tool or user error where the CPE code listed is to generic, triggering a false alarm.
Setup (please complete the following information if applicable):

OS: Azure / Windows 10
IDE: Intellij Idea
Library/Libraries: azure-identity-1.10.4.jar

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[ ] Query Added
[ ] Setup information Added

Hi @bart-adriaanse
The Security Updates table at the bottom of https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-36415 has the patched build numbers for the .NET, Java, Python, and JS versions of the package.  For Java, that is 1.10.2.


Install shpec on CI

Currently, we're just running shpec directly from the repo. Given the issues addressed in #86, I think it makes sense to actually install shpec on the CI box.
Currently, that'll mean simply running make install as a setup script. In the case of zsh, we'll additionally source the antigen plugin.

Great Idea! :). We could make a script to do that on travis, script that could be used for all the script that are going to use shpec for tests.
Something that would like rejeep script for evm, that you call in your install with a command curl -fsSkL https://gist.github.com/rejeep/7736123/raw > travis.sh && source ./travis.sh  (cf)

Beside the .zsenv  hack we done should be do too! (we could have a template zshenv in the repo)
I restarted my diraction test now that zsh plugin got updated, but I also got the bug about the array!
cf

I started a branch, #88. The idea is simple: actually install and use shpec in the way that we recommend.
While it's currently "working," there's one issue: it doesn't actually run shpec in each specific shell. This means that the disable -r end stuff for zsh doesn't need to be there, because the tests for SHELL=zsh don't really run shpec with the zsh executable.
I began to dig into that, but then I paused: is that really how we're recommending people use shpec? Calling something like zsh bin/shpec or zsh -c 'shpec' doesn't seem right, and that's certainly not the same as what happens when someone runs shpec in a particular shell.
@hlangeveld I know your intention with the POSIX stuff was to make shpec more portable. But what does that mean to the end-user? Are we saying "if you force shpec to run in your POSIX-compliant shell, it'll work"? Because my understanding is that without a shebang, the script will simply run with sh, whatever that happens to be on your machine.
I understand the desire to use your favorite POSIX shell's features within a shpec test. However, I don't believe that's what's going to happen with the desired installation method. Maybe that means we need a better installation method. Whatever the case, I could use some edification/clarification.

about zsh indeed, the only thing we need to get rid of the zsh -c, it to put both setopt sh_word_split and disable -r end in .zshenv

Was trying out to install shepc throught bpkg.
Works perfectly fine as you mike expect. But trying out to run it on zsh, won't work because we don't place a shebang, and so behave as zsh, which is not what we want.
An idea to contourn this, will be to create zshpec; dashpec, kshpec, also installed from bpkg, with their own shebang (and eventual disable), and that will source the shpec.
I gave a try with zsh:
 $  cat /usr/local/bin/zshpec
  #!/bin/zsh
  source "$(dirname $0:A)/shpec"

and it works good!!
Beside we could place it the disable -r end, and setopt....
beside, that will make explicit what bash we are working on.
(and for instance test my own posix shell package with all the different shell by invoking dashpec, zshpec
(for for ci, use a $SHPEC env)
@rylnd What you think? :)

That's a good idea; I'm wondering if/how the installer(s) would decide which one to use. @hlangeveld @locochris do you guys have any thoughts on all this?

I made a draft in the train.
Have a look : https://github.com/AdrieanKhisbe/shpec/tree/features/xshpec.
Ideally the installer would install all of them.
(had a look to the install.sh but didn't get own it really install things. make install felt like infinite loop)

(note: with this the antigen plugin would be somethat obsolette. and bkpg recommanded way to install :) )

On 05/29/15 19:07, Ryland Herrick wrote:

That's a good idea; I'm wondering if/how the installer(s) would decide which one to use. @hlangeveld https://github.com/hlangeveld
@locochris https://github.com/locochris do you guys have any thoughts on all this?

I've been quiet this week; my father in law had a heart attack while on holiday and is now in a hospital half a continent away.
I needed to wrap up some stuff for my last week at work.  Anyone looking to hire an infracoder in NL or remote?
Tomorrow I have an annual meeting for the 50 year old SF association - we get to decide if we continue or disband.
I will give this some thought later in the weekend, or monday.

There's no hurry. :)
Have a good weekend and Courage!!

@AdrieanKhisbe - besides being confusing to the end user, I'm not sure if we need a dashpec, ksphec, zshpec - aren't we running shpec across all shells to prove shpec's posix compliance; ie. that it will work the same in any shell?  (For the rubyists - would you get confused if rspec was called jrspec when you were running tests in jruby?)
@rylnd - the way I see it our current tests are unit tests:

they're actually testing across all shells; which is a good thing
they don't require an installation dependency (on shpec) just to run as unit tests; which is a good thing
they're not being run in the way we recommend in the README; but that should be OK as they're unit tests

That said I think its a good thing (yet rarely done in the wild) if you can test the actual installation (and usage) instructions documented in your README.  So I guess what we're missing here are installation tests.
Perhaps:
shpec/
└── installation
    ├── bash_shpec.sh
    ├── dash_shpec.sh
    ├── ksh_shpec.sh
    └── zsh_shpec.sh

And in the .travis.yml have the matrix:
    - LINT=true
    - UNIT_TEST_SHELL=bash
    - UNIT_TEST_SHELL=dash
    - UNIT_TEST_SHELL=ksh
    - UNIT_TEST_SHELL=zsh
    - INSTALLATION_TEST_SHELL=bash
    - INSTALLATION_TEST_SHELL=dash
    - INSTALLATION_TEST_SHELL=ksh
    - INSTALLATION_TEST_SHELL=zsh

Although I'd probably only bother testing the installation in bash (for a happy path) and zsh (for its complicated path).  And keep in mind, we can only get coverage for these installation tests on ubuntu (ie. on travis); we won't have tests to know if the installation works in other OS's.
@hlangeveld - from more than just a name on the screen, to another name on the screen - some heartfelt thoughts & best wishes.  (BTW "infracoder" = "coding infrastucture"?,  chef etc?; "SF association" = ??)

@locochris I see your point about possible confusion.
The problem is that if just run shpec as a program (letting it somewhere in the PATH), since there is no shebang, it's interpreted as bash or sh (not sure which one).
So it don't have access to shell specificities. it's not zsh, ksh, dash that is running the script;.
Hence the Xshpec with they specific shebang.
The point of posix, was to have a program that is interpreted the same way accross all the shell. (posix ones).

Hello all!:)
Hope you had a good June month!
(just trying to relaunch discussion :) )

Okay, I'm resurrecting this thread. I'm revisiting it because I have a need: reproducing #102. Let's examine the state of things.
The following statements are true, at least to the best of my knowledge:

We test various shells by running the executable directly with the shell (e.g. dash bin/shpec)
Within the test suite, we "run" shpec by calling the (oh so confusingly named) shpec function defined in the executable.
When you run shpec in the prescribed way (add the executable to your PATH), it will always be executed by /bin/sh as it has no shebang.

This one I'm least confident on, but I found no way to change this behavior (besides changing what /bin/sh is, which I think is beyond the scope of this problem).

This leads me to the following conclusions:

Statement 1 means we're not running our suite in the same way that users are. This has already been discussed above, and this is not a problem because
Running the suite in this manner verifies that even if the user somehow did set their system's default shell (/bin/sh) to another POSIX shell everything would still work properly.
I agree with @locochris on the points mentioned above, and this issue can be closed for now.

If there are disagreements with the above, please let me know and we can discuss further.


Strapi plugin doesn't seem to work

When I tried to install the plugin as per manual it doesn't seem to work. When I try to open corresponding tab in browser, it just didn't open.

package.json

{
  "name": "***",
  "private": true,
  "version": "***",
  "description": "A Strapi application",
  "scripts": {
    "develop": "strapi develop",
    "start": "strapi start",
    "build": "strapi build",
    "strapi": "strapi"
  },
  "dependencies": {
    "@strapi/plugin-documentation": "^4.5.5",
    "@strapi/plugin-i18n": "^4.5.5",
    "@strapi/plugin-users-permissions": "^4.5.5",
    "@strapi/strapi": "^4.5.5",
    "better-sqlite3": "7.4.6",
    "mysql": "^2.18.1",
    "strapi-plugin-plausible": "^1.0.5"
  },
  "author": {
    "name": "A Strapi developer"
  },
  "strapi": {
    "uuid": "***"
  },
  "engines": {
    "node": ">=14.19.1 <=18.x.x",
    "npm": ">=6.0.0"
  },
}

plugins.js

module.exports = () => ({
  plausible: {
    config: {
      sharedLink: "https://plausible.io/share/second.level.domain?auth=someToken" // Auth disabled when creating link
    }
  }
});

I've ran npm run build && npm run develop.
Got this:

Issue: nothing is displayed at the plugin page, nothing is displayed at plausible.io console (shows message "Waiting for first pageview
on second.level.domain".

Turned out the problem was in CSP, adding this to config helped:
 {
    name: "strapi::security",
    config: {
      contentSecurityPolicy: {
        useDefaults: true,
        directives: {
          "frame-src": ["https://plausible.io"],
        ...


question /implementation

Hi mike again! :)
i have been using your bus script for quite some time and the AI .. is just terrible
and i feel that the bus  + driver should not despawn , just drive a route like in real life
i have been looking around amongst other scripts , and poking around youtube for bus routes... however every bus script the AI remains terrible ( yours is the least buggy so far )
is it possible to expand your script with own coords ? so that the AI will actually use that route and not despawn ?
( i have seen script that made use of the path.xml and the coords on that )
infact i have made a small blip script which resembles those coords

(not all blips fit ofcourse)

thnx for your quick reply :D
hmm i was thinking .. let this be an empty bus without a driver , which has lanes and stops , the route would be good
the stops , a little akward but .. lets say the AI kicks in when enter/exits the bus ?
eg a vector3 which enables/disables the driver, but remains static
like so : coord x.y.z
(legionsquare approach) AI enable
wait  20 secs (people approach, enter)
( door closes AI kicks in until)
coord x.y.z
resume route
... i will see what i can adjust in your code and make it , less dynamic , but doable :D

i considered writing something that emulated actual bus behaviour.
but gave up after i realized i'd be writing custom handling for fugging every situation that wasnt related to "task go to coords"
Good Luck!
feel free to add me on discord or twitter or something. if you have any problems i'm MORE than happy to share whatever knowledge you might glean from my previous attempts.
I work 14 hour days these days, so i'm not around much but i'll reply when i can.

I have made a couple changes to the existing AI. lowered calls to the AI. it SEEMS better, but it may be less able to stay focused in situations like gunfire.
not really a fix to your request, but something i had been meaning to try for a while that SEEMS better in terms of PED task behaviour.

hi mike , i will check tonight :)


CASMTRIAGE-3475 Update base.packages

Summary and Scope
Summarize what has changed. Explain why this PR is necessary. What is impacted? Is this a new feature, critical bug fix, etc?
Is this change backwards incompatible, backwards compatible, or a backwards compatible bugfix?
Issues and Related PRs
List and characterize relationship to Jira/Github issues and other pull requests. Be sure to list dependencies.

Resolves [issue id](issue link)
Change will also be needed in <insert branch name here>
Future work required by [issue id](issue link)
Documentation changes required in [issue id](issue link)
Merge with/before/after <insert PR URL here>

Testing
List the environments in which these changes were tested.
Tested on:

<development system>
Local development environment
Virtual Shasta

Test description:
How were the changes tested and success verified? If schema changes were part of this change, how were those handled in your upgrade/downgrade testing?

Were the install/upgrade-based validation checks/tests run (goss tests/install-validation doc)?
Were continuous integration tests run? If not, why?
Was upgrade tested? If not, why?
Was downgrade tested? If not, why?
Were new tests (or test issues/Jiras) created for this change?

Risks and Mitigations
Are there known issues with these changes? Any other special considerations?
Pull Request Checklist

[ ] Version number(s) incremented, if applicable
[ ] Copyrights updated
[ ] License file intact
[ ] Target branch correct
[ ] CHANGELOG.md updated
[ ] Testing is appropriate and complete, if applicable
[ ] HPC Product Announcement prepared, if applicable

Fixed the commit:

The RPM is 0.0.34-1 not 0.0.34, common mistake
The RPM also needed a bump in the LiveCD

Rebased. This was branched from an outdated main.


Add Clubstep (Geeser Remix) RobTop Cut Ver. by GeezerOkee

Song Name
Clubstep (Geeser Remix) RobTop Cut Ver.
Artist Name
GeezerOkee
Source
Youtube
Youtube Link
JWUg2ZILWbM
Direct File Link
No response
Song ID
13
Start Offset [ms]
0
Extra
Created from Auto Nong.
Created for level: 59699245

reject. good song, but this song has already been added

reject. good song, but this song has already been added

LMAO WHAT???
this masterpiece have never been added to the nong of song id 13 which is clubstep

accept


"Horia", "Cloșca", "Tichilești" are listed in the drop-down menu

Description:
When user writes "Horia", "Cloșca", "Tichilești" the location appears in the drop-down menu next to the comune and the county which is a part according to the law
Precondition:
The website is up an running.

Step 1
Write in the search bar "Horia"

Expected results
The location appears in the drop-down menu as "Sat Horia, CONSTANȚA, (Horia)"

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Cloșca"

Expected results
The location appears in the drop-down menu as "Sat Cloșca, CONSTANȚA, (Horia)"

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Tichilești"

Expected results
The location appears in the drop-down menu as "Sat Tichilești, CONSTANȚA, (Horia)"


perf: disable five small syntax style linters

DO NOT MERGE: just to see what the performance effect is does not touch the longLine or longFile linters, a benchmarking including them was conducted in #19551.

!bench


pep8 ignores config files in project

The pep8 testing ignores configuration given in config files (.pep8, tox.ini or setup.cfg) in the project (or workspace) folder, when it should read those files.
Having a .pep8file in the user home works, though, but global options are not always appropriate. Interestingly, pylintrcfiles for configuring the static code analysis are picked up correctly.
Windows 8, Spyder 2.3.4, pep8 1.6.2., pylint 1.4.2

FYI, pep8 offers an API interface, so pep8 checking could also be done via an API, like it is done for pyflakes currently. The API can also check for config, including local configuration files.

@bilderbuchi, would you like to submit a PR about it?

I'm not yet familiar enough with the codebase, and currently don't have the free time to do it, unfortunately. :-( I just wanted to record what I found in case it's useful for whoever picks this up in the future.

I'm interested in looking into this at some point soon. Could one of the main devs give a pointer to where and how pep8/pycodestyle is called in the project? I can do the sleuthing myself if necessary though.

I'm interested in looking into this at some point soon. Could one of the main devs give a pointer to where and how pep8/pycodestyle is called in the project? I can do the sleuthing myself if necessary though.

Pycodestyle automaticaly detects config file, the problem is that spyder creates a temporary file and call pycodestyle against that file
I think the best solution is to use the API (as @bilderbuchi suggested) http://pep8.readthedocs.io/en/latest/api.html Change check_with_pep8 function to use that API instead of saving a temp file,and calling the executable.


Completd all

Related Issue
PR Number is :- #887
[Cite any related issue(s) this pull request addresses. If none, simply state "None”]
Added some images .
Description
[Please include a brief description of the changes or features added]
Added some pics in the faq session , changes in the resource images
Type of PR

[ ] Bug fix
[x] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes]

Checklist:

[X] I have performed a self-review of my code
[X] I have read and followed the Contribution Guidelines.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[X] I have commented my code, particularly in hard-to-understand areas.

Additional context:
[Include any additional information or context that might be helpful for reviewers.]
website looks like this

Please don't raise a PR if the issue is not assigned. There was no need to change all the pictures.


Search Bar loses Focus when you don't type fast.

Description
I'm having trouble with Meshery's website's search function. When I type to search for blogs, it shows results for each letter I enter. However, if I don't type quickly, it removes my cursor, which makes the search frustrating. For instance, I need to click the search bar multiple times to search for something like 'How to bind Kubernetes'.
Expected Behavior
The cursor should remain in the search bar without being interrupted or removed, regardless of typing speed.
Screenshots

Enviroment:

OS: Windows
Browser: Chrome
Version:
Device: Desktop

Contributor Guides and Handbook
The meshery.io website uses Jekyll and GitHub Pages. Site content is found under the master branch.

📚 See the Contributing to Meshery.io Website section of the readme.md and other contributing instructions, too.
🎨 Wireframes and designs for Meshery UI in Figma (open invite)
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

@Layak-Ali , this is related to layer5 website . Please open this issue on https://github.com/layer5io/layer5/issues

okay @Ghat0tkach


Thrivecart Embedded Vimeo Video

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a new site support request
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that none of provided URLs violate any copyrights or contain any DRM to the best of my knowledge
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and am willing to share it if required

Region
USA
Example URLs
https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
Provide a description that is worded well enough to be understood
I tried yt-dlp with URL given above. Got "Unsupported URL" message. Content creator states host is Thrivecart. (I am just user, I don't know how to verify).
Next, on the playing video I right-clicked and chose "open frame in new window". The new URL is Vimeo.  https://player.vimeo.com/video/867026229?h=0bcd10eae4&badge=0&autopause=0&player_id=0&app_id=58479    I tried this URL and yt-dlp successfully downloaded the video.
My credentials may be needed, happy to provide privately.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
yt-dlp -vU https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
[debug] Command-line config: ['-vU', 'https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/']
[debug] Portable config "C:\Users\home\Downloads\ytdlp_try\yt-dlp.conf": ['-o', '%(title).150B [%(id)s].%(ext)s']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp [7ea278792] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.22631-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-full_build-www.gyan.dev (setts), ffprobe 7.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1837 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version<EMAIL_ADDRESS>from yt-dlp/yt-dlp
yt-dlp is up to date<EMAIL_ADDRESS>from yt-dlp/yt-dlp)
[generic] Extracting URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
[generic] masterclass-video: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] masterclass-video: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1624, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1759, in __extract_info
  File "yt_dlp\extractor\common.py", line 742, in extract
  File "yt_dlp\extractor\generic.py", line 2553, in _real_extract
yt_dlp.utils.UnsupportedError: Unsupported URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/

If the site requires login, can you add a verbose log where you pass logged-in cookies to yt-dlp please

Yes, how do I mark it private or provide it privately please?

It shouldn't contain any identifying or sensitive information. Cookie values are not recorded in the log

just asking for the output of a command like
yt-dlp --cookies cookies.txt -v "https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/"

Thank you for the assist! Here it is:
yt-dlp --cookies cookies.txt -v "https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/"
[debug] Command-line config: ['--cookies', 'cookies.txt', '-v', 'https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/']
[debug] Portable config "C:\Users\home\Downloads\ytdlp_try\yt-dlp.conf": ['-o', '%(title).150B [%(id)s].%(ext)s']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp [7ea278792] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.22631-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-full_build-www.gyan.dev (setts), ffprobe 7.1-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.08.30, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.2.3, websockets-13.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1837 extractors
[generic] Extracting URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
[generic] masterclass-video: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] masterclass-video: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/
Traceback (most recent call last):
File "yt_dlp\YoutubeDL.py", line 1624, in wrapper
File "yt_dlp\YoutubeDL.py", line 1759, in __extract_info
File "yt_dlp\extractor\common.py", line 742, in extract
File "yt_dlp\extractor\generic.py", line 2553, in _real_extract
yt_dlp.utils.UnsupportedError: Unsupported URL: https://cart.audreyslittlefarm.com/l/how-to-start-grow-a-raised-bed-garden-masterclass/masterclass-video/


🐛 harmonize: Fix harmonize ipython when used on already harmonized datasets

Fixes https://github.com/owid/etl/issues/3559
I encountered the same issue and patched it quickly, I haven't looked thoroughly into it, but it worked for my case.

@antea04 when you have time, could you try it out and see if it solves the issue for you, too?

Thanks @lucasrodes, I'm not sure if the current behavior is what we'd want, but at least now harmonize_ipython can be used when all country names are already harmonized. We can revisit it later and tweak it again in a separate PR.


Audio slider for music overlaps button's when clicking on the button

Made pull request for this issue.  This image was taken in Opera browser so the appearance of the slider is different,  The issue did exist when using chrome as well.

I'd like to add spacing beside the 'Pause Music' button that's identical to the right button for consistency (screenshot)

Submitted a PR


[QUERY] How to set 'Deployment slot setting' when updating slot's application settings?

Library name and version
Azure.ResourceManager.AppService 1.0.2
Query/Question
So I want to set some of my application settings per deployment slot, but I see no respective setting for this option in the AppServiceConfigurationDictionary.
Here's the code I tried using to add slot-specific settings:
var azureFunction = await resourceGroup.GetWebSiteAsync("azureFunctionName");
WebSiteSlotResource slot = await azureFunction.GetWebSiteSlotAsync("slotName");
AppServiceConfigurationDictionary applicationSettings = await slot.GetApplicationSettingsSlotAsync();
applicationSettings.Properties["test"] = "test";
await slot.UpdateApplicationSettingsSlotAsync(applicationSettings);

But it only Updates the slot's settings, none of which are 'slot-specific'.
To be clear, I am trying to achieve the same result as with this command:
az functionapp config appsettings set --name <FUNCTION_APP_NAME> \
--resource-group <RESOURCE_GROUP_NAME> --slot <SLOT_NAME> \
--slot-settings AzureWebJobs.QueueTrigger.Disabled=true

Environment

Azure Functions v4, .NET Framework 4.8
Visual Studio 17.6.3

Any update on this?

Same. Any update? thanks

Hi, @Mario-GG , based on the CLI command you used, the corresponding method in SDK shoule be: https://github.com/Azure/azure-sdk-for-net/blob/dd64b8ada6985425bae3fac17681e3f80952f87b/sdk/websites/Azure.ResourceManager.AppService/src/Generated/SlotConfigNamesResource.cs#L173

It's probably a separate issue, but when trying to get the SlotConfigNamesResourceData, I'm getting
"Value cannot be null. Arg_ParamName_Name" on resourceId.

It's probably a separate issue, but when trying to get the SlotConfigNamesResourceData, I'm getting
"Value cannot be null. Arg_ParamName_Name" on resourceId.

Could you help to file another issue so that we could have a track on that.

Could you help to file another issue so that we could have a track on that.

I'll aim to get it done tomorrow.


sample commit to fixup about page

Test to see if about page can be updated.

This also has the x's added to the help about legend for closed greenways.


add doxygen configuration

doxygen for auto generated documentation.
Preview:

To generate need just run doxygen. Let's take a look and compare efforts and results.

Doxygen generates documentation for API and internals, including call graphs.
Sphinx AFAIK generates only API documentation.

Where do the generated docs get saved? Is there any docs anywhere as to how to install the utility?

Generated documentation is saved by default to directory html.
About Doxygen: https://www.doxygen.nl/download.html

If we decide to include this, we should include docs about installing/using it - otherwise this is just an unexplained config file

Yes. it can live in docs. Thank you.

Putting a hold on this until the docs around it I requested are added

What's the latest on this? Are we aiming for Sphinx only? Thanks!

Doxygen has several unique features. Just two of them: graphs, annotated directory tree.

Who is the target user for the documentation and how does any of these features benefit the target user?

Our target audience includes not only users of the ilab CLI but also developers contributing to the project. Developers, in particular, benefit significantly from Doxygen features, as it facilitates comfortable and efficient project navigation.


[Storage] [DataMovement] [Shares] Preserve Permission Key

Adding ability to preserve Permission Key on Share Files when attempting transfers from Share File to Share File
Changes include:

Updating ShareDirectoryStorageResourceContainer upon listing files and directories, getting the permission key from the source and applying it to the destination
Updating ShareFileStorageResource to get permission key from source and set it on the destination when transferring single share files

Other changes include:

Fixing Access Tier tests, since we removed the ability to preserve Access Tier (I may have to double check on this)

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Storage.DataMovement


A method for change Dateformat

Please add some public method to DatePickerDialog for change DateFormat of MONTH_FORMAT, YEAR_FORMAT and DAY_FORMAT objects. SO that we can program number of char in mSelectedMonthTextView variable.    "This is not required for locale=english, but in some other languages we have to set full month name,"

I'm using the built in formatter, which should return a string that makes sense in all supported locales.
(For example in Dutch it returns a string of length 4 rather than 3 when using MMM)
I'm reluctant to add this option, because it's an option I can't test myself. I will have no idea that a layout change I will do will actually make sense or end up in weird truncation errors in different locales / for different string patterns.
People have forked this library to support different locales which had requirements that were hard to elegantly support with the current codebase (either because they were technically hard, or because I lack the cultural knowledge to properly assess them).
I am more than happy to point to any such forks from the README.

I understood. Thanks for the reply.


many input pins lead to many queued calls to process()

The current design calls process() each time a data is pushed. However we should check (with a weak_ptr?) that the data is still valid, otherwise invalidate the call.

The previous commit is a kind of counter-example. Modules should copy the data internally. For ex, a mux could want to process the data differently than the input FIFO.


Add tagline

Is your feature request related to a problem? Please describe.
Taglines are a good way to make the visitor remember the product. It's always better to have some to strengthen the visitor's memory about the service being provided.
Describe the solution you'd like
Add the tagline as shown in the image below just above the footer:

Make sure to improve the look of this section by updating the color and the font.
Compare and create the PR into the feature/frontend branch, and not the master branch. Make sure that the current webpage looks something like the screenshot shown below, and not the one available online since the feature/frontend branch contains the updated version of the website.

Additional context
Time Limit: 2 days
PS: The task will be assigned on an FCFS basis.

Hi, can I do this one please

Sure! Go ahead!

Thanks!!
I have a small doubt tho; where should the contact us button point to?
Or is it really required since there is a contact part directly underneath the tagline?

@amaaniqbal, I have made a PR, please a take a look when free.
I did not actually add the button because keeping a Contact link in a tagline somehow didn't seem right to me; but I will gladly do so if it is needed.
Thanks!!

Oh sorry! I missed it. I should have clarified. A Call-to-action button is expected there. For now, you may put the Sign-Up button over there, later we can update it depending upon the requirement.


[fi] Deterministic parameter generation

This PR extends the existing FI setup to generate FI parameters deterministically. This is helpful for certain gear where a random parameter generation is not meaningfull (e.g., X/Y table for LFI).
Each FI gear needs to implement the deterministic parameter generation by themselves. This PR extends the Husky VCC glitcher to either randomly or deterministically generate the paramters.
Closes #253.

Thanks @vogelpi!


Haskell-ghc-mod: ghc-mod command browse failed with error Error

[Enter steps to reproduce below:]
I tried to install Atom & Ide-Haskell according to the instructions at https://atom.io/packages/ide-haskell. Now whenever I start Atom I get this error.
(Note: I removed most path and environment info because it seemed to contain private data.)
Atom Version: 1.8.0
System: Microsoft Windows 10 Home
Thrown From: haskell-ghc-mod package, v1.14.6
Stack Trace
Haskell-ghc-mod: ghc-mod command browse failed with error Error
At caps: undefined
URI: undefined
Args: -d,-p,--,Lib
message: Command failed: ghc-mod browse -d -p -- Lib
Invalid option `-p'

Usage: ghc-mod browse [-o|--operators] [-d|--detailed] [-q|--qualified] MODULE
  List symbols in a module

log:
-0.317s: "Main created"
-0.317s: "Main updating"
-0.317s: "getProcessOptions(C:\\Users\\uXs\\TestProject)"
-0.316s: "Looking for cabal sandbox..."
-0.315s: "Looking for stack sandbox..."
-0.315s: "Running stack with PATH [snip]
-0.314s: "Running stack path,--snapshot-install-root,--local-install-root with opts = ",{"encoding":"utf-8","stdio":"pipe","cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.311s: "Main buffer is set"
-0.31s: "Lib created"
-0.31s: "Lib updating"
-0.31s: "Prelude created"
-0.31s: "Prelude updating"
-0.31s: "Lib moduleName mismatch: Main != Lib"
-0.31s: "Prelude moduleName mismatch: Main != Prelude"
-0.265s: "Reading cabal sandbox config failed with ",{"errno":-4058,"code":"ENOENT","syscall":"open","path":"C:\\Users\\uXs\\TestProject\\cabal.sandbox.config"}
-0.265s: "No cabal sandbox found"
-0.069s: "Got response from stack path,--snapshot-install-root,--local-install-root",{"stdout":"snapshot-install-root: C:\\Users\\uXs\\AppData\\Roaming\\stack\\snapshots\\09b8ef97\r\nlocal-install-root: C:\\Users\\uXs\\TestProject\\.stack-work\\install\\9a7902b4\r\n","stderr":""}
-0.069s: "Found stack sandbox ","C:\\Users\\uXs\\TestProject\\.stack-work\\install\\9a7902b4\\bin","C:\\Users\\uXs\\AppData\\Roaming\\stack\\snapshots\\09b8ef97\\bin"
-0.069s: "PATH = C:\\Users\\uXs\\TestProject\\.stack-work\\install\\9a7902b4\\bin;C:\\Users\\uXs\\AppData\\Roaming\\stack\\snapshots\\09b8ef97\\bin;[snip]"
-0.068s: "Running ghc-mod version with opts = ",{"timeout":5000,"cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.05s: "Got response from ghc-mod version",{"stdout":"ghc-mod version <IP_ADDRESS> compiled by GHC 7.10.3\r\n","stderr":""}
-0.05s: "Ghc-mod 5,5,0,0 built with 7.10.3"
-0.05s: "{\"version\":[5,5,0,0],\"fileMap\":true,\"quoteArgs\":true,\"optparse\":true,\"typeConstraints\":true,\"browseParents\":true,\"interactiveCaseSplit\":true}"
-0.05s: "Running stack ghc,--,--version with opts = ",{"timeout":5000,"cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.047s: "Running ghc --version with opts = ",{"timeout":5000,"cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.043s: "Running ghc --version failed with ",{"code":"ENOENT","errno":"ENOENT","syscall":"spawn ghc","path":"ghc","spawnargs":["--version"],"cmd":"ghc --version"}
-0.027s: {"code":"ENOENT","errno":"ENOENT","syscall":"spawn ghc","path":"ghc","spawnargs":["--version"],"cmd":"ghc --version"}
-0.021s: "Running ghc-mod browse,-d,-p,--,Lib with opts = ",{"cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.019s: "Running ghc-mod browse,-d,-p,--,Main with opts = ",{"cwd":"C:\\Users\\uXs\\TestProject","env":[snip]
-0.001s: "Invalid option `-p'\r\n\r\nUsage: ghc-mod browse [-o|--operators] [-d|--detailed] [-q|--qualified] MODULE\r\n  List symbols in a module\r\n"
-0.001s: "Running ghc-mod browse,-d,-p,--,Lib failed with ",{"killed":false,"code":1,"signal":null,"cmd":"ghc-mod browse -d -p -- Lib"}
0s: "Error: Command failed: ghc-mod browse -d -p -- Lib\nInvalid option `-p'\r\n\r\nUsage: ghc-mod browse [-o|--operators] [-d|--detailed] [-q|--qualified] MODULE\r\n  List symbols in a module\r\n"

Error
    at file:///C:/Users/uXs/.atom/packages/haskell-ghc-mod/lib/util.coffee:60:26
    at ChildProcess.exithandler (child_process.js:232:5)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:818:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)

Commands

Config
{
  "core": {},
  "haskell-ghc-mod": {
    "experimental": true,
    "highlightTooltips": false
  }
}

Installed Packages
# User
autocomplete-haskell, v0.6.5
haskell-ghc-mod, v1.14.6
ide-haskell, v1.6.5
ide-haskell-cabal, v1.4.1
language-haskell, v1.7.17

# Dev
No dev packages

Any ideas?

Disable experimental option in hgm config. You need to build ghc-mod from 5.6 branch in github repo for that to work.

That works, thanks!

I'm having same issue but with "experimental" disabled, any ideas?

@galderz, please create a new issue and include traces. I'm suspecting it's a different issue with similar symptoms.
To copy formatted traces to clipboard:

then just paste those here.


Support raising ReferenceError from parser; add temporary workaround for invalid left-hand-side expressions

After fixing precommit failure and two notes, the patch would look good to me.

@sand1k, thank you for your notes. Pull request is updated.

Am I see correctly that this is just introducing a new syntax error type? LGTM in this case.

@zherczeg, the patch introduces way to raise ReferenceError from parser. As, sometimes, ReferenceError is an "early error" (for example, 1 = 1-like expressions), it should be thrown before execution begins.
Also, second commit of the patch adds two checks that can raise ReferenceError to handle invalid left-hand-side in an assignment expression.

Ok, land it.


[LinearProgress] Port component

[x] Component
[x] Tests (at least unit tests)
[x] Docs
[x] Demo

FWIW: https://github.com/mbrookes/react-jsstyles-theme-experiment/blob/master/src/components/LinearProgress.js

I guess this can be closed

This component seems to no longer accept the color parameter. Is this on purpose or is it just that the port was done from an older version of the original component?
What I really what is to change the color of the bar once it has reached 100%. I've been trying to hack the component to get this functionality, but the best I came up with was to just return a new LinearProgress component with value 0 and the background color as what I wanted. This adds some weird animations, and I'll probably just skip this.


chainId inconsistencies

Current behaviour:
Just start ganache-cli:

web3.eth.net.getId() returns<PHONE_NUMBER>336
the chainId opcode (EIP1334) returns 1
the eth_chainId RPC call returns 0x539 (1337)

Expected behaviour:

some consistency

Thanks for reporting. Closing as this issue is being tracked by https://github.com/trufflesuite/ganache-core/issues/515, https://github.com/trufflesuite/ganache-core/issues/557, and
https://github.com/trufflesuite/ganache-core/issues/575. The issue will be fixed in v3, tracked by https://github.com/trufflesuite/ganache-core/issues/451, milestone: https://github.com/trufflesuite/ganache-core/milestone/1.


Incorrect count of tests in queue

When run finished , zafira displays statistic about tests in queue

by design for renamed tests. fixed.


How can I use this on an ARM machine?

I tried to run your docker image on an aarch64 Linux Ubuntu 20.04 LTS but get the following error:
The requested image's platform (linux/amd64) does not match the detected host platform (linux/amd64/v8)
How can I use this Docker project on an ARM machine?

You will have to build it the entire thing for arm. This would actually be a good improvement, and would involve the use of matrices in the github actions yaml. Would also probably have to rebuild the wine container aswell.
I'll be happy to take a PR on this, and also help you along the way.

https://github.com/AndreRH/hangover might be of use.

Could be fixed through #28


chore(docs): Update single field syntax

Fixing #57.

Coverage remained the same at 100.0% when pulling cf12069ee4d4a6e4004f31572dfb993cee7cbe9b on docs-update into 97e837e82166e65462ec0942cdad46b20c4ba3b3 on master.


Keyboard Manager crashes when remapping Nintendo Switch Pro Controller buttons

Environment
Windows 10 Enterprise
Version 2004 (Build 19.041.329)

Power Toys version: v0.18.2

Steps to reproduce

Connect Nintendo Switch Pro Controller to PC via Bluetooth
Open Powertoys -> Keyboard Manager
Click on Remap Key
Click "+"-icon
Click a button on the controller
Powertoys crashes + no remap happened

Expected behavior
I expect, that if I'm trying to remap Buttons on a Nintendo Switch Pro Controller, it would give me a error message or it just remaps the buttons.
Actual behavior
Power Toys closes unexpectedly and no remap whatsoever is applied.

hi @joni135, we can't repro this so we're going to need a little more information:

can you view your event logs to see what's causing the crash?
is this happening everytime
does this happen with every button


"# IO::CatHandle::AutoLines can only be mixed into an IO::CatHandle"

This is the error that old and new tests are throwing now with raku 2020.01 and 2020.02. For some reason, it works with 2019.11

Looks like the tests on IO::CatHandle are also broken now??


Dry::Container::Error: There is already an item registered with the key "nominal.string" on jruby-head

Describe the bug
This is probably a jruby + dry-container problem. Decided I'd report it here because it's more likely to be found here by the next person and maybe someone can recommend how to fix it? The repro also needs to be reduced to this gem only before opening that bug. Maybe add jruby-head to CI?
Coming from https://github.com/ruby-grape/grape/runs/4645390152?check_suite_focus=true, dry-types is failing on jruby-head.
/home/runner/.rubies/jruby-head/bin/jruby -I/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/rspec-core-3.10.1/lib:/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/rspec-support-3.10.3/lib /home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/rspec-core-3.10.1/exe/rspec --pattern spec/\*\*/\*_spec.rb  --exclude-pattern spec/integration/\*\*/\*_spec.rb
/home/runner/.rubies/jruby-head/lib/ruby/stdlib/time.rb:3: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::IDENTITY

/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Undefined
An error occurred while loading ./spec/grape/api/custom_validations_spec.rb. - Did you mean?
                    rspec ./spec/grape/exceptions/validation_spec.rb
                    rspec ./spec/grape/dsl/validations_spec.rb
                    rspec ./spec/grape/validations_spec.rb

Failure/Error: require 'dry-types'

LoadError:
  load error: dry/types/core -- java.lang.NullPointerException: null
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:231:in `<main>'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry-types.rb:3:in `<main>'
# ./lib/grape/validations/types/dry_type_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types/array_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types/build_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types.rb:3:in `<main>'
# ./vendor/bundle/jruby/3.0.0/gems/activesupport-7.0.0/lib/active_support/dependencies/autoload.rb:72:in `block in eager_load!'
# ./vendor/bundle/jruby/3.0.0/gems/activesupport-7.0.0/lib/active_support/dependencies/autoload.rb:72:in `eager_load!'
# ./spec/support/eager_load.rb:14:in `block in eager_load!'
# ./spec/support/eager_load.rb:9:in `eager_load!'
# ./spec/spec_helper.rb:26:in `<main>'
# ./spec/grape/api/custom_validations_spec.rb:3:in `<main>'
# ------------------
# --- Caused by: ---
# Java::JavaLang::NullPointerException:
#   org.jruby.dist/org.jruby.ir.operands.Hash.retrieve(Hash.java:132)

An error occurred while loading ./spec/grape/api/deeply_included_options_spec.rb.
Failure/Error: require 'dry-types'

Dry::Container::Error:
  There is already an item registered with the key "nominal.string"
# ./vendor/bundle/jruby/3.0.0/gems/dry-container-0.9.0/lib/dry/container/registry.rb:38:in `block in call'
# ./vendor/bundle/jruby/3.0.0/gems/dry-container-0.9.0/lib/dry/container/registry.rb:36:in `call'
# ./vendor/bundle/jruby/3.0.0/gems/dry-container-0.9.0/lib/dry/container/mixin.rb:105:in `register'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:78:in `register'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:51:in `block in Types'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:49:in `<module:Types>'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:6:in `<module:Dry>'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:5:in `<main>'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:231:in `<main>'
# ./vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry-types.rb:3:in `<main>'
# ./lib/grape/validations/types/dry_type_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types/array_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types/build_coercer.rb:3:in `<main>'
# ./lib/grape/validations/types.rb:3:in `<main>'
# ./vendor/bundle/jruby/3.0.0/gems/activesupport-7.0.0/lib/active_support/dependencies/autoload.rb:72:in `block in eager_load!'
# ./vendor/bundle/jruby/3.0.0/gems/activesupport-7.0.0/lib/active_support/dependencies/autoload.rb:72:in `eager_load!'
# ./spec/support/eager_load.rb:14:in `block in eager_load!'
# ./spec/support/eager_load.rb:9:in `eager_load!'
# ./spec/spec_helper.rb:26:in `<main>'
# ./spec/grape/api/deeply_included_options_spec.rb:3:in `<main>'
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_ARRAY
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_HASH
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_STRING
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_SET
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_OPTS
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Self
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:35: warning: already initialized constant Dry::Types::TYPE_SPEC_REGEX
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:15: warning: already initialized constant Dry::Types::KERNEL_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:20: warning: already initialized constant Dry::Types::METHOD_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:25: warning: already initialized constant Dry::Types::METHOD_COERCIBLE_METHODS
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:37: warning: already initialized constant Dry::Types::NON_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:40: warning: already initialized constant Dry::Types::ALL_PRIMITIVES
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:43: warning: already initialized constant Dry::Types::COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:46: warning: already initialized constant Dry::Types::NON_NIL
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::IDENTITY
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Undefined
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_ARRAY
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_HASH
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_STRING
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_SET
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_OPTS
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Self
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:35: warning: already initialized constant Dry::Types::TYPE_SPEC_REGEX
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:15: warning: already initialized constant Dry::Types::KERNEL_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:20: warning: already initialized constant Dry::Types::METHOD_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:25: warning: already initialized constant Dry::Types::METHOD_COERCIBLE_METHODS
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:37: warning: already initialized constant Dry::Types::NON_COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:40: warning: already initialized constant Dry::Types::ALL_PRIMITIVES
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:43: warning: already initialized constant Dry::Types::COERCIBLE
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types/core.rb:46: warning: already initialized constant Dry::Types::NON_NIL
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::IDENTITY
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Undefined
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_ARRAY
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_HASH
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_STRING
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_SET
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::EMPTY_OPTS
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-core-0.7.1/lib/dry/core/constants.rb:112: warning: already initialized constant Dry::Types::Self
/home/runner/work/grape/grape/vendor/bundle/jruby/3.0.0/gems/dry-types-1.5.1/lib/dry/types.rb:35: warning: already initialized constant Dry::Types::TYPE_SPEC_REGEX

To Reproduce
git clone git@github.com:dblock/grape.git
cd grape
git checkout efc25f2495f03b36c82f64ea3951cbf325df6e3b
rvm use jruby-head
rake

https://github.com/jruby/jruby.git @  5134c73021.
My environment
~/source/opensearch-project$ ruby --version
jruby <IP_ADDRESS>-SNAPSHOT (3.0.2) 2021-12-27 5134c73021 OpenJDK 64-Bit Server VM 14.0.2+12-Ubuntu-120.04 on 14.0.2+12-Ubuntu-120.04 +jit [linux-x86_64]

Some code seems to get loaded twice, dry-types specifically. I don't think dry-container or dry-types has anything to do with it. I can run dry-types against jruby-head though, 1 sec.

ok, it fails with a NPE :) I'll move the issue there. We don't run against head branches on purpose, it takes more time to have a tested build/PR. Issues like this happen very occasionally.


Check for illegal style parameters

This should fix #106.
The sub contained illegal style values, especially font size "0". I now do some sanity check on styles before adding them to our list.
It's pretty bad that UWP just plain crashes when cues contain unexpected values.

Well, that was quick.
I suspected it had to do with the 0s, but I was tired last night and plan to start the digging in the morning.

Yeah, no problem, it was easy to fix.


Optimizing tree shaking

Right now, all styled components are created at module load time, even if preprocessed with the babel plugin.
Would it be possible to instead convert them like this:
actual code:
const MyBox = styled.div`border: thick solid pink`

post-processed by plugin:
const MyBox = props => <styled.Wrap Cmp="div" css={[`border: thick solid pink`]}/>

This simply defines a function without any side effects, and it can be tree shaken if not used. The current approach is a function call with possible side effects and as such can't be tree shaken.
The only difference is that styled components css classes will be created on first use, not when defined.
Maybe even this is possible? It would shave off a tree level:
const MyBox = props => <div {...props} className={styled.getClassNames(props, [`border: thick solid pink`])}/>

After the discussion at https://spectrum.chat/thread/a8359eb6-741b-477c-be38-a343ffeda9f6 it became clear that due to the SC API, there's no way around creating them at module load time. Closing.

What we can do though is utilise Uglify’s “ #PURE” annotation. If we pull that off right in the lib and our Babel plugin, I think we can get dead-code elimination in uglify to help us out

Ah yes indeed! I wonder if the annotation could be done on the library
instead?
On Tue, Oct 17, 2017, 8:19 PM Phil Plückthun<EMAIL_ADDRESS>wrote:

What we can do though is utilise Uglify’s “ #PURE” annotation. If we
pull that off right in the lib and our Babel plugin, I think we can get
dead-code elimination in uglify to help us out
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/styled-components/styled-components/issues/1195#issuecomment-337318569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADWlqm_j9DtqETaYKJZ_uLhRA4PXF5Cks5stO4_gaJpZM4Pp864
.

any news here? Would be great to have tree shaking support!


What is the right link for running the wget?

I have tried the original url and also https://python.langchain.com/en/latest/ as suggested in some issues but it doesn't download properly.

Do https://api.python.langchain.com/en/latest/api_reference.html
It seems the original url https://langchain.readthedocs.io/en/latest/ now redirects to https://api.python.langchain.com/en/latest/api_reference.html

Yeah but those are not the docs, they are the API reference.

I also encountered the same problem, unable to load the Langchain document correctly, and the biggest failure of this project lies here

in ingest.sh change "https://python.langchain.com/en/latest"https://python.langchain.com/en/latest to "https://api.python.langchain.com/en/latest"


iTC now using "no" for Norwegian

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I have read the Contribution Guidelines

Issue Description
We added Norwegian as app store description on iTunes Connect and downloaded the metadata. A folder no was created.
However, in Xcode nn is used for Norwegian. no is an unsupported language code.
So the problem is this:

if you put no in the Snapfile my iOS 10 simulator runs in English, producing Englisch screenshots in the Screenshots/no folder. Thus the wrong language is uploaded.
if you put nn in the Snapfile, the simulator produces Norwegian screenshots, but they end up in an Screenshots/nn folder. Since deliver is looking for the Norwegian screenshots in the wrong folder nothing is being uploaded.

I suggest that for uploading no screenshots you also accept files residing in the nn folder.

Just a quick comment.
We use two language codes in Norway, nb (NO-nb, Bokmål) and nn (NO-nn, Nynorsk).
nb is the preferred written standard.
We use only nb i our projects.

@odrobnik @bjornoss Thanks for sharing this. I reaching out to the core team so we can fix this. Let me know if either of you is interested in submitting a PR. Thanks!

Hey @odrobnik! Thanks for submitting the issue... I tried reproducing this and found that using nb as @bjornoss stated is the best approach. While there a number of Norwegian localization options in Xcode, Norwegian Bokmål is the preferred standard and works alongside deliver with screenshots and metadata being saved into ./fastlane/screenshots/no/ and ./fastlane/metadata/no/ directories respectively.
In the Snapfile that I used to get this issue solved, I used the following languages array:
languages([
  "en-US",
  "no"
])

This is what it looks in Xcode for my project's Localization settings.

Please let us know if you have any more questions! 🚀

@ohwutup When you use languages no, does it pass nb as locale to Simulator?

@odrobnik snapshot doesn't modify the language codes, so it would just pass whatever you provide in your Snapfile.
You can also provide 2 values for one language, to define its exact locale (not sure if that helps):
languages([
  "en-US",
   ["pt", "pt_BR"] # Portuguese with Brazilian locale
])

@KrauseFx i.e. If the language in Snapfile is either nn or nb, then the sim language should be that, but the output folder name needs to be no

@odrobnik I think we would want to keep the output folder matching the code that is in your Snapfile for consistency. While Xcode uses nb, we found through our testing that no works when passed to the simulator and on iTC.

Then there IS still a problem. If I pass no in Snapfile the sim runs in English. If I pass nn, the it runs in nn, but the folder name for delivery is incorrect!
Von meinem iPhone gesendet

Am 26.10.2016 um 23:53 schrieb Felix Krause<EMAIL_ADDRESS>@odrobnik snapshot doesn't modify the language codes, so it would just pass whatever you provide in your Snapfile.
You can also provide 2 values for one language, to define its exact locale (not sure if that helps):
languages([
"en-US",
["pt", "pt_BR"] # Portuguese with Brazilian locale
])
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

We use no in the Snapfile and the simulator runs in Norwegian as expected.
We save our strings in the nb.lproj folder.

@odrobnik we see the same situation as @bjornoss. Is there any chance you could provide us with a sample project that demonstrates that issue?

The problem is not with nb, but with nn. If you pass no to sim, it does not pick up the nb localization. Our workaround will probably have to be that we switch to use nb as well.

@odrobnik Thanks for working with us. If you can provide a sample project we'll be able to action this a lot faster. Is your workaround working for you?

I moved my nn localization to nb. Snapfile has nb. output arrives in no. Works for me.
Your problem is, if nb is the only localization and you put nb in Snapfile output is put into nb.

Thanks for confirming that @odrobnik! Out of curiosity, is there a reason why you were using nn in the first place?

@ohwutup I cannot tell you why I ended up with nn probably just picked one that sounded reasonable to me, as I don't know which is the more predominant language in Norway.

Great to hear this got resolved 👍

@KrauseFx @ohwutup Now something weird happened. I ran snapshot and even though I have nb in the languages, the screenshots did end up in the nb folder. deliver - freshly updated - did not pick up the screenshots in there, but showed a confirmation page with empty no screenshots.
I have to copy screenshots from nb into no for it to work.
snapshot is 1.6.3
deliver is 1.15.1


1.20.4

Could you port this mod to 1.20.4

1.20.2 version works for me on 1.20.4

Prism Launcher won't let me launch Minecraft with the 1.20.2 version of the mod. Please update to 1.20.4

It's in the dev version, release once I get to check all 65 open issues on what can be quickly fixed/added rn.


Add Microsoft User Data

I'm just curious - does Microsoft server really returns user_data property as json object? MicrosoftUserPayload seems JWT payload, looks like it need to do additional work ? (e.g parse access_token or id_token as JWT)

I've added new function, please review it and merge to your dev branch.

does id_token means this?
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow#request-an-id-token-as-well-or-hybrid-flow
I don't think id_token is part of MicrosoftOAuthResponse, it should be in MicrosoftOAuthCode.

No, I'll still use 'code' response type not 'id_token' response type, but the ID Token is returned as part of MSAL.NET's AuthenticationResult when you use openid scope (which included when using MSAL.NET). So it will be the part of MicrosoftOAuthResponse.

Make sure to also use 'profile' scope when you test this so user info (name, user name, etc not empty), which is included when you use MSAL.NET.
You can also include 'email' scope when you want to get an email address as part of id_token payload.