[SPARK-18871][SQL] New test cases for IN/NOT IN subquery

What changes were proposed in this pull request?
This PR extends the existing IN/NOT IN subquery test cases coverage, adds more test cases to the IN subquery test suite.
Based on the discussion, we will create  subquery/in-subquery sub structure under sql/core/src/test/resources/sql-tests/inputs directory.
This is the high level grouping for IN subquery:
subquery/in-subquery/
subquery/in-subquery/simple-in.sql
subquery/in-subquery/in-group-by.sql (in parent side, subquery, and both)
subquery/in-subquery/not-in-group-by.sql
subquery/in-subquery/in-order-by.sql
subquery/in-subquery/in-limit.sql
subquery/in-subquery/in-having.sql
subquery/in-subquery/in-joins.sql
subquery/in-subquery/not-in-joins.sql
subquery/in-subquery/in-set-operations.sql
subquery/in-subquery/in-with-cte.sql
subquery/in-subquery/not-in-with-cte.sql
subquery/in-subquery/in-multiple-columns.sql`
We will deliver it through multiple prs, this is the first pr for the IN subquery, it has
subquery/in-subquery/simple-in.sql
subquery/in-subquery/in-group-by.sql (in parent side, subquery, and both)
We will also provide the result from another DB.
How was this patch tested?
This patch is adding tests.

Can one of the admins verify this patch?

ok to test

cc @hvanhovell @nsyca

ok to test

Test build #70350 has started for PR 16337 at commit ee1e14e.

@gatorsmile I think you need to ask @shaneknapp for jenkins permissions.

Test build #70350 has finished for PR 16337 at commit ee1e14e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70350/
Test PASSed.

@kevinyu98 this looks like pretty solid work. A couple of questions:

This is a huge PR.  This will be hard to review. Is there a change that we can split this up in more digestible pieces.
On a related note. Given the size, it might be easy to miss a correctness issue.
Are the tests part of a test kit?

@hvanhovell: On the correctness, how about running the same set of test cases against a second SQL system and comparing the results? If the two systems yield the same result, we can assume Spark has the correct result.
Besides the correctness, what aspect of review would you be thinking? Areas of test coverage? Two more sets of test cases for EXISTS and scalar subquery will be submitted in the coming week. I want to make sure we will follow the same process.
Note that there are negative test cases (test cases that currently raise Analysis exceptions) we have not included in this PR. We plan to put those in a separate PR with some minor code change to suppress the output of the full LogicalPlan along with the error messages. This is because the infrastructure of SQLQueryTestSuite might not be designed to compare two equivalent LogicalPlans of different ExprIds.
We want to have sufficient test coverage for all the three forms of subquery. This way when we will submit the code to defer the pull up, we can run against these sets of test cases to mitigate any undesired effect from the code change.
I don't understand your last point (are the tests part of a test kit?). Could you please elaborate?

@nsyca @kevinyu98
1. Correctness

how about running the same set of test cases against a second SQL system and comparing the results?
Yeah, that works. I am just trying to make sure that nothing slips through the cracks here.

2. Size
The PR is huge and that makes it relatively hard to review (without missing things). Testing against another DB would help a lot for reviewing purposes (it would effectively m).
3. Test kit
I was impressed by the completeness of the tests. I was just wondering if you have wrote them (kudos), or if they are a part of an existing test frame work.
4. Negative testing
We might have to clean-up (remove expr_ids) from the LogicalPlan a little bit to make negative testing work. An Analysis Exception should contain the failing plan, and we could use code similar to https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala#L30-L74 to get this working.

On 1&2: We will show the output from a comparison of the result sets with a second DB.
On 3: We wrote them. It's labour intensive. However, we plan to work on a better method of adding test coverage through auto generation. Let's collaborate via a separate PR on that topic.
On a side note, in the future there will be much much more test cases in the collections and I am concerned about the length of the run. We shall start thinking about organizing test cases in multiple collections. First-tier are to be run as a sanity check for merging PRs. Subsequent tiers catch up later on a snapshot of multiple PRs.
On 4: Thanks for the suggestion. We will check it out.

Test build #70362 has started for PR 16337 at commit c2ca009.

On 1 & 2 - it might be nice if we can use the spark JDBC connector to run the tests directly on a different system. That would make comparisons trivial.
On 3 -  what is the current run-time of all tests? There is a tagging facility for such tests, but I have never used it in practice.

@hvanhovell Does Spark JDBC connector act purely as a JDBC driver which supposes to send an entire SQL statement to run on the destination system? If there are part of the SQL statement processed in Spark, I'm afraid that is a contamination that we do not want in this verification exercise.

Test build #70362 has finished for PR 16337 at commit c2ca009.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70362/
Test FAILed.

Merged build finished. Test FAILed.

In Spark, we can use the following way to send the whole query to the underlying data source.
CREATE TEMPORARY TABLE jdbcTable
USING jdbc
OPTIONS (
    url "jdbc:postgresql:dbserver",
    dbtable "(SELECT * FROM mytable) tmp"
);

Could we split the test suites based on the subquery types?

With aliases.
With IN or NOT IN

Subqueries with IN
Subqueries with NOT IN.

In UPDATE, DELETE, and INSERT statements.
With comparison operators.
With ANY, SOME, or ALL.
With EXISTS or NOT EXISTS.

Subqueries with EXISTS
Subqueries with NOT EXISTS.

In place of an expression.

Can we create some higher level logical grouping and put these queries into multiple files? It's not as useful to dump 200 queries in one file.

We will separate this file to multiple files.

Let me try to summarize the comments around the structure of the test files here:

A single file of 200+ test cases are too big. We prefer smaller files with logical groupings.
File name with serial number is not the way Spark names files.

I'd like to generate more discussions before we come to a conclusion.

It is possible to group test cases and what we tried to loosely group them is naming them in groups in the test file like TC 01.xx. 01 is effectively the group number. We can easily change to put one group in each file.
Sometimes grouping rigidly is not desirable, or impossible. Does a test case of 'EXISTS .. OR NOT IN' go to the 'EXISTS' group, the 'NOT IN' group, or the 'disjunctive subquery' group? Does a test case of 'EXISTS ( .. ) UNION EXISTS ( .. )' go into the same group as 'EXISTS ( .. UNION .. )', or the first goes to the 'UNION' suite and the latter 'subquery' suite? Shall we have test cases with one classification go to the "simple" set and the ones with more than one way to classify go to the "complex" set? Overtime, people will pile up most of them in the "complex" set and it will be bloated. And we will end up with "complex-1", "complex-2", etc.
Arguably we have a purpose when writing a test case but sometimes it triggers an unrelated problem. If a test case is intended to test a subquery functionality but ends up revealing a missed opportunity in join reordering, should we move it into the 'join reordering' suite and leave it in the 'subquery' suite?
With the current one level flat structure in sql/core/src/test/resources/sql-tests/inputs/, we could possibly end up with thousands of files in the (near) future if a file contains only a handful of test cases. What is a good solution? Should we create a subdirectory named subquery/ and break up the test cases into small files under this directory?

I don't think we have a silver bullet for this kind of problem. Let's brainstorm here. I (or someone else) could moderate the discussion. Eventually we will need to pick one way or the another. And if we need to change it in the future, we pay the price for it.

I actually don't mind having 01, 02, 03, etc, but still some higher level grouping would be useful.

I have tested a few runs on SQLQueryTestSuite to confirm it allows to have sub-directories under sql/core/src/test/resources/sql-tests/[inputs|results] to group test files further. By reading the code, I'm pretty sure it supports multi-level sub-directories. The only requirement is the name of the file needs to be unique globally under inputs/.
With that knowledge, I propose we have this structure under directory sql/core/src/test/resources/sql-tests/[inputs:
subquery/
subquery/in-subquery/
subquery/in-subquery/simple-in.sql
subquery/in-subquery/simple-not-in.sql
subquery/in-subquery/in-group-by.sql (in parent side, subquery, and both)
subquery/in-subquery/not-in-group-by.sql
subquery/in-subquery/in-order-by.sql
subquery/in-subquery/in-limit.sql
subquery/in-subquery/in-having.sql
subquery/in-subquery/in-joins.sql
subquery/in-subquery/not-in-joins.sql
subquery/in-subquery/in-set-operations.sql
subquery/in-subquery/in-with-cte.sql
subquery/in-subquery/not-in-with-cte.sql
subquery/in-subquery/in-multiple-columns.sql
…
subquery/exists-subquery/
subquery/scalar-subquery/
```

Each test file will contain approximately 10-20 test cases. Some of the test cases will be able to classified in multiple ways. In that case, the tester will use his/her best judgement to classify them, or maybe we can entertain the idea of "complex" in the file name (I personally don't like it as those terms are subjective. To someone a SQL with a-few-table joins is complex. Others may see it not.)

We can run a single test file in `.../inputs` or any sub-directory under it with the following command

`build/sbt "~sql/test-only *SQLQueryTestSuite -- -z <filename>.sql"`

The downside is <filename> needs to be the exact name. It does not allow wildcard characters at this point. Perhaps this is something we can enhance in the future.

Lastly, we will break up this in-subquery test cases to multiple PRs.

If the uniqueness is a problem, we can also just update SQLQueryTestSuite to use the path rather than just file name.

We can live with the uniqueness at this point. I'd like to get your feedback (positive, I hope) on the structure of the directories and the naming convention of the test files so we can move forward with the break up of the test cases to submit the PRs. Thanks.

yea those sound great.

Test build #70457 has started for PR 16337 at commit 9c584fb.

Hello All:
I have divided the test case to small groups based the discussion, and this pr will be the first pr for the IN subquery, it covers the simple and group-by cases.
This the run time from running on my local macbook.
$ build/sbt "~sql/test-only *SQLQueryTestSuite -- -z in-group-by.sql"
[info] Run completed in 23 seconds, 876 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
$ build/sbt "~sql/test-only *SQLQueryTestSuite -- -z simple-in.sql"
[info] Run completed in 9 seconds, 986 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

What are the refernce query results we can compare? For example, from DB2 or Hive?

Nat will run against DB2 and provide result.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70457/
Test FAILed.

Merged build finished. Test FAILed.

when I click the link, it shows
Test Result (no failures), not sure why the Test Failed.

Test build #70469 has started for PR 16337 at commit 9c584fb.

Test build #70469 has finished for PR 16337 at commit 9c584fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70469/
Test PASSed.

I will give an update on the result checking against a second database in two installments.
The first one is for the file in-group-by.sql.
Recommended action:
Comment out TC 01.12 and 01.13
General comments
The second database I used to test is DB2 for Linux/Unix/Windows version 11.1. I need to do some minor modifications on the test files to run on DB2.

Data type mappings: String -> Varchar(30), Short -> smallint, Long -> biting
String representation in DB2 uses single quote. All the input string are altered to DB2 format.
The last value of the last row of t1 has invalid date format. Spark suppresses the conversion error and replaces with NULL.
DB2 does not do this implicit conversion hence changing the input value to NULL. (Is the invalid value an intentional act or an accidental
human error?)
DB2 does not support count(distinct(*)) (in TC01.09/query 15) hence changing to count(distinct(  ).

The result checking

All except 2 test cases produce different output.
In TC 01.12 and 13 (query 18 and 19), DB2 reports a syntax error due to the columns T2B in 01.12 and T2C in 01.13 are used in an invalid context.
TC 01.12 seems to be another case of SPARK-18863.
TC 01.13 is a case to debate here.

A simplified script to reproduce:
Seq((1,1,1)).toDF("t1a", "t1b", "t1c").createOrReplaceTempView("t1")
Seq((1,1,1)).toDF("t2a", "t2b", "t2c").createOrReplaceTempView("t2")
Seq((1,1,1)).toDF("t3a", "t3b", "t3c").createOrReplaceTempView("t3")

— TC 01.12
select * from t1 where t1a in
(select min(t2a) from t2 where t2a = t2a and t2c >= 1 group by t2c having t2c in
(select t3c from t3 group by t3c, t3b having t2b > 6 and t3b > t2b ))

== Parsed Logical Plan ==
'Project [*]
+- 'Filter 't1a IN (list#803)
   :  +- 'Filter 't2c IN (list#802)
   :     :  +- 'Filter (('t2b > 6) && ('t3b > 't2b))
   :     :     +- 'Aggregate ['t3c, 't3b], ['t3c]
   :     :        +- 'UnresolvedRelation `t3`
   :     +- 'Aggregate ['t2c], [unresolvedalias('min('t2a), None)]
   :        +- 'Filter (('t2a = 't2a) && ('t2c >= 1))
   :           +- 'UnresolvedRelation `t2`
   +- 'UnresolvedRelation `t1`

== Analyzed Logical Plan ==
t1a: int, t1b: int, t1c: int
Project [t1a#764, t1b#765, t1c#766]
+- Filter predicate-subquery#803 [(t1a#764 = min(t2a)#816)]
   :  +- Project [min(t2a)#816]
   :     +- !Filter predicate-subquery#802 [(t2c#781 = t3c#796) && (t2b#780 > 6) && (t3b#795 > t2b#780)]
   :        :  +- Project [t3c#796, t3b#795]
   :        :     +- Aggregate [t3c#796, t3b#795], [t3c#796, t3b#795]
   :        :        +- SubqueryAlias t3, `t3`
   :        :           +- Project [_1#790 AS t3a#794, _2#791 AS t3b#795, _3#792 AS t3c#796]
   :        :              +- LocalRelation [_1#790, _2#791, _3#792]
   :        +- Aggregate [t2c#781], [min(t2a#779) AS min(t2a)#816, t2c#781]
   :           +- Filter ((t2a#779 = t2a#779) && (t2c#781 >= 1))
   :              +- SubqueryAlias t2, `t2`
   :                 +- Project [_1#775 AS t2a#779, _2#776 AS t2b#780, _3#777 AS t2c#781]
   :                    +- LocalRelation [_1#775, _2#776, _3#777]
   +- SubqueryAlias t1, `t1`
      +- Project [_1#760 AS t1a#764, _2#761 AS t1b#765, _3#762 AS t1c#766]
         +- LocalRelation [_1#760, _2#761, _3#762]

== Optimized Logical Plan ==
Project [_1#760 AS t1a#764, _2#761 AS t1b#765, _3#762 AS t1c#766]
+- Join LeftSemi, (_1#760 = min(t2a)#816)
   :- LocalRelation [_1#760, _2#761, _3#762]
   +- Aggregate [t2c#781], [min(t2a#779) AS min(t2a)#816]
      +- Project [_1#775 AS t2a#779, _3#777 AS t2c#781]
         +- Join LeftSemi, (((_3#777 = t3c#796) && (_2#776 > 6)) && (t3b#795 > _2#776))
            :- Filter (_3#777 >= 1)
            :  +- LocalRelation [_1#775, _2#776, _3#777]
            +- Aggregate [t3c#796, t3b#795], [t3c#796, t3b#795]
               +- LocalRelation [t3b#795, t3c#796]

I don't see the column t2b#780 being part of the output of the (lower) Aggregate operator. Somehow the LeftSemi  join for t2b > 6 is just moved down below the Aggregate over t2. This does not look right to me. I will record this problem in SPARK-18863 for further investigation.
-- TC 01.13
select * from (select * from t2 where t2a in (select t1a from t1 where t1b = t2b)) t2 where t2a in
(select t2a from t2 where t2a = t2a and t2c > 1 group by t2a having t2c > 8)

== Parsed Logical Plan ==
'Project [*]
+- 'Filter 't2a IN (list#824)
   :  +- 'Filter ('t2c > 8)
   :     +- 'Aggregate ['t2a], ['t2a]
   :        +- 'Filter (('t2a = 't2a) && ('t2c > 1))
   :           +- 'UnresolvedRelation `t2`
   +- 'SubqueryAlias t2
      +- 'Project [*]
         +- 'Filter 't2a IN (list#823)
            :  +- 'Project ['t1a]
            :     +- 'Filter ('t1b = 't2b)
            :        +- 'UnresolvedRelation `t1`
            +- 'UnresolvedRelation `t2`

== Analyzed Logical Plan ==
t2a: int, t2b: int, t2c: int
Project [t2a#779, t2b#780, t2c#781]
+- Filter predicate-subquery#824 [(t2a#779 = t2a#779#835) && (t2c#781 > 8)]
   :  +- Project [t2a#779 AS t2a#779#835]
   :     +- Aggregate [t2a#779], [t2a#779]
   :        +- Filter ((t2a#779 = t2a#779) && (t2c#781 > 1))
   :           +- SubqueryAlias t2, `t2`
   :              +- Project [_1#775 AS t2a#779, _2#776 AS t2b#780, _3#777 AS t2c#781]
   :                 +- LocalRelation [_1#775, _2#776, _3#777]
   +- SubqueryAlias t2
      +- Project [t2a#779, t2b#780, t2c#781]
         +- Filter predicate-subquery#823 [(t2a#779 = t1a#764) && (t1b#765 = t2b#780)]
            :  +- Project [t1a#764, t1b#765]
            :     +- SubqueryAlias t1, `t1`
            :        +- Project [_1#760 AS t1a#764, _2#761 AS t1b#765, _3#762 AS t1c#766]
            :           +- LocalRelation [_1#760, _2#761, _3#762]
            +- SubqueryAlias t2, `t2`
               +- Project [_1#775 AS t2a#779, _2#776 AS t2b#780, _3#777 AS t2c#781]
                  +- LocalRelation [_1#775, _2#776, _3#777]

The column t2c in having t2c > 8 should be mapped to the closest scope of t2 but since t2c is not part of the group by column, this is an invalid usage. Spark then moves to find t2c in an outer scope mapping the column to t2c#781 of table t2 in the parent table. I don't know how the scoping rules are defined in Spark but the way it works could cause unintentional effect to users.

Modified test file used to run on DB2
Output of the run result on DB2

Second installment: subquery/in-subquery/simple-in.sql
Recommended action:
Comment out TC 01.08 and 01.10
The result checking

All except 2 test cases produce different output.
In both TC 01.08 and 10 (query 14 and 16), there are NOT IN subqueries
that reference outer columns containing null data, namely t1i in 01.08, and t2b in 01.10.
These two cases reveal a problem related to the nullAware LeftAnti operator which is
rewritten from a NOT IN subquery.

Instead of walking through the original complex test cases, I can demonstrate the problem
with a simpler script:
Seq((1, 2)).toDF("a1", "b1").createOrReplaceTempView("t1")
Seq[(java.lang.Integer, java.lang.Integer)]((1, null)).toDF("a2", "b2").createOrReplaceTempView("t2")

// The expected result is 1 row of (1,2) as shown in the next statement.
sql("select * from t1 where a1 not in (select a2 from t2 where b2 = b1)").show
+---+---+
| a1| b1|
+---+---+
+---+---+

sql("select * from t1 where a1 not in (select a2 from t2 where b2 = 2)").show
+---+---+
| a1| b1|
+---+---+
|  1|  2|
+---+---+

== Analyzed Logical Plan ==
a1: int, b1: int
Project [a1#684, b1#685]
+- Filter NOT predicate-subquery#717 [(a1#684 = a2#695) && (b2#696 = b1#685)]
   :  +- Project [a2#695, b2#696]
   :     +- SubqueryAlias t2, `t2`
   :        +- Project [_1#692 AS a2#695, _2#693 AS b2#696]
   :           +- LocalRelation [_1#692, _2#693]
   +- SubqueryAlias t1, `t1`
      +- Project [_1#681 AS a1#684, _2#682 AS b1#685]
         +- LocalRelation [_1#681, _2#682]

== Optimized Logical Plan ==
Project [_1#681 AS a1#684, _2#682 AS b1#685]
+- Join LeftAnti, ((isnull((_1#681 = a2#695)) || isnull((b2#696 = _2#682))) || ((_1#681 = a2#695) && (b2#696 = _2#682)))
   :- LocalRelation [_1#681, _2#682]
   +- LocalRelation [a2#695, b2#696]

When the comparison between b1 and b2 is unknown, the subquery returns no row hence the result is all the rows from the parent. The problem in the plan above is the expression
isnull((b2#696 = _2#682)) is evaluated to isnull( unknown ) which is true — that results in the entire expression of the LeftAnti is true. By the definition of LeftAnti, the rows are matched and should be filtered. I have opened SPARK-18966 to track this problem.

Modified test file of simple-in.sql used to run on DB2
Output of the run result on DB2

Test build #70528 has started for PR 16337 at commit 1c1900a.

Hello Nat, thanks for the result checking and detail explanation. Based on your assessment, I have comment out 4 test cases for now, pending on the investigation of those open JIRAs.

Test build #70528 has finished for PR 16337 at commit 1c1900a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70528/
Test FAILed.

I just run build/sbt "test-only org.apache.spark.sql.streaming.StreamSuite" on my local machine, also the whole sql suite, it works fine. Can you re-run the test? Thanks

Retest this please.

Test build #70535 has started for PR 16337 at commit 1c1900a.

Test build #70535 has finished for PR 16337 at commit 1c1900a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70535/
Test PASSed.

Test build #70768 has started for PR 16337 at commit 7c129d9.

Test build #70768 has started for PR 16337 at commit 7c129d9.

Test build #70768 has finished for PR 16337 at commit 7c129d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #70768 has finished for PR 16337 at commit 7c129d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70768/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70768/
Test PASSed.

Merged build finished. Test PASSed.

Test build #70782 has started for PR 16337 at commit 895bb35.

Test build #70782 has finished for PR 16337 at commit 895bb35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

I compared the results and confirmed the results are consistent. LGTM

@gatorsmile @nsyca @kevinyu98 can we pull the trigger on this one?

I believe it's ready to do so. Thank you, @gatorsmile, for your relentless effort to review this PR. This PR will serve as a baseline for subsequent PRs we will submit for the remaining test cases on subquery. Hopefully we can get each of them in in a single pass of review.
@hvanhovell, I'd like to bring your attention to a design doc for the first phase of coding I posted in SPARK-18874 for reviewing. Your comments are appreciated so we could have our delivery of the code for the first phase right after we have all the test cases in our collection merged.

Will merge it if nobody objects it in the next 24 hours. : )

Thanks! Merging to master.


Create a Website for the Project

Create a public website for Nellygateway

Can be a very simple website
Explain briefly what Nelly is and how it is helpful
Link to GitHub and Dockerhub

Fixed by Gianluca


Test with MEL spectrogram

Though you provide code in issue #1 , it doesn't work
This block change (along with eps 1e-9)
mel = spec_to_mel_torch(
                spec,
                hps.data.filter_length,
                hps.data.n_mel_channels,
                hps.data.sampling_rate,
                hps.data.mel_fmin,
                hps.data.mel_fmax)

        mel_for_loss = spec_to_mel_torch(
                spec,
                hps.data.filter_length,
                hps.data.n_mel_channels,
                hps.data.sampling_rate,
                hps.data.mel_fmin,
                hps.data.mel_fmax_for_loss)

leads to error
RuntimeError: Given groups=1, weight of size [512, 513, 7], expected input[16, 80, 32] to have 513 channels, but got 80 channels instead

My vctk_bigvgan.json
{
  "train": {
    "log_interval": 200,
    "eval_interval": 1000,
    "seed": 1234,
    "epochs": 20000,
    "learning_rate": 1e-4,
    "betas": [0.8, 0.99],
    "eps": 1e-9,
    "batch_size":16,
    "fp16_run": true,
    "lr_decay": 0.999875,
    "segment_size": 8192,
    "init_lr_ratio": 1,
    "warmup_epochs": 0,
    "c_mel": 45

  },
  "data": {
    "training_files": "./dataset/VCTK-Corpus/preprocessed_npz",
    "validation_files":"./dataset/VCTK-Corpus/preprocessed_npz",
    "text_cleaners":["english_cleaners2"],
    "max_wav_value": 32768.0,
    "sampling_rate": 22050,
    "filter_length": 1024,
    "hop_length": 256,
    "win_length": 1024,
    "n_mel_channels": 80,
    "mel_fmin": 0.0,
    "mel_fmax": 12000,
    "mel_fmax_for_loss": null,
    "add_blank": true,
    "n_speakers": 43,
    "cleaned_text": true,
    "aug_rate": 1.0,
    "top_db": 20
  },
  "model": {
    "p_dropout": 0.1,
    "resblock_kernel_sizes": [3,7,11],
    "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]],
    "upsample_rates": [8,8,2,2],
    "upsample_initial_channel": 512,
    "upsample_kernel_sizes": [16,16,4,4],
    "use_spectral_norm": false

  }
}

IT'd be nice if it was actually possible to test with MEL spectrogram.

    net_g = SynthesizerTrn(

        **hps.data.filter_length // 2 + 1,**
        hps.train.segment_size // hps.data.hop_length,
        **hps.model, rank=rank).cuda(rank)

Change input size for your model


⚠ GitLab Server has degraded performance

In 42f7f23, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 13457 ms

Resolved: GitLab Server performance has improved in e8d463f.


Upload 1.0.7 to github

Hey 👋
Latest npm version of the package is 1.0.7, latest version on github is 1.0.6.
Would be nice to see 1.0.7 on github too.

I'll check into this. Thanks.

Pushed it up. Thanks.


Logging error and, maby mpv problem.

Good day devs.
the following error has occurred. so, we can't get a log.
Note: we are using debian10.
and, we'll send more logs because, we also noticed, when it's been open for a long time, the application crashed.
here is the errors:
Exception inside python-mpv event loop:
Traceback (most recent call last):
File "/opt/TTMediaBot/mpv.py", line 872, in _loop
self._log_handler(ev['level'], ev['prefix'], ev['text'])
File "/opt/TTMediaBot/bot/player/init.py", line 208, in log_handler
logging.log(level, "{}: {}: {}".format(level, component, message))
File "/usr/lib/python3.9/logging/init.py", line 2113, in log
root.log(level, msg, *args, **kwargs)
File "/usr/lib/python3.9/logging/init.py", line 1504, in log
raise TypeError("level must be an integer")
TypeError: level must be an integer
Exception inside python-mpv event loop:
Traceback (most recent call last):
File "/opt/TTMediaBot/mpv.py", line 872, in _loop
self._log_handler(ev['level'], ev['prefix'], ev['text'])
File "/opt/TTMediaBot/bot/player/init.py", line 208, in log_handler
logging.log(level, "{}: {}: {}".format(level, component, message))
File "/usr/lib/python3.9/logging/init.py", line 2113, in log
root.log(level, msg, *args, **kwargs)
File "/usr/lib/python3.9/logging/init.py", line 1504, in log
raise TypeError("level must be an integer")
TypeError: level must be an integer

Hi devs. I think the Issue is, the application uses a lot of system memory in the long run. so it is killed by debian memory management. but, we still have a problem with logs. We still can't get logs because of the error.

Ok, we dont know how it happeneds but, the bot is not responding to commands.
here is the errors.
Exception in thread TeamTalkThread:
ValueError: 1024 is not a valid UserStatusMode
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/opt/TTMediaBot/bot/TeamTalk/thread.py", line 43, in run
self.ttclient.message_queue.put(self.ttclient.get_message(msg.textmessage))
File "/opt/TTMediaBot/bot/TeamTalk/init.py", line 296, in get_message
return Message(re.sub(re_line_endings, '', _str(msg.szMessage)), self.get_user(msg.nFromUserID), self.get_channel(msg.nChannelID), MessageType(msg.nMsgType))
File "/opt/TTMediaBot/bot/TeamTalk/init.py", line 313, in get_user
gender = UserStatusMode(user.nStatusMode)
File "/usr/lib/python3.9/enum.py", line 360, in call
return cls.new(cls, value)
File "/usr/lib/python3.9/enum.py", line 684, in new
raise exc
File "/usr/lib/python3.9/enum.py", line 668, in new
result = cls.missing(value)
File "/usr/lib/python3.9/enum.py", line 808, in missing
possible_member = cls.create_pseudo_member(value)
File "/usr/lib/python3.9/enum.py", line 823, in create_pseudo_member
raise ValueError("%r is not a valid %s" % (value, cls.qualname))
ValueError: 1024 is not a valid UserStatusMode

I think the error was caused by someone changing the status to question mode, or status while streaming a mediafile, and send commands to bots.
Here's the error I get from the terminal screen:
Exception in thread TeamTalkThread:
ValueError: 6144 is not a valid UserStatusMode
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in bootstrap_inner
self.run()
File "D:\AudioPlayer\thetest\bot\TeamTalk\thread.py", line 43, in run
self.ttclient.message_queue.put(self.ttclient.get_message(msg.textmessage))
File "D:\AudioPlayer\thetest\bot\TeamTalk_init.py", line 296, in get_message
return Message(re.sub(re_line_endings, '', str(msg.szMessage)), self.get_user(msg.nFromUserID), self.get_channel(msg.nChannelID), MessageType(msg.nMsgType))
File "D:\AudioPlayer\thetest\bot\TeamTalk_init.py", line 313, in get_user
gender = UserStatusMode(user.nStatusMode)
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\enum.py", line 384, in call
return cls.new(cls, value)
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\enum.py", line 709, in new
raise exc
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\enum.py", line 692, in new
result = cls.missing(value)
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\enum.py", line 837, in missing
possible_member = cls.create_pseudo_member(value)
File "C:\Users\rexya\AppData\Local\Programs\Python\Python39\lib\enum.py", line 852, in create_pseudo_member
raise ValueError("%r is not a valid %s" % (value, cls.qualname))
ValueError: 6144 is not a valid UserStatusMode
I get this error when I Stream a mediafile, and send the s command to bot.

o, I think, you are right.

Thanks. This should be fixed now. Please test the latest commit and report the results.

Thanks. This should be fixed now. Please test the latest commit and report the results.

fix now, thanks alot. :)


Replace aurora_data_api with psycopg2 connections

All SQL queries to the metadb are done via an HTTPS connection using the aurora-data-api package. This imposes the tedious process of converting aurora-data-api client to psycopg2 client via PyTest mocking within local testing to interact with local Postgres containers that emulate the metadb.
One alternative to this is to use a local stack version of the metadb to use the aurora-data-api within local testing queries. However, AWS RDS falls under the local stack Pro tier which requires a monthly paid subscription.
The best solution so far I can think of includes the following changes:
For testing:

Enable metadb HTTP endpoint for e2e testing and use aurora-data-api for queries. The reason why psycopg2 clients can't be used for querying the metadb for e2e testing is that serverless Aurora databases are not assigned public IP addresses and can only be accessed within its associated VPC network.
Use psycopg2 client and local Docker Postgres container for unit and integration tests

For module:

Create a security group rule for the metadb that allows inbound traffic from the security group IDs associated with the module resources that need to perform queries on the metadb (ECS, Lambda)
Create Terraform variable for users to define additional security group ids to associate with metadb

Given the ability to use a proxy docker container to simulate aurora-data-api requests as implemented in PR #20,  aurora-data-api clients will not be replaced with psycopg2 clients.
The idea of integration tests using Local Stack to host resources can still prevail although the idea of managing the metadb's security group network pool within the Terraform module is not required anymore given future implementation of the aurora-data-api proxy container within tests.


2.3.{0,1,2} no longer installs on older npm versions

This is a new issue to continue discussion around this (previous https://github.com/mochajs/mocha/issues/1867 and https://github.com/mochajs/mocha/issues/1868). There were multiple attempts to fix this regression in 2.3.x, but there is still an issue. The currently known issue is the jade upgrade: jade was upgraded in 2.3.0 up past a major breaking version, which dropped npm supported versions.
@boneskull had some thoughts on what to do here: https://github.com/mochajs/mocha/issues/1868#issuecomment-138420357

Proposed fix for patch release: https://github.com/mochajs/mocha/pull/1888

Reverted jade in master via #1888.

Thank you, the current master branch installs fine on the older version of npm. Looking forward to the next mocha release :)

@dougwilson I trust you'll let us know of any similar issues in the future.  I can't guarantee Mocha v3 will keep the contract, but I'd like to ensure v2 will always work with npm pre-v1.3.7.  thanks for taking the time to let us know about the issue
cc @mochajs/mocha


Chained-assignment of structs after property getter skips a $clone()

Steps To Reproduce
https://deck.net/4cc8a491b7c1bf5c040361db565bfefe
Expected Result
false
false

Actual Result
true
false

Can you please fix this one asap or give me a hint on how to fix it myself :S

Setting [Immutable] on the Struct will get it working for now, but there is still an issue that needs to be fixed.
[Immutable]
public struct Vector3
{
    public float x;
    public float y;
    public float z;
}

Won't work. It needs to be mutable. A workaround won't suffice in this case :( Any tips on which files/methods I need to look at in order to give you a PR with a fix?

See code like AssigmentBlock.cs L436 - JS.Funcs.CLONE is used for structs and nullables.


[Youtube] RegexNotFoundError('Unable to extract Initial JS player n function name ...

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ ] I've searched the bugtracker for similar bug reports including closed ones
[ ] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE
ThinkPad-E570:~$ youtube-dl --verbose https://youtu.be/304guCcmt5w
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://youtu.be/304guCcmt5w']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] 304guCcmt5w: Downloading webpage
[youtube] 304guCcmt5w: Downloading player 0abde7de
ERROR: 304guCcmt5w: Traceback (most recent call last):
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1434, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1407, in _extract_n_function
    funcname = self._extract_n_function_name(jscode)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1394, in _extract_n_function_name
    return self._search_regex(
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1434, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1407, in _extract_n_function
    funcname = self._extract_n_function_name(jscode)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1394, in _extract_n_function_name
    return self._search_regex(
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1434, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1407, in _extract_n_function
    funcname = self._extract_n_function_name(jscode)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1394, in _extract_n_function_name
    return self._search_regex(
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1866, in _real_extract
    self._unthrottle_format_urls(video_id, player_url, formats)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1451, in _unthrottle_format_urls
    n_response = self._n_descramble(n_param, player_url, video_id)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1441, in _n_descramble
    raise ExtractorError(traceback.format_exc(), cause=e, video_id=video_id)
youtube_dl.utils.ExtractorError: 304guCcmt5w: Traceback (most recent call last):
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1434, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1407, in _extract_n_function
    funcname = self._extract_n_function_name(jscode)
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/youtube.py", line 1394, in _extract_n_function_name
    return self._search_regex(
  File "/home/luis/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
ThinkPad-E570:~$

Description

WRITE DESCRIPTION HERE
failing commands:
youtube-dl https://youtu.be/304guCcmt5w
youtube-dl -F https://youtu.be/304guCcmt5w
youtube-dl -f18 https://youtu.be/304guCcmt5w
with similar problem

Works fine:
PS C:\> youtube-dl.exe --version
2021.12.17

PS C:\> youtube-dl https://youtu.be/304guCcmt5w
[youtube] 304guCcmt5w: Downloading webpage
[download] Resuming download at byte 404099
[download] Destination: Conversando sobre Ucrania con Manolo Monereo-304guCcmt5w.f137.mp4
[download]   0.3% of 177.94MiB at 80.74KiB/s ETA 37:31

OP didn't reveal that the yt-dl version is a later one taken from the git master. YT changed the player JS again but OP has not updated with the latest yt-dl master.

I dont get it, just comparing the output, we have the exact same version:
[debug] youtube-dl version 2021.12.17

With other languages, a "version" or "tag" is tied to a single commit, is that not the case here?

If you hack your installation around, the version can stay the same.  No idea what happens when a commit is installed with pip.

I've just had the same issue. Just reinstall the module.
sudo -H pip uninstall youtube-dl
sudo -H pip install --upgrade youtube-dl

https://github.com/ytdl-org/youtube-dl/issues/30856

Looks like this is broken again, on master even

i meet the same issue on the newest code on master branch

Fixed again in a0068bd6bec16008bda7a39caecccbf84881c603.


updated README

added scope informations
updated environment variable for pierone

fixes #590

👍

:+1:

:+1:


INVALID_PDU 304 error for int getUnreadSMSLocs(). what problem here? please assist me

int getUnreadSMSLocs(int* buf, int maxItems)
{
String seqStart = "+CMGL: ";
String response = "";
// Issue the command and wait for the response.
byte status = A6command("AT+CMGL=\"REC UNREAD\"", "\xff\r\nOK\r\n", "\r\nOK\r\n", A6_CMD_TIMEOUT, 2, &response);

int seqStartLen = seqStart.length();
int responseLen = response.length();
int index, occurrences = 0;

// Start looking for the +CMGL string.
for (int i = 0; i < (responseLen - seqStartLen); i++) {
    // If we found a response and it's less than occurrences, add it.
    if (response.substring(i, i + seqStartLen) == seqStart && occurrences < maxItems) {
        // Parse the position out of the reply.
        sscanf(response.substring(i, i + 12).c_str(), "+CMGL: %u,%*s", &index);

        buf[occurrences] = index;
        occurrences++;
    }
}
return occurrences;

}

I don't understand this. Please elaborate on the problem with steps to reproduce.

I don't understand this. Please elaborate on the problem with steps to reproduce.

when i exucte getUnreadSMSLocs(int* buf, int maxItems) this function i getting time out error
or
A6command("AT+CMGL="REC UNREAD"", "\xff\r\nOK\r\n", "\r\nOK\r\n", A6_CMD_TIMEOUT, 2, &response);
changed function "AT+CMGL=0" inplace of "AT+CMGL="REC UNREAD""
A6command("AT+CMGL=0", "\xff\r\nOK\r\n", "\r\nOK\r\n", A6_CMD_TIMEOUT, 2, &response);

when i exucte getUnreadSMSLocs(int* buf, int maxItems) this function i getting time out error
or
A6command("AT+CMGL="REC UNREAD"", "\xff\r\nOK\r\n", "\r\nOK\r\n", A6_CMD_TIMEOUT, 2, &response);
changed function "AT+CMGL=0" inplace of "AT+CMGL="REC UNREAD""
A6command("AT+CMGL=0", "\xff\r\nOK\r\n", "\r\nOK\r\n", A6_CMD_TIMEOUT, 2, &response);

Did that work?

Did that work?

no
+CMS ERROR: 304 Invalid PDU mode parameter  comes
and other all function work  properly

no
+CMS ERROR: 304 Invalid PDU mode parameter  comes
and other all function work  properly

So the problem is that you're getting a timeout when trying to read SMS with getUnreadSMSLocs?

So the problem is that you're getting a timeout when trying to read SMS with getUnreadSMSLocs?

i am using hardware serial inplace of software serial and using mega 2560

i am using hardware serial inplace of software serial and using mega 2560

yes if i increase #define A6_CMD_TIMEOUT 2000 then also time out error or +CMS ERROR: 304  comes

yes if i increase #define A6_CMD_TIMEOUT 2000 then also time out error or +CMS ERROR: 304  comes

Hmm, that's odd. Can you try setting SMS to text mode again?
A6command("AT+CMGF=1", "OK", "yy", A6_CMD_TIMEOUT, 2, NULL);

Hmm, that's odd. Can you try setting SMS to text mode again?
A6command("AT+CMGF=1", "OK", "yy", A6_CMD_TIMEOUT, 2, NULL);

this is my cade
callInfo cinfo = checkCallStatus();
if (cinfo.direction == DIR_INCOMING)
{
Serial.println(cinfo.number);
hangUp();
unreadSMSNum =getUnreadSMSLocs(unreadSMSLocs, 30);
Serial.println("unreadSMSNum");
Serial.println(unreadSMSNum);
for (int i = 0; i < unreadSMSNum; i++)
{
Serial.print("New message at index: ");
Serial.println(unreadSMSLocs[i], DEC);
        sms =readSMS(unreadSMSLocs[i]);
        Serial.println(sms.number);
        Serial.println(sms.date);
        Serial.println(sms.message);
    }
}

this is my cade
callInfo cinfo = checkCallStatus();
if (cinfo.direction == DIR_INCOMING)
{
Serial.println(cinfo.number);
hangUp();
unreadSMSNum =getUnreadSMSLocs(unreadSMSLocs, 30);
Serial.println("unreadSMSNum");
Serial.println(unreadSMSNum);
for (int i = 0; i < unreadSMSNum; i++)
{
Serial.print("New message at index: ");
Serial.println(unreadSMSLocs[i], DEC);
        sms =readSMS(unreadSMSLocs[i]);
        Serial.println(sms.number);
        Serial.println(sms.date);
        Serial.println(sms.message);
    }
}

Does SMS work after that?

Does SMS work after that?

now its work
responce like this
Timed out.
Issuing command: ATH
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGF=1
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGL="REC UNREAD"
Reply in 1058 ms:
OK
Reply OK.
unreadSMSNum
0
Issuing command: AT+CLCC
Reply in 1055 ms:
OK

now its work
responce like this
Timed out.
Issuing command: ATH
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGF=1
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGL="REC UNREAD"
Reply in 1058 ms:
OK
Reply OK.
unreadSMSNum
0
Issuing command: AT+CLCC
Reply in 1055 ms:
OK

but every time i got the unreadSMSNum is 0

but every time i got the unreadSMSNum is 0

and not enter in this loop
for (int i = 0; i < unreadSMSNum; i++)
{
Serial.print("New message at index: ");
Serial.println(unreadSMSLocs[i], DEC);
        sms =readSMS(unreadSMSLocs[i]);
        Serial.println(sms.number);
        Serial.println(sms.date);
        Serial.println(sms.message);
    }

and not enter in this loop
for (int i = 0; i < unreadSMSNum; i++)
{
Serial.print("New message at index: ");
Serial.println(unreadSMSLocs[i], DEC);
        sms =readSMS(unreadSMSLocs[i]);
        Serial.println(sms.number);
        Serial.println(sms.date);
        Serial.println(sms.message);
    }

if i send the new message then also reponce like
Reply OK.
Issuing command: AT+CMGF=1
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGL="REC UNREAD"
Reply in 1058 ms:
OK
Reply OK.
unreadSMSNum
0
Issuing command: AT+CLCC
Reply in 1055 ms:
OK

if i send the new message then also reponce like
Reply OK.
Issuing command: AT+CMGF=1
Reply in 1055 ms:
OK
Reply OK.
Issuing command: AT+CMGL="REC UNREAD"
Reply in 1058 ms:
OK
Reply OK.
unreadSMSNum
0
Issuing command: AT+CLCC
Reply in 1055 ms:
OK


"Archive and Clean Up Approval" process

As part of "Archive and Clean Up Approval" process flow it sends email with "Approve" and "Reject" , clicking this link generates "permission error to access CDS database", how to resolve this?

Did you use a security group for permissions when setting up the CoE?
You can check by going to the admin center, and selecting the environment.
Under Details > Security Group, it will either show the name of the security group or "Not Assigned"

I'm getting something similar.
I didn't have the CoE Environment set with a security group, but I've changed that now, but it's not made a difference.
I've checked the security roles applied to my account and it has Approvals User, Common Data Service User and System Administrator.
I get the following error when I attempt to approve a "to be archived" app or flow:
Encountered a general permissions error trying to access the CDS database. This could be caused by modification of the approvals administrator or user roles, or by an incompatible plugin. Detailed message: 'Message: Principal user (Id=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, type=8, roleCount=1, privilegeCount=94, accessMode=4), is missing prvReadAsyncOperation privilege (Id=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) on OTC=4700 for entity 'asyncoperation'. context.Caller=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Code: 0x80040231 InnerError: Type: System.ServiceModel.FaultException1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]] Message: Principal user (Id=dfa23889-d7d6-ea11-a813-000d3acae853, type=8, roleCount=1, privilegeCount=94, accessMode=4), is missing prvReadAsyncOperation privilege (Id=e84acc8f-0c61-43e3-a716-20964a483bdf) on OTC=4700 for entity 'asyncoperation'. context.Caller=dc0161b6-a11e-4722-b36d-ca1686abcf8f StackTrace: at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.Execute(OrganizationRequest request, InvocationContext invocationContext, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode, ExecutionContext executionContext, Dictionary2 optionalParameters) at Microsoft.Crm.Extensibility.OData.CrmODataExecutionContext.Upsert(Entity entity) at Microsoft.Crm.Extensibility.OData.CrmODataExecutionContext.Update(Entity entity, UpdateOption updateOption) at Microsoft.Crm.Extensibility.OData.CrmODataServiceDataProvider.UpdateEdmEntity(CrmODataExecutionContext context, String edmEntityName, String entityKeyValue, EdmEntityObject entityObject) at Microsoft.Crm.Extensibility.OData.EntityController.PatchEntityImplementation(String& entityName, String key, EdmEntityObject entityDelta) at Microsoft.Crm.Extensibility.OData.CrmODataUtilities.<>c__DisplayClass10_0`2.b__0() at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger,

Sorry for not being clear before. The reason I was asking about the Security Role is because we suggest you do not use one.
Flow approvals use CDS entities under the covers, but do not require licenses/permissions to the CDS generally speaking.
Once you add the security role users will need to be authorized to the CDS / the environment as a whole to have access to any part of it, including approvals.
Permissions here work out of the box for most people, as long as they aren't using a Security Role, so I'm afraid you'll need to reach out to product support if you are having approvals permissions issues.

Sorry for not being clear before. The reason I was asking about the Security Role is because we suggest you do not use one.
Flow approvals use CDS entities under the covers, but do not require licenses/permissions to the CDS generally speaking.
Once you add the security role users will need to be authorized to the CDS / the environment as a whole to have access to any part of it, including approvals.
Permissions here work out of the box for most people, as long as they aren't using a Security Role, so I'm afraid you'll need to reach out to product support if you are having approvals permissions issues.

Thanks @JeneferM-MSFT what do you mean by product support? It does not work if I add user to approval role, even I tried by giving all roles and power platform admin access, no luck :)
Could you confirm that it is actually working to CoE kit as I tried it in multiple environments :(

LOL I'll try raising a support request - but it might be interesting given: "Microsoft Support won't help you with issues related to this kit, but they will help with related, underlying platform and feature issues." (From https://docs.microsoft.com/en-us/power-platform/guidance/coe/feedback-support)

Correct, this set of flows continues to work for most CoE users, so the issue is probably not with our toolkit but with the way permissions are set up for you, or with the product itself.
Please let us know if you discover something that can be generalized for the toolkit and we will try to include it in the FAQ
Thanks for using CoE

Until we find a resolution for this permission issue we really can not use it :(

Only this one set of flows should be blocked your permissions issues.
Monitoring, nurture, and the other compliance features do but rely on approvals.
I suggest you check to make sure you don't rely on a security group, as stated above too

I do not have any security group assigned. If approval does not work here then does it not mean that other approval will also not work?

Correct. Your Environment does not seem to be supporting approvals. But this approval flow is only one small part of the CoE solution. Most of the toolkit does not rely on it. The entire Core solution which focuses on Monitoring/Aggregation/DLP/Finding Orphaned Apps is not impacted for example.

In summary, the Approvals Users is a Security Role that's automatically created and not shipped with the CoE Starter Kit. If there's any issues with how it is populated out of the box, please raise a Service Ticket and Support can log this as an issue accordingly.

What will I have to do to this environment to support approval?

Again, sorry, you are hitting an issue either with the product or your tenant configuration, not the CoE toolkit. We cannot support you with that, you'll need to contact product support.

We have a local repro of this now so re-opening. Will let you know what we hear.

Thanks for using CoE and posting this. The product team made some shifts which require we make some changes too.
Can you please see if the attached resolves this for you?
CenterofExcellenceAuditComponents_1_19_managed.zip

@JeneferM-MSFT  do you know what does it mean? Do I need to delete old solution?

Sorry someone had added components to the solution I was unaware of. Try this instead.
No need to delete the old, in fact dont do that. Instead upgrade and chose to overwrite customizations.
CenterofExcellenceAuditComponents_1_20_managed.zip

I've just upgraded to 1.20, but that hasn't changed the permissions issue with existing Approval tasks.
I haven't generated any new approvals to check if new ones are fixed.
Should it have fixed the permissions issue?
I still haven't logged it with support - too busy - will try to get that done this weekend.

Craig, be sure to hit overwrite when you install the upgrade. It is not the default.

Thanks Jenefer.
Just did an upgrade/overwrite - but I still get the same permission error.
Including again, just in case there's a fine detail difference:
Encountered a general permissions error trying to access the CDS database. This could be caused by modification of the approvals administrator or user roles, or by an incompatible plugin. Detailed message: 'Message: Principal user (Id=dfa23889-d7d6-ea11-a813-000d3acae853, type=8, roleCount=1, privilegeCount=94, accessMode=4), is missing prvReadAsyncOperation privilege (Id=e84acc8f-0c61-43e3-a716-20964a483bdf) on OTC=4700 for entity 'asyncoperation'. context.Caller=dc0161b6-a11e-4722-b36d-ca1686abcf8f Code: 0x80040231 InnerError: Type: System.ServiceModel.FaultException1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=31bf3856ad364e35]] Message: Principal user (Id=dfa23889-d7d6-ea11-a813-000d3acae853, type=8, roleCount=1, privilegeCount=94, accessMode=4), is missing prvReadAsyncOperation privilege (Id=e84acc8f-0c61-43e3-a716-20964a483bdf) on OTC=4700 for entity 'asyncoperation'. context.Caller=dc0161b6-a11e-4722-b36d-ca1686abcf8f StackTrace: at Microsoft.Crm.Extensibility.OrganizationSdkServiceInternal.Execute(OrganizationRequest request, InvocationContext invocationContext, CallerOriginToken callerOriginToken, WebServiceType serviceType, Boolean checkAdminMode, ExecutionContext executionContext, Dictionary2 optionalParameters) at Microsoft.Crm.Extensibility.OData.CrmODataExecutionContext.Upsert(Entity entity) at Microsoft.Crm.Extensibility.OData.CrmODataExecutionContext.Update(Entity entity, UpdateOption updateOption) at Microsoft.Crm.Extensibility.OData.CrmODataServiceDataProvider.UpdateEdmEntity(CrmODataExecutionContext context, String edmEntityName, String entityKeyValue, EdmEntityObject entityObject) at Microsoft.Crm.Extensibility.OData.EntityController.PatchEntityImplementation(String& entityName, String key, EdmEntityObject entityDelta) at Microsoft.Crm.Extensibility.OData.CrmODataUtilities.<>c__DisplayClass11_0`2.b__0() at Microsoft.PowerApps.CoreFramework.ActivityLoggerExtensions.Execute[TResult](ILogger logger,

If you have any of the old flows around (you did a save-as for example) this will still repro.
No flows with a trigger based on that entity can be turned on anymore.

FYI Deleted the solution and re-imported it.   Now the Approve/Reject for Archiving apps is working.
Not sure why over-write didn't fix it.
Thanks guys!

Yes, suspect I didn't bother re-copying the flows after updating the solution.


Add Notekit and Komikku

Hi I want to add two new gtk app

Btw I don't know why there are four commit. Because my "news flash" pr already merged, it supposed to be only two commit isn't it? If it will mess with commit history, I'll make new pr. Sorry for the inconvenience, I don't quite understand how git works 😅😅

Wow, how did I not know about NoteKit — that looks really nice!

I don't know why there are four commit. Since my "news flash" pr already merged, it supposed to be only two commit isn't it?

You didn't rebase this PR branch (your fork's master) on top of the master here.
Doesn't matter, github will squash everything with one click :)

You didn't rebase this PR branch (your fork's master) on top of the master here.

Doesn't matter, github will squash everything with one click :)

Ah thanks, I understand it now


Disable the "Warp mouse" setting, if it is not available.

Control.warp_mouse() is not always available, for example Web doesn't allow it, and Wayland doesn't support it yet AFAIK (I haven't tested this). So on these platforms, the setting should be disabled and maybe some kind of warning should be given.

So, I looked in docs and found that:
"Note: warp_mouse() is only supported on Windows, macOS and Linux. It has no effect on Android, iOS and Web." So, effectively you can move mouse only on Linux X11, MacOS and Windows, everywhere else you can't due to better security

I think the Wayland scenario should be reported as an upstream bug here, and we should follow what the documentation claims.

I think the Wayland scenario could be reported as an upstream bug here, other than that, we should follow what the documentation claims.

It's not a bug, it's feature (TM)
I think easiest and best way to do, is to just make big red label: "THIS FEATURE MIGHT NOT WORK ON YOUR SYSTEM"

That's not the case here IMO. The security argument doesn't hold that well and there's already discussion on improving the situation as linked, it's just that not many people are interested in working on it, from what I can see, as the pointer-constraints-v1 already has partial support for this feature, albeit a bit shortsighted.
Honeslty, I like to think of this as security feature (even if it isn't). I kind of enjoy when programs is not being able to manipulate my cursor whenever they feel like it, particularly games. (For example, Elder Ring has bad habit of blocking cursor inside it's own window, preventing me from moving cursor to second monitor)


GraphQL implementation of Airlock

Hi,
I'm building an app where back- and frontend are completely separated. The backend is a GraphQL API built with Laravel and I use Airlock to handle authentication to the frontend app. The frontend is a React application using Apollo as the client.
The flow of authenticating users (logging in) is clear when using a REST API:

Perform a GET request from the frontend to the backend API via /airlock/csrf-cookie. This will set the correct session cookies to be able to log in
Perform a POST request with the necessary credentials from the frontend to the backend API  via /login.
In the subsequent requests to the backend API I'll be authenticated because I'll have the correct session cookies

This all done by using the predefined GET and POST routes that ship with Laravel/Airlock. Since I'd like to use a 100% GraphQL implementation, I wonder how I can achieve the same functionality but with GraphQL endpoints (Query/Mutation). I'm fairly new at GraphQL and I'm using Lighthouse as the GraphQL server.

Hi there,
Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!
Thanks.


fix #2751

After review, the only code that is kind of duplicated is the code between BaseSample._get_categories and MetadataTemplate.categories; which we are merging in a single helper function in this PR + made MetadataTemplate.categories a property.

Thanks @antgonza

Thanks @antgonza


Error loading script in Shopify

When trying to load the minified picoModal-3.0.0.min.js version in Shopify I get the following error
Refused to execute https://vue-dawn.myshopify.com/assets/picoModal-3.0.0.min.js as script because "X-Content-Type-Options: nosniff" was given and its Content-Type is not a script MIME type

This is not an issue with this library, it's misconfigured hosting.
Resolving

Sorry James
Problem solved.
I was  not loading properly the library
thanks
On Sat, Nov 13, 2021 at 11:12 AM James @.***> wrote:

Closed #27 https://github.com/Nycto/PicoModal/issues/27.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Nycto/PicoModal/issues/27#event-5613459098, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIZZ5TUW5PF5FBKGLZJL6DUL22BRANCNFSM5H655HOA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Join me on Vsee video chat. It is free http://vsee.com/u/lpva1m


Notebook controller: should createNotebookCellExecutionTask be only callable from within the NotebookExecuteHandler

Testing #122216
It seems to me that createNotebookCellExecutionTask should only be called when I am being executed from my executeHandler, as such should an object be passed into that handler that allows me to do things like creating cell execution tasks?

It seems to me that createNotebookCellExecutionTask should only be called when I am being executed from my executeHandler,

That's a good observation and generally true but there are cases which led to the current API (not saying it cannot be done better...)

you might connect to a kernel that is already running a cell. in that case the execution task needs to be created with explicit user action
you might have a collaborative kernel which is used from a different client and "run" can happen from anywhere
when running cell B your kernel might decide that dependent cell A must also run ( GH issue notebooks does that with cells that use and define variables)


Tabablz Item Template unable to set Tab title. Just shows as path to item template source

Generates correct amount of tabs and functionality, but unable to control the title setter. Are there any examples of using this to dynamically create tabs with custom settings?

I was able to figure it out. I needed to use the header template. Still much to learn from this awesome library. Below is the basic example I created for future reference.


gpsd:WARN: Overlong packet of 105 chars rejected.

I am getting this error when i start gpsd and the latitude and longitude are not logged.
Any pointers ?

Hmm, I've never seen this error. It seems like your device is giving us very long NMEA packets. Can you post the packets here? If you have netcat installed, nc -l -u <IP_ADDRESS> <port number> could print the packets to the terminal depending on your netcat version (if it doesn't look up the manual with man netcat)

Rationale from gpsd's source code:
/*
 * We've had reports that on the Garmin GPS-10 the device sometimes
 * (1:1000 or so) sends garbage packets that have a valid checksum
 * but are like 2 successive NMEA packets merged together in one
 * with some fields lost.  Usually these are much longer than the
 * legal limit for NMEA, so we can cope by just tossing out overlong
 * packets.  This may be a generic bug of all Garmin chipsets.
 */
if (strlen(sentence) > NMEA_MAX) {
    gpsd_report(LOG_WARN, "Overlong packet of %zd chars rejected.\n",
	            strlen(sentence));
    return ONLINE_SET;
}

Your device is probably sending invalid NMEA packets to your applications. I can see if we can work around it if you post the raw packets.

okay, so when the gpsclient (andriod phone)is not able to lock on a latitude and longitude when its under a roof with no clear sky access it was throwing this error. As soon as I went under the clear sky all worked well.
Thanks!

Good to know. Currently the app forwards NMEA packets from the GPS as-is, I'll probably add some validation (maximum length, delimiters and checksum) in the next version and show a warning if they fail.


Implement sub instruction

Hello!
First of all, very cool project, far from done, but the base looks good! 😄
I've been too lazy to start my own emulator project properly, so I'll just go ahead and start implementing the instruction set here, hope you don't mind. I'll be opening PR's, unless you don't want contributions, I don't mind either way. The plan is to implement the whole set, as time allows.
On to the PR then, just a basic substraction implementation with tests checking whether it produces correct results and sets condition flags correctly. It'll probably need more work in the future...
DISCLAIMER: I'm new to Rust so there might be some anti-patterns appearing in my code. Just something to watch out for.

@glyphpoch Super cool! Welcome!
Totally open to contributions! I started this one summer while university was out, and just haven't had time to really touch it much since. 😔 I'd still love to see it through eventually. The intention was very much for it to be a way to learn more about Rust for myself and hopefully teach others in the process, so I hope you can learn something from what you see here.
I've been wanting to get back to this project, hopefully some time this month after I finish up a bit of work on another side project. Feel free to continue working on it though if you want and I'll review all the changes in the hopefully-near future. 😊


Investigate using p-t-p device for "ptp" plugin

The old p-t-p dialup links were set up so that all interfaces had the same IP on the server. I think we can use the same idea here. This should allow us to use only one IP per container in the case of "ptp" plugin.

@eyakubovich This is somewhat deprecated by #63, or do you see any follow-up work left?

Implemented (to the best it's possible) by #63


fix: Rename executable file name at AWS deploy script

Renamed executable file name at AWS deploy script
Pre-launch Checklist

[x] I read the Contribute page and followed the process outlined there for submitting PRs.
[x] This update contains only one single feature or bug fix and nothing else. (If you are submitting multiple fixes, please make multiple PRs.)
[x] I read and followed the Dart Style Guide and formatted the code with dart format.
[ ] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///), and made sure that the documentation follows the same style as other Serverpod documentation. I checked spelling and grammar.
[ ] I added new tests to check the change I am making.
[ ] All existing and new tests are passing.
[x] Any breaking changes are documented below.

If you need help, consider asking for advice on the discussion board.
I have read the CLA Document and I hereby sign the CLA
Breaking changes
n/a

I have read the CLA Document and I hereby sign the CLA

recheck

recheck

I have read the CLA Document and I hereby sign the CLA

recheck

Does this work?
The reason for executing the .dill is that it is a kernal compiled binary. E.g. we are executing an AOT compiled app rather than running it as a JIT.


Fix bug on ansible variable db_prefix

if ansible node and target node is not the same node, it will have problem.
- include: db/recreateDb.yml
  vars:
    dbName: "{{ db.whisk.auth }}"
    forceRecreation: False

the db.whisk.auth references varaible : db_prefix, above ansible task's db_prefix value is ansible node's  {{ ansible_user_id|lower }}_{{ ansible_hostname|lower }}_  (Note: it will get ansible node's user_id and hostname)
- name: (re)start controller
  docker_container:
    name: controller{{ groups['controllers'].index(inventory_hostname) }}
    env:
       "DB_WHISK_AUTHS": "{{ db.whisk.auth }}"
       ...
       ...

But above ansible task's  db.whisk.auth's db_prefix value is controller node's  {{ ansible_user_id|lower }}_{{ ansible_hostname|lower }}_  (Note: it will get controller node's user_id and hostname)
So if ansible node is not same as controller node, the controller's ENV DB_WHISK_AUTHS will reference a not exist auth db address.
So it is better to write the db_prefix value to hardcode to keep the default running env well.
If you don't like the db_prefix value, can change it what you like.

In principle we can accept this change - it's partly addressed by #2524

It seems has some problem for the travis-ci, https://travis-ci.org/apache/incubator-openwhisk/builds/262507767?utm_source=github_status&utm_medium=notification
But it seems has no relation with this pr.

@ningyougang can you adjust docker-machine to be whisk_dockermachine_ as well?

@markusthoemmes , i have modified to whisk_dockermachine_

Can merge this PR?


Evaluate resolvelib

As a parallel or perhaps complementary track to #29: resolvelib provides an abstract interface for resolving dependencies that are produced by a client-supplied "provider" (presumably something like us querying the PyPI APIs).
This might be too generic of a library for our use case, but it might be useful if we go further into reworking pip-tools to supply a Python API.
PyPI Link: https://pypi.org/project/resolvelib/

Some questions we want answered w/r/t resolvelib:

Does it resolve entire dependency trees using the given provider, or only specific versions of the supplied requirements?
Are there samples of using it against PyPI's APIs?

Not a complete answer, but this looks like it uses PyPI's APIs: https://github.com/sarugaku/resolvelib/blob/19c09128394a4fddfd11c16c204a0f2246e98058/examples/pypi_wheel_provider.py#L114

I pasted the PyPIProvider into a file and had a play round with resolvelib. A few things:

The PyPIProvider is not part of the package, the package itself just contains the core dependency resolution logic. If we want to use resolvelib, we'll have to rip that provider out and maintain it in pip-audit.
It works as you'd expect. For a requirement like flask==2.0.1, you get a result like this:

Result(mapping={'flask': <flask==2.0.1>, 'werkzeug': <werkzeug==2.0.1>, 'jinja2': <jinja2==3.0.1>, 'itsdangerous': <itsdangerous==2.0.1>, 'click': <click==8.0.1>, 'markupsafe': <markupsafe==2.0.1>}, graph=<resolvelib.structs.DirectedGraph object at 0x1064fb310>, criteria={'flask': Criterion((<Requirement('flask==2.0.1')>, via=None)), 'werkzeug': Criterion((<Requirement('Werkzeug>=2.0')>, via=<flask==2.0.1>)), 'jinja2': Criterion((<Requirement('Jinja2>=3.0')>, via=<flask==2.0.1>)), 'itsdangerous': Criterion((<Requirement('itsdangerous>=2.0')>, via=<flask==2.0.1>)), 'click': Criterion((<Requirement('click>=7.1.2')>, via=<flask==2.0.1>)), 'markupsafe': Criterion((<Requirement('MarkupSafe>=2.0')>, via=<jinja2==3.0.1>))})

So as you can see, it also picks up the transitive dependencies like markupsafe which comes from the jinja2 dependency.

One more thing I noticed is that when I try with ansible, I get this error:

Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/resolvelib/env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/tetsuo/Workspace/resolvelib/env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
resolvelib.resolvers.RequirementsConflicted: Requirements conflict: <Requirement('ansible')>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/resolvelib/resolve_test.py", line 256, in <module>
    result = resolver.resolve(requirements)
  File "/Users/tetsuo/Workspace/resolvelib/env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/tetsuo/Workspace/resolvelib/env/lib/python3.9/site-packages/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=<Requirement('ansible')>, parent=None)]

I haven't looked too deeply into this so I'm not sure whether this is about the PyPI provider or the core resolver logic. I'm sure we can figure it out if we decide to use the library, but I figured I'd mention it just to warn that it might not be a "plug and play" deal.

@woodruffw
To answer your questions specifically:

Does it resolve entire dependency trees using the given provider, or only specific versions of the supplied requirements?

It seems to get the entire dependency tree. I think the example I posted above of flask shows it because it includes transitive dependencies by checking flask => jinja2 => markupsafe.

Are there samples of using it against PyPI's APIs?

Yep, as you linked, it's in the examples/ directory. As I mentioned in the comment above, there are some details that we need to figure out if we decide to go down that path.

Awesome, thanks for digging into that. Adapting the PyPIProvider seems like a promising route given the information above.
I'm not sure if it's related, but the source of the problem with ansible might be that the current get_project_from_pypi helper only handles "wheel" distributions, not source distributions. It's possible that ansible itself or one of its deps is a source-dist only distribution, causing the resolution to fail.

Yeah, looking at the simple index for ansible, they've never done a non-RC wheel release: https://pypi.org/simple/ansible/
...so the current provider ignores all of the other versions. Our task here would be to support source distributions as well, which (I think) involves just two steps:

Updating the PyPIProvider to not filter out sdists
Adding an equivalent to get_metadata_for_wheel for sdists

(We shouldn't need to parse the metadata ourselves, but here's the relevant most recent PEP anyways: https://www.python.org/dev/peps/pep-0566/)

I have a PR open with just the basics + unit tests. If we decide to merge it, I intend to open issues for:

Supporting extras
Supporting source distributions


Update README.md

Added flat config example
Added license array example
Which issue does this PR address?
Clarity around options available

Thanks!


Default operator selection behavior

If no packages listed, download the entire catalog at heads
If packages are listed without version or channel, download only the listed packages at channel heads
If packages are listed with versions but no channels specified, download that package version and required versions of each channel to channel head, including head.
If packages are listed with channels, download those channel head versions
If packages are listed with channels and versions, download the requested version and all versions to channel head, including head

@afflom reminder that if this is not closed by TP, we need to update the docs to reflect current functionality

@jpower432 this is obsolete. I recommend closing.