[MySQL] Retaining a primary key in a scramble

When creating a scramble, VerdictDB does not keep the primary key information of the original tables. This makes the MySQL engine chooses an inferior join plan when a query involves joins.
Please refer to the following page for the syntax:
https://dev.mysql.com/doc/refman/5.6/en/create-table-select.html
Please add a new method getPrimaryKey() in DbmsConnection.java; and use that method for retrieving the primary key information in the scrambling plan.

I have the same issue when i running queries with join, using MySQL in verdictdb-tutorial.

the issue is solved ?

No, the code may need to be cleaned and tested more.

Ok, I really wanted to test the join with VerdictDB today.

Then, let me quickly test git fetch origin pull/314/head:joezhong-fix-312 without merging it into master yet.

Thank you very much

According to my tests, the code is not optimized for mysql, so it was even slower. I will request a fix and test again after the fix.

I saw that there is a new code, but it has not been tested yet.

A question, does this error also happen when we use VerdictDB in PostgreSQL?

I can affect PostgreSQL as well.
However, there is no problem for SQL-on-Hadoop databases since they don't have the primary key.

I have confirmed that the code in this pull request reduces the latency for MySQL join queries. For now, the code is not merged until all tests pass. The test errors seem to be due to CircleCI's issues.

Ok, good :) so the code still can not be used today ?

@solangepaz You can still use the code as follows:
git pull
git checkout joezhong-fix-312
mvn -DskipTests -DtestPhase=false -DdeployPhase=false clean install

And change VerdictDB's version to "0.5.7-SNAPSHOT" in your pom file.

Does this is valid in the verdictdb tutorial?

If you look at the tutorial directory, there is pom.xml, which contains verdictdb as one of its dependencies.

so I change this line to 0.5.7 ?

I did what you said, but I still get worse results with VerdictDB.

Please "mvn package" again and re-create scrambles if you haven't.
This is the latencies I see for TPCH 1G.
>>> verdict.sql('show tables')
10 row(s) in the result (0.016 seconds)
        table
0    customer
1    lineitem
2  lineitem_x
3      nation
4      orders
5    orders_x
6        part
7    partsupp
8      region
9    supplier
>>>
>>>
>>> verdict.sql("""
... bypass
... SELECT SUM(l_extendedprice * (1 - l_discount))
... FROM customer, orders, lineitem
... WHERE c_mktsegment = 'BUILDING'
... AND c_custkey = o_custkey
... AND l_orderkey = o_orderkey
... """)
1 row(s) in the result (1.0 mins 24.035 seconds)
  SUM(l_extendedprice * (1 - l_discount))
0<PHONE_NUMBER>2.3543
>>>
>>>
>>> verdict.sql("""
... SELECT SUM(l_extendedprice * (1 - l_discount))
... FROM customer, orders, lineitem
... WHERE c_mktsegment = 'BUILDING'
... AND c_custkey = o_custkey
... AND l_orderkey = o_orderkey
... """)
12:05:26.433 [Thread-2] INFO  o.v.sqlreader.ScrambleTableReplacer - Automatic table replacement: tpch1gtest.orders -> tpch1gtest.orders_x
12:05:26.434 [Thread-2] INFO  o.v.sqlreader.ScrambleTableReplacer - Automatic table replacement: tpch1gtest.lineitem -> tpch1gtest.lineitem_x
1 row(s) in the result (35.784 seconds)
                  s4
0<PHONE_NUMBER>8.78130
>>>

Is the query blocked on the step of creating scramble tables? I have encountered this problem before. Enlarging MySQL buffer pool size would solve this problem.

The query creates the scramble tables, but does not show a result after

The following should be used in your pom:
    <dependency>
            <groupId>org.verdictdb</groupId>
            <artifactId>verdictdb-core</artifactId>
            <version>0.5.7-SNAPSHOT</version>
    </dependency>

The version of verdictdb-tutorial does not matter.

I have this error when i used that in my pom:
ubuntu@verdictdb2:~/verdictdb-tutorial/verdictdb_on_mysql$ ./run.sh localhost 3306 tpch1g create
java.sql.SQLException: No suitable driver found for jdbc:verdict:mysql://localhost:3306/tpch1g?autoReconnect=true&useSSL=false
        at java.sql.DriverManager.getConnection(DriverManager.java:689)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at VerdictTutorialMain.main(VerdictTutorialMain.java:53)
Exception in thread "main" java.lang.NullPointerException
        at VerdictTutorialMain.createScrambleTable(VerdictTutorialMain.java:108)
        at VerdictTutorialMain.main(VerdictTutorialMain.java:70)

I am working on the issue. The fix is there, but I need to merge branches somehow.

Meanwhile, you can just use full table names like "tpch1g.lineitem" in queries.

I've tried this befoI had tried this before and the error is the same, but if I do not use version 0.5.7-SNAPSHOT this error disappears when creating the scramble table.

Sorry for the inconvenience, but doing the following should resolve the issue. FYI, in the tutorial, I have added "run2" for the join query.
In "verdictdb" directory:
git pull
git checkout yongjoo
mvn -DskipTests -DtestPhase=false -DdeployPhase=false clean install

In "verdictdb-tutorial" directory
git pull

Change the version on line 48 to "0.5.7-SNAPSHOT" (as you did before). Then,
mvn package
./run.sh localhost 3306 tpch1g create
./run.sh localhost 3306 tpch1g run
./run.sh localhost 3306 tpch1g run2

Below is what I got
pyongjoo@umich:verdictdb_on_mysql $ ./run.sh localhost 3306 tpch1gtest create
Creating a scrambled table for lineitem...
14:54:36.515 [main] INFO  o.v.c.ScramblingCoordinator - Starts to create a new UNIFORM scramble tpch1gtest.lineitem_scramble from tpch1gtest.lineitem
14:54:36.516 [main] INFO  o.v.c.ScramblingCoordinator - Method: UNIFORM
14:54:36.517 [main] INFO  o.v.c.ScramblingCoordinator - Relative size: 1.000000 (or equivalently, 100.0000 %)
14:57:24.241 [main] INFO  o.v.c.ScramblingCoordinator - Finished creating tpch1gtest.lineitem_scramble
Scrambled table for lineitem has been created.
Time Taken = 168 s
Creating a scrambled table for orders...
14:57:24.330 [main] INFO  o.v.c.ScramblingCoordinator - Starts to create a new UNIFORM scramble tpch1gtest.orders_scramble from tpch1gtest.orders
14:57:24.331 [main] INFO  o.v.c.ScramblingCoordinator - Method: UNIFORM
14:57:24.331 [main] INFO  o.v.c.ScramblingCoordinator - Relative size: 1.000000 (or equivalently, 100.0000 %)
14:57:54.467 [main] INFO  o.v.c.ScramblingCoordinator - Finished creating tpch1gtest.orders_scramble
Scrambled table for orders has been created.
Time Taken = 30 s
pyongjoo@umich:verdictdb_on_mysql $
pyongjoo@umich:verdictdb_on_mysql $
pyongjoo@umich:verdictdb_on_mysql $ ./run.sh localhost 3306 tpch1gtest run
Without VerdictDB: average(l_extendedprice) = 38255.138485
Time Taken = 62 s
15:01:16.167 [main] INFO  o.v.sqlreader.ScrambleTableReplacer - Automatic table replacement: tpch1gtest.lineitem -> tpch1gtest.lineitem_scramble
With VerdictDB: average(l_extendedprice) = 38228.40949764581
Time Taken = 19 s
pyongjoo@umich:verdictdb_on_mysql $
pyongjoo@umich:verdictdb_on_mysql $
pyongjoo@umich:verdictdb_on_mysql $ ./run.sh localhost 3306 tpch1gtest run2
Without VerdictDB: SUM(l_extendedprice * (1 - l_discount)) = 4.41412435523543E10
Time Taken = 85 s
15:03:11.029 [main] INFO  o.v.sqlreader.ScrambleTableReplacer - Automatic table replacement: tpch1gtest.orders -> tpch1gtest.orders_scramble
15:03:11.030 [main] INFO  o.v.sqlreader.ScrambleTableReplacer - Automatic table replacement: tpch1gtest.lineitem -> tpch1gtest.lineitem_scramble
With VerdictDB: SUM(l_extendedprice * (1 - l_discount)) = 4.41249582491536E10
Time Taken = 33 s

Now i have this error, when i run ./run.sh localhost 3306 tpch1g create:
Creating a scrambled table for lineitem... 20:50:57.727 [main] INFO  o.v.c.ScramblingCoordinator - Starts to create a new UNIFORM scramble tpch1g.lineitem_scramble from tpch1g.lineitem 20:50:57.728 [main] INFO  o.v.c.ScramblingCoordinator - Method: UNIFORM 20:50:57.729 [main] INFO  o.v.c.ScramblingCoordinator - Relative size: 1.000000 (or equivalently, 100.0000 %) org.verdictdb.exception.VerdictDBDbmsException: Issued the following query: create table tpch1g.lineitem_scramble (PRIMARY KEY (L_ORDERKEY, L_PARTKEY, verdictdbblock)) partition by list columns (verdictdbblock) (partition p0 values in (0), partition p1 values in (1), partition p2 values in (2), partition p3 values in (3), partition p4 values in (4), partition p5 values in (5), partition p6 values in (6)) select t.L_ORDERKEY, t.L_PARTKEY, t.L_SUPPKEY, t.L_LINENUMBER, t.L_QUANTITY, t.L_EXTENDEDPRICE, t.L_DISCOUNT, t.L_TAX, t.L_RETURNFLAG, t.L_LINESTATUS, t.L_SHIPDATE, t.L_COMMITDATE, t.L_RECEIPTDATE, t.L_SHIPINSTRUCT, t.L_SHIPMODE, t.L_COMMENT, 0 as verdictdbtier, cast(floor(rand() * 7) as unsigned) as verdictdbblockfromtpch1g.lineitemas t Duplicate entry '2080546-112379-0' for key 'PRIMARY'         at org.verdictdb.connection.JdbcConnection.executeSingle(JdbcConnection.java:210)         at org.verdictdb.connection.JdbcConnection.execute(JdbcConnection.java:170)         at org.verdictdb.connection.CachedDbmsConnection.execute(CachedDbmsConnection.java:49)         at org.verdictdb.core.execplan.ExecutableNodeRunner.execute(ExecutableNodeRunner.java:424)         at org.verdictdb.core.execplan.ExecutableNodeRunner.executeAndBroadcast(ExecutableNodeRunner.java:407)         at org.verdictdb.core.execplan.ExecutableNodeRunner.run(ExecutableNodeRunner.java:335)         at java.lang.Thread.run(Thread.java:748) Exception in thread "main" java.lang.RuntimeException: org.verdictdb.exception.VerdictDBDbmsException: Issued the following query: create tabletpch1g.lineitem_scramble (PRIMARY KEY (L_ORDERKEY, L_PARTKEY, verdictdbblock)) partition by list columns (verdictdbblock) (partition p0 values in (0), partition p1 values in (1), partition p2 values in (2), partition p3 values in (3), partition p4 values in (4), partition p5 values in (5), partition p6 values in (6)) select t.L_ORDERKEY, t.L_PARTKEY, t.L_SUPPKEY, t.L_LINENUMBER, t.L_QUANTITY, t.L_EXTENDEDPRICE, t.L_DISCOUNT, t.L_TAX, t.L_RETURNFLAG, t.L_LINESTATUS, t.L_SHIPDATE, t.L_COMMITDATE, t.L_RECEIPTDATE, t.L_SHIPINSTRUCT, t.L_SHIPMODE, t.L_COMMENT, 0 as verdictdbtier, cast(floor(rand() * 7) as unsigned) as verdictdbblockfromtpch1g.lineitemas t Duplicate entry '2080546-112379-0' for key 'PRIMARY'         at org.verdictdb.core.resulthandler.ExecutionTokenReader.takeOne(ExecutionTokenReader.java:53)         at org.verdictdb.core.resulthandler.ExecutionTokenReader.next(ExecutionTokenReader.java:83)         at org.verdictdb.core.execplan.ExecutablePlanRunner.runTillEnd(ExecutablePlanRunner.java:61)         at org.verdictdb.coordinator.ScramblingCoordinator.scramble(ScramblingCoordinator.java:310)         at org.verdictdb.coordinator.ScramblingCoordinator.scramble(ScramblingCoordinator.java:193)         at org.verdictdb.coordinator.ExecutionContext.sql(ExecutionContext.java:171)         at org.verdictdb.jdbc41.VerdictStatement.execute(VerdictStatement.java:109)         at VerdictTutorialMain.createScrambleTable(VerdictTutorialMain.java:179)         at VerdictTutorialMain.main(VerdictTutorialMain.java:70) Caused by: org.verdictdb.exception.VerdictDBDbmsException: Issued the following query: create tabletpch1g.lineitem_scramble (PRIMARY KEY (L_ORDERKEY, L_PARTKEY, verdictdbblock)) partition by list columns (verdictdbblock) (partition p0 values in (0), partition p1 values in (1), partition p2 values in (2), partition p3 values in (3), partition p4 values in (4), partition p5 values in (5), partition p6 values in (6)) select t.L_ORDERKEY, t.L_PARTKEY, t.L_SUPPKEY, t.L_LINENUMBER, t.L_QUANTITY, t.L_EXTENDEDPRICE, t.L_DISCOUNT, t.L_TAX, t.L_RETURNFLAG, t.L_LINESTATUS, t.L_SHIPDATE, t.L_COMMITDATE, t.L_RECEIPTDATE, t.L_SHIPINSTRUCT, t.L_SHIPMODE, t.L_COMMENT, 0 as verdictdbtier, cast(floor(rand() * 7) as unsigned) as verdictdbblockfromtpch1g.lineitemas t Duplicate entry '2080546-112379-0' for key 'PRIMARY'         at org.verdictdb.connection.JdbcConnection.executeSingle(JdbcConnection.java:210)         at org.verdictdb.connection.JdbcConnection.execute(JdbcConnection.java:170)         at org.verdictdb.connection.CachedDbmsConnection.execute(CachedDbmsConnection.java:49)         at org.verdictdb.core.execplan.ExecutableNodeRunner.execute(ExecutableNodeRunner.java:424)         at org.verdictdb.core.execplan.ExecutableNodeRunner.executeAndBroadcast(ExecutableNodeRunner.java:407)         at org.verdictdb.core.execplan.ExecutableNodeRunner.run(ExecutableNodeRunner.java:335)         at java.lang.Thread.run(Thread.java:748)
Not is more simple run the verdictDB in python ? If yes,  what changes do I have to make to use 0.5.7-SNAPSHOT ?

Hmm. I think either your lineitem table had declared (L_ORDERKEY, L_PARTKEY) as a primary key or VerdictDB's primary key retrieval is not correctly working for your mysql version... I think the latter is the case.
Can you tell me your MySQL version?
@Beastjoe Please find a docker image that matches the version of @solangepaz, and check the correctness. Btw, if you added a unit test, I think it should be checked with MySQL 5.5 by CircleCI.

@solangepaz I just uploaded a latest version of pyverdict to pypi, which you can install as follows:
pip install pyverdict --upgrade

Please see https://verdictdb.org/documentation/quick_start/ for some examples. (I was working on documentation, but haven't finished it yet.) The only thing is that it requires Python 3.7 (but installing it is easy with miniconda).

I have MySQL in version 5.7.24


submission of johey_5

@JorisHeyse Can you check why the "50-50 broadband splitter" for your "MZI_DC" device is on layer 31/0?

indeed, it would be preferable to use the updated EBeam library, in which this component is on the correct layer.  We can remap the layers during fabrication, but it does add a risk in case it isn't done automatically.

Thank you for pointing that out, I hadn't noticed it. I am using the SiEPIC PDK for ipkiss. I did some searching and found that the reference gds files of the DC cell also has the device on the 31/0 layer which is probably why it also get exported to that layer. I can fix this by using the gds file of the updated EBeam library.
However, I see my design already got merged in the main repository. Do I need to submit a new design a create a new pull request or is it fine for now?


AzureRmWebAppDeploymentV4: No package found with specified pattern: *.deploy.cmd

Type: Question
Enter Task Name: AzureRmWebAppDeploymentV4
Environment

Server: Azure Pipelines
Agent: Hosted

Issue Description
I'm using this yaml to deploy an ASPNET Core WebAPI project to an Azure App Service.
trigger:
- feature/new-template

pool:
  vmImage: windows-latest

variables:
  buildConfiguration: 'Release'

steps:
- task: DotNetCoreCLI@2
  inputs:
    command: 'restore'
    projects: '**/My.Web.API.csproj'
    feedsToUse: 'config'
    nugetConfigPath: 'nuget.config'

- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
    projects: '**/My.Web.API.csproj'
    arguments: '--configuration $(buildConfiguration)'

- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: PublishBuildArtifacts@1
  condition: succeededOrFailed()
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

- task: AzureRmWebAppDeployment@4
  inputs:
    ConnectionType: 'PublishProfile'
    PublishProfilePath: '$(System.DefaultWorkingDirectory)/**/my-app-service-name - Web Deploy.pubxml'
    appType: apiApp
    DeploymentType: webDeploy
    WebAppName: my-app-service-name
    packageForLinux: '$(Build.ArtifactStagingDirectory)'
    PublishProfilePassword: 'S1Yqfg...'

And I get this error:
##[error]Error: No package found with specified pattern: D:\a\1\*.deploy.cmd<br/>Check if the package mentioned in the task is published as an artifact in the build or a previous stage and downloaded in the current job.

Did you find a solution for this? I'm having the same problem.

A similar issue happened to me, too.
Here's the setup:


wget the dataset in Travis CI

Sorry...  I should have put work in progress on this one...  It does not work yet.

It's ok. Just pull again when you finished. Really appreciate your contribution!

Sorry, I don't know referer_url. Maybe I will share the links of Baidu drive file in another way. I will search this.
I only run on python 2 now.

Today there are 600 days left until Python 2 end of life:
from datetime import date
print((date(2020, 1, 1) - http://date.today()).days, 'days until Python 2 end of life.')


[Feature Request] Custom name FFmpeg.dll

When specifying a file FFmpeg.dll overview ScreenToGif only sees files with the name - FFmpeg.dll . This is not right. I don't need to download FFmpeg for every program that needs it, I have it lying quietly on the disk under the name FFmpeg64.dll and it is with this name that it works in several programs. The ScreenToGif overview should see all the dll's, not just those with a specific name. A trifle, of course, but it is desirable to fix it.

ScreenToGif uses an ffmpeg executable, not a dll.
And you should be able to select a custom file, with a different name with no problem.

Exactly. And I tried to slip a dll and thought that it didn’t work because of the name. I'm sorry.


build: fix schematics TS errors

Fixes some TS errors being logged out when building the schematics due to an outdated path in the exclude and an incorrect mapping in the paths.

@crisbeto I've found the issue why Gulp didn't report the compilation failures. See #13312. Also everything worked even with errors because the files that failed were just test-cases that weren't referenced anywhere.


modal don't dismiss

when press hardware back on android, trigger an event to hide modal, first time it's ok, then reopen the modal, then hide modal  fail!

have you resolved it??

@xinwanggit  搞定了，这是react-native的问题，我自己写了个基于绝对定位的弹出层也是一样的问题。最后我自己写了个动画，通过控制view的高度完美解决

always set the modalOpen state to false


Release notes for v2.1.8 and v19.1.3

Closes #4966
Closes #4967

This change is

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/6c0d2e9590dbe8a9dbbf135e2283639f5a7e86d2/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/44670b4982598b070a991c39f0df73d8fb150f35/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/add48962736b99363c468b8875633d4a0b78c086/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/b9830abcdd251cb3ec1ee4c7e63c46e921499e56/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/de770d36199b0b63b0ab5588addc0448c92e6e04/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/3716ee9212f7b2ac205f5f5e6271226dfe0bc780/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/8aec51790bed1f64a65138c2fef59971651a1f31/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

releases/v19.1.3.md, line 63 at r1 (raw file):
Previously, jseldess (Jesse Seldess) wrote…
One more:
- Expanded the recommended [Topology Patterns](../v19.1/topology-patterns.html) for running CockroachDB in a cloud environment, each with required configurations and latency and resiliency characteristics. [#4820](https://github.com/cockroachdb/docs/pull/4820)

Action, we should add a few more:
- Made the Java code samples in [Build a Java App with CockroachDB](../v19.1/build-a-java-app-with-cockroachdb.html) simpler and more idiomatic. [#4855](https://github.com/cockroachdb/docs/pull/4855)
- Documented [what happens when a node runs out of disk space](../v19.1/operational-faqs.html#what-happens-when-a-node-runs-out-of-disk-space) and how to [create a ballast file](../v19.1/debug-ballast.html) to prepare for this case. [#5000](https://github.com/cockroachdb/docs/pull/5000)

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/5c16da5a23641f0acb286be3cd2590a4021df0ce/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/18f29fbe6fe2e99346a1318e957108e721a51c66/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/c47906f24d426d33bbdeefe337cdefb0a9e05035/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/ed37548f6b5f224e0942cffaf5b786af7f98309f/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/988f1666922db3565db07138f44183837ba45c9e/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/988f1666922db3565db07138f44183837ba45c9e/
Edited pages:

releases/v19.1.3.md
releases/v2.1.8.md

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/988f1666922db3565db07138f44183837ba45c9e/


Consider exposing languageEncodeBytesArray

Is your feature request related to a problem? Please describe.
I have a custom account creation script (I am not using examples.CreateAccount), but I need to encode my account public key in the right format in my script. To do so I needed to essentially re-implement a function you already have in flow-go-sdk: languageEncodeBytesArray.
Describe the solution you'd like
I figured it might be convenient for users like me if you exposed the languageEncodeBytesArray fucntion.
Describe alternatives you've considered
I just re-implemented the function myself for now, which honestly is fine.
Additional context
I have none, I'm afraid.

Closing this because languageEncodeBytesArray is not longer in the templates package. I recommend using transaction arguments to pass byte arrays into transactions.


Vim in iTerm2

Hey @altercation,
Thanks for a great color scheme. It's pure genius. I have it set up in MacVim and am loving it.
I'm having difficulty setting it up for iTerm2/terminal vim. I downloaded and installed the latest nightly of iTerm2 and successfully imported the color scheme. The terminal window is now nice and blue. However, when I run vim in that terminal there is no syntax highlighting and the background color turns brownish.

I'm using the latest solarized code from the repo.
What am I doing wrong?
Thanks!

I see the same thing, gray dircolors with bright colors on.


implemented Account details Page and status with Ago.

implemented Account details Page and status with Ago.

Updating the requirements of the Account page -
Page view 1 -

Page view 2 -

Some early comments on the page being developed -

We do not need page view 1, it is redundant. Since we will anyway be showing the transactions in end of page 2.

- We need to rename sender to address amount the headers.
- The following values can be used for the Account Details.

Header
Value
Comments

Account
accountDetails.address
----

User Ops Executed
accountDetails.userOpsSenderLength + accountDetails.userOpTargetLength
Let this be a single number with no copy or link to anything

Deployed at
accountDetails.blockTime + " from the following hash - " + accountDetails.userOpHash
Let the userOpHash link to/userOpHash/<accountDetail.userOpHash> page. We can represent the blockTime using moment js and deploy how long ago the account was created. similar to how we display block time in userOps table ( Ago column )

Deposit Amount
accountDetails.depositAmount
This number will be of the form ETH , so expect the number to grow up 10^18 and more. So we need to converted it to the form of ETH/GWEI/WEI. You can check in utils for the function.

Status seems to be shown wrong in the main page.

Update on the latest commit -
I will go over the functional changes needed to be done before getting into the code

Please  link all sender and receiver addresses to the account page. We do not need the address page.

There is a miss-match on the icon sizes , are we using different components for the table. We could use the same internal component for the values if so.

Please move away from using userOpsSender and userOpsTarget in the account page to render the table. It will no longer be sent in the API


Allow to set if Variables, Styles or Hex values are applied on applying tokens

When I have tokens linked with variables and styles, I want to set what should be applied on Figma when tokens are applied through the plugin.
Introduce a setting in the settings tab that will allow a default to be set, with each option explained.

Variables
Styles
Hex values

This behaviour should happen for all token types depending on whether they are applicable in variables and styles.

It would be great to let users know which ones were skipped (and why) so they can self solve by creating manual variables or  new tokens.
The other option @six7 mentioned in slack is to duplicating the token with a suffix, one for each value in the single token.
So padding-token with value 0, 8 would be

padding-token-SUFFIX = 0
padding-token-SUFFIXALT = 8

This sounds great!
If they change their mind later, how can they switch to an alternate representation? Ex: Variables today, styles tomorrow
If the representation doesn't exist any more, like if variables are deleted, then what?
Are the settings on the file side, or the plug-in side? If I want to have multiple files using the same tokens can I have 1 file showing styles and another showing variables? This gets into the multiple file sync needs convo.

Will there also be consideration for the Figma feature of referencing a variable in a style for this one?

I'm going under the assumption that 'Hex' should actually be Raw value?
For example if I don't want a border-radius variable to be applied, and just want the number.
Can you confirm @esthercheran ?

@LukeFinch yes the raw value

@SamIam4Hyma would it better to have this setting specific to a document?

UX Research shows that this feature is desired, but the wording in the design mockups was confusing and most users were unsure that the settings is the best place for this menu option.
If we have an "apply to" settings, could it also be accessed there?

Added to FeatureBase Roadmap : ❖⚙ Choose to express a token as a style, variable or raw design property

Will be part of 2.0 RC7


Unit test to ensure retransmitted message reaches the peer exchange

Problem
Need an end to end test where

A peer registers a unsolicited message handler
Message sent by the sender gets dropped
The message is retransmitted via CRMP after ack timeout
The peer receives the message via handler
The sender get the ack for the message
All the state is cleaned up correctly

Change overview
Added the test mentioned in the problem statement.
Testing
CheckResendApplicationMessageWithPeerExchange is the new test.
Only unit test was added, and no functional code was modified.

rebased


Exit gracefully if the file to be compiled doesn't exist

Provide proper error message and exit gracefully.
refer

I would like to work on this issue...


Update Get-DtcTransaction.md

Missing description has been updated now.

Docs Build status updates of commit 350c3a6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2012r2-ps/msdtc/Get-DtcTransaction.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Replaces #1392.

Hi @sowjanya365 and @e0i,
Reviewing this PR, I noticed some errors not related to this PR. In the CimSession section, of the docset/winserver2012r2-ps/msdtc/Get-DtcTransaction.md file, there are two markdown links that need fixing. Along with their link headers needing to be updated from "http" to "https". These can be resolved in a new PR.
Thanks!


Drop python2.7?

This repository still claims to support python2.7, however the work to maintain that will only got up -- lots of libraries are dropping py27 support (and a recent PR seems to indicate the version of Automat we end with no longer works with py27).

Python 2 is EOL for so long, I don't think there is any need for much discussion here. Even major distros (IIRC at least Fedora, Nix, Gentoo) have actively started efforts of removing as much Python 2 code from their packages as possible. So even if this application did still maintain Python 2 support, it would likely not be packaged in a way that allows running it with Python 2 whatsoever.

six is not used anywhere, so I guess it can be removed from install_requires, right?


Unit tests for project configuration

https://github.com/VirtusLab/scala-cli/pull/361 shows that our integration test suite isn't perfect and we need more finer-grain unit tests to verify if various configuration options are working correctly.

IMO we can close it issue. Now, we have classes TestInputs.scala which is used for unit test, for example here.


fix(deps): update dependency python-multipart to ^0.0.6

This PR contains the following updates:

Package
Type
Update
Change

python-multipart (changelog)
dependencies
patch
^0.0.5 -> ^0.0.6

Release Notes

andrew-d/python-multipart
v0.0.6
Compare Source

Migrate package installation to pyproject.toml (PEP 621) #54.
Use yaml.safe_load instead of yaml.load #46.
Add support for Python 3.11, drop EOL 3.6 #51.
Add support for Python 3.8-3.10, drop EOL 2.7-3.5 #42.
QuerystringParser: don't raise an AttributeError in __repr__ #30.

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


Fixes #782

I'm inclined to force merge this as soon as tests pass.  @addiehl - wondering if you recognise this as a stray axiom that should be defining some term, rather than hanging around as a GCI assertion about all things that are in_taxon some human.

@matentzn this kind of thing could potentially do some serious damage.  I'm wondering what tests we might think up to try to avoid this happening again.  It might be sufficient to add some disjointness declarations - or perhaps also to have a warning for GCIs lacking a genus on the left.

@matentzn @nicolevasilevsky - I think worth running a second release today to ensure this is not floating around causing weird inferences in the OBO world.  Do you have time to run one?


Add convenience methods for Duration

FromHours
FromDays

coverage: 99.315%. first build
when pulling 46827ff03c04b974bb2404f912ccc15ee0ac6a02 on users/alexpeck/dur2
into 13c005a9269335aed15b964cf33b075a9c2d7d19 on main.


Bug: [AutocompletePlugin] - Cannot assign to read only property '_cachedNodes' of object '#' on undo

Lexical version: v0.20.0
Steps To Reproduce

Go to lexical playground
Turn on autocomplete from settings
focus on the editor and select any format(bold for e.g)
type "test"
press tab when you see the suggestion
undo

The current behavior
https://github.com/user-attachments/assets/85c551e6-055c-4687-9d7f-cd8c10a9f72a
I'm not sure if it's related to the autocomplete plugin but here is the stack trace:
Uncaught TypeError: Cannot assign to read only property '_cachedNodes' of object '#<RangeSelection>'
    at RangeSelection.setCachedNodes (main-C507j1EI.js:48:6194)
    at d0.getWritable (main-C507j1EI.js:45:11751)
    at d0.markDirty (main-C507j1EI.js:45:15474)
    at Array.tt (main-C507j1EI.js:55:27349)
    at $applyTransforms (main-C507j1EI.js:51:27878)
    at $applyAllTransforms (main-C507j1EI.js:51:28705)
    at $beginUpdate (main-C507j1EI.js:51:35644)
    at updateEditor (main-C507j1EI.js:51:36873)
    at LexicalEditor.update (main-C507j1EI.js:51:59695)
    at triggerCommandListeners (main-C507j1EI.js:51:33874)

The expected behavior
This error should not occur

This generally happens when $setSelection(selection) is used with a frozen selection from a previous state when the correct option is to use $setSelection(selection.clone())


Does it support Oracle DB??

Here is what to include in your request to make sure we implement a solution as quickly as possible.
1. Description
Does this framework support Oracle DB??
2. Exception
If you are seeing an exception, include the full exception details (message and stack trace).
Exception message:
Stack trace:

3. Fiddle or Project
If you are able,
Provide a Fiddle that reproduce the issue: https://dotnetfiddle.net/25Vjsn
Or provide a project/solution that we can run to reproduce the issue.

Make sure the project compile
Make sure to provide only the code that is required to reproduce the issue, not the whole project
You can send private code here<EMAIL_ADDRESS>
Otherwise, make sure to include as much information as possible to help our team to reproduce the issue.
4. Any further technical details
Add any relevant detail can help us.

Hello @BhusanBibhuti ,
Yes most of our features if not all our features should support it.
Did you get an issue with a feature?
Best Regards,
Jon

Sponsorship
Help us improve this library
Performance Libraries
context.BulkInsert(list, options => options.BatchSize = 1000);
Entity Framework Extensions • Bulk Operations •  Dapper Plus
Runtime Evaluation
Eval.Execute("x + y", new {x = 1, y = 2}); // return 3
C# Eval Function • SQL Eval Function


[RFC] Allow a LabVIEW service to return DoubleXYData

[x] This contribution adheres to CONTRIBUTING.md.

What does this Pull Request accomplish?
This change allows a LabVIEW service to return DoubleXYData in its outputs. This method basically uses 'duck typing' in that the any cluster in the Measurement Results.ctl is inspected to see if it matches the pattern of 'two double arrays in a cluster'. If it does, it is considered DoubleXYData and the parameter type will be TypeMessage and the message type will be ni.protobuf.types.DoubleXYData.
Other options I thought of would involve us shipping some kind of 'official' typedef that represents a DoubleXYData cluster. This could be one we create in LabVIEW or possibly one of the typedefs created by generating the xydata.proto file into a LabVIEW project. I haven't tried doing this code generation yet, but plan to soon.
Why should this Pull Request be merged?
Not ready to merge - trying to decide on the approach.
What testing has been done?
None - RFC.

@pbirkhol-ni @jasonmreding I updated the data type checking. I'm turning this into an actual PR now. It can't be tested and checked in until my other PR #400 is submitted.


Build for android

Whei i do the steps in the README the applications show that there's no ./build_pcapplusplus.sh: line 107: ./configure-android.sh: No such file or directory inside the directory.
❯ cd ToyVpn-PcapPlusPlus
❯ ls
app                    gradle              LICENSE           settings.gradle
build.gradle           gradlew             local.properties
build_pcapplusplus.sh  gradlew.bat         README.md
docs                   import-summary.txt  server
❯ ./build_pcapplusplus.sh /home/calper/Android/Sdk/ndk/25.1.8937393 /home/calper/projects/PcapPlusPlus
Please specify the NDK path using the '--ndk-path' switch. Exiting...
❯ ./build_pcapplusplus.sh --ndk-path /home/calper/Android/Sdk/ndk/25.1.8937393 --pcapplusplus-path /home/calper/projects/PcapPlusPlus
./build_pcapplusplus.sh: line 107: ./configure-android.sh: No such file or directory
❯ ls
app           build_pcapplusplus.sh  gradle   gradlew.bat         LICENSE           README.md  settings.gradle
build.gradle  docs                   gradlew  import-summary.txt  local.properties  server
❯ code .
~/projects/ToyVpn-PcapPlusPlus master*

@calperxd that's a good point! We recently switched from a custom build system to CMake, so if you get the latest PcapPlusPlus code from master the configure-android.sh no longer exists.
As a workaround, you can fetch the code from the latest official release: https://github.com/seladb/PcapPlusPlus/releases/tag/v22.11, which doesn't yet include the new CMake build system


Meta: Add lint-commits script and lintcommits Github Actions workflows

Copied from serenity with some light modifications :^)
Also add a pre-commit config to use the lint commits script and run cargo fmt.

Closes #182


Sync docker/docker refs with files mistakenly edited in docker.github.io repo

- What I did
Synced improvements to docs/reference/* that were accidentally applied in docker/docker.github.io to docker/docker since it is supposed to be the source of truth for references
- How I did it
docker/docker.github.io branch master actually represents the 1.12.x branch of docker/docker. #27518 represents an attempt to sync those changes with the v1.12.x branch of docker/docker, but we realized that changing the TOML to YAML would break the make docs target which still exists in the 1.12.x branch. So this PR is simply the same patch as in #27518 but applied to docker/docker master instead.
The implication of only applying these changes to docker/docker master is that we can't improve the reference docs on docs.docker.com until Docker 1.13 is released. because we can't refresh the 1.12.x reference docs anymore (because of TOML / YAML incompatibility). Not sure what the answer is here.
Thoughts appreciated @thaJeztah @dnephin @johndmulhausen

IMHO we need to stop storing these files in two repositories.
People are going to continue to get confused, because having the same files in two repositories is extremely confusing.
If the source of truth for these files in the docker/docker repository, why do copies of these files need to be checked into docker/docker.github.io? Why can't the process of building the docs just checkout whatever other repositories it needs to get these files?

@mstanleyjones Could we make a "post-processor" that makes either branch read toml or yaml and output the desired format? Or is this just a temporary problem?

@aaronlehmann This is a temporary problem until we get a more automated approach going. That automated approach will use the files in docker/docker.

@mstanleyjones While you have these files open, can you put an HTML comment at the top of these pages that specify that the home for these files is in docker/docker? Hopefully that helps to prevent confusion in the future. I'm also going to create a YAML manifest that heads off "edit this page" at the pass.

@johndmulhausen done, that was boring!!!


[5.6] Allow concatenation & file for IP whitelisting when using artisan:down

In #24003, we can add IP addresses to access the site, when using php artisan down. One example is:
php artisan down --allow=<IP_ADDRESS> --allow=<IP_ADDRESS>/16

The issue with this is that in order to white list multiple IP addresses we have to add the allow option multiple times.
This PR further modifies the behavior to allow a user to do the following:

Allow multiple IP addresses through a single allow option by using a , or | symbol:

php artisan down --allow=<IP_ADDRESS>,<IP_ADDRESS>/16

Add a file path to allow option. The file will contain IP addresses in the following format:

<IP_ADDRESS>
<IP_ADDRESS>/16

I would just make a composer script in your composer.json that does the long command you want.


Add dubbo test case

Add dubbo test case.

Codecov Report

Merging #20 into master will increase coverage by 0.63%.
The diff coverage is 84%.

@@            Coverage Diff             @@
##           master      #20      +/-   ##
==========================================
+ Coverage   47.93%   48.57%   +0.63%
==========================================
  Files          45       45
  Lines        1185     1190       +5
  Branches      185      185
==========================================
+ Hits          568      578      +10
+ Misses        472      466       -6
- Partials      145      146       +1

Impacted Files
Coverage Δ

...fa/rpc/boot/container/ConsumerConfigContainer.java
40% <ø> (+6.66%)
:arrow_up:

...boot/context/ApplicationContextClosedListener.java
62.5% <ø> (ø)
:arrow_up:

...ay/sofa/rpc/boot/config/SofaBootRpcProperties.java
28.03% <ø> (ø)
:arrow_up:

...a/rpc/boot/health/RpcAfterHealthCheckCallback.java
70% <0%> (+6.36%)
:arrow_up:

...fa/rpc/boot/runtime/parser/GlobalFilterParser.java
15% <0%> (ø)
:arrow_up:

...ay/sofa/rpc/boot/container/RpcFilterContainer.java
38.46% <100%> (+3.32%)
:arrow_up:

...pc/boot/runtime/converter/RpcBindingConverter.java
53.9% <100%> (ø)
:arrow_up:

...ofa/rpc/boot/context/SofaBootRpcStartListener.java
100% <100%> (ø)
:arrow_up:

...ay/sofa/rpc/boot/SofaBootRpcAutoConfiguration.java
100% <100%> (ø)
:arrow_up:

...m/alipay/sofa/rpc/boot/container/SpringBridge.java
75% <80%> (-2.78%)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4d967c...17ea768. Read the comment docs.


[Improvement][UT] Improve Worker runner coverage

Purpose of the pull request
resolve: https://github.com/apache/dolphinscheduler/issues/15396
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Hi, @pegasas , please fix CI

Hi, @pegasas , please fix CI

It seems there's a lot refactor after last merge.
I will fix & re-commit.

Hi, @pegasas , please fix CI

It seems there's a lot refactor after last merge. I will fix & re-commit.

fixed.

@pegasas please check the failed CI

@pegasas please check the failed CI

Done.
run spotless successfully in my local

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (8a35e8b) 38.55% compared to head (982f745) 38.89%.

:exclamation: Current head 982f745 differs from pull request most recent head dba04ec. Consider uploading reports for the commit dba04ec to get more accurate results

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15428      +/-   ##
============================================
+ Coverage     38.55%   38.89%   +0.34%
- Complexity     4776     4820      +44
============================================
  Files          1310     1310
  Lines         44880    44875       -5
  Branches       4808     4785      -23
============================================
+ Hits          17304    17455     +151
+ Misses        25693    25524     -169
- Partials       1883     1896      +13

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

will merge tomorrow if no more comments


test: add failing assertion to flaky swift test and ignore it

Signed-off-by: Snow Pettersen<EMAIL_ADDRESS>Adds an assertion that we don't get a cancel callback, which fails consistently locally. Ignore the test
until we can fix the underlying issue
Description:
Risk Level: Low
Testing: n/a
Docs Changes: n/a
Release Notes: n/a

@goaway @buildbreaker if either of you can re-approve this it should reduce more of the CI failure rate on other PRs


An UI to add hosts and services dynamically

With an UI, you can allow non technical person to add more tests (whiteout restart server)

Done in watchmen 3.x


detect if ansible is available on the guest host

otherwise use ansible_local
This is mainly a temporary fix until vagrant-lxc works with ansible_local
But this way we can test the rawbot-virtualization at least on windows

The idea to use ansible_local when ansible is not available seems really a great idea waiting for a fix of the lxc issue.
Don't sure I get the systemd issue however, it should be installed in the jessie image since it was causing issue on shutdown. @sephii can you confirm the image has no systemd ?
Maybe it got pulled by one of the package from dotdeb ?

The systemd package is installed in the box but sysvinit is used because of the shutdown issue you mentioned.

So it's safe to delete that again in the playbook?
It's only a problem when using ansible_local, then it can't restart nginx. It works fine with just ansible

I think we could remove the systemd package in the base playbook (or we could recreate the base boxes but that's a pain to do).

This MR adds
+- name: remove systemd
 +  apt: name=systemd state=absent
 +  sudo: yes

so that's already solved ;)

It seems to run on windows, with one little change as windows doesn't support symlinks
I added what to do (copying Vagrantfile instead of symlinking and adding an ansible.cfg to the roles dir) in the README.md. This is only needed if you have developers with Windows, so I don't think this should be the default.

Here's the discussed new way.

One little thing just came to my mind: There's a BC break now, so if you point your submodule to this new  version, your old project won't work anymore (since the symlink from Vagrantfile to virtualization/rawbot-virtualization/Vagrantfile will be broken).
I see 3 possibilities

Just leave it like that and let people realize it when they try to start their vagrant box later

Make a Vagrantfile in the old place which spits out some errors and warnings what to do now

Keep the old one

and 3) have the disadvantage that there is a Vagrantfile, which looks like it's the real thing, but it isn't, it's just a BC placeholder (of course one can include in the file a comment about that)

and for 1). Such BC breaks are not nice (and it should be documented somewhere what to do in that case), but I don't know how many projects you have with this setup and how often you update them.

I took the liberty to make other commits on master taking most of your idea.
I fixed the 'ansible_local' thing a bit differently : https://github.com/team-rawbot/rawbot-virtualization/commit/32625674b4eabf3f34949f5944bc4cac51034b32
I think I also found a way to avoid the BC break : https://github.com/team-rawbot/rawbot-virtualization/commit/28d6be0f475d26ae34bc61b3b4f37269bd2e1596
Basically, I didn't renamed the Vagrantfile, I added some compatibility layer in it... Didn't tested it thoroughly for now tho. I'll try that once I merged some other PR in a big test party :P


Upgrade golang to 1.19.12 to address CVE-2023-29409

Signed-off-by: Muhammad Falak R Wani<EMAIL_ADDRESS>
Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

[x] The toolchain has been rebuilt successfully (or no changes were made to it)
[x] The toolchain/worker package manifests are up-to-date
[x] Any updated packages successfully build (or no packages were changed)
[x] Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
[x] Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
[x] All package sources are available
[x] cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
[x] LICENSE-MAP files are up-to-date (./SPECS/LICENSES-AND-NOTICES/data/licenses.json, ./SPECS/LICENSES-AND-NOTICES/LICENSES-MAP.md, ./SPECS/LICENSES-AND-NOTICES/LICENSE-EXCEPTIONS.PHOTON)
[x] All source files have up-to-date hashes in the *.signatures.json files
[x] sudo make go-tidy-all and sudo make go-test-coverage pass
[x] Documentation has been updated to match any changes to the build system
[ ] Ready to merge

Summary

What does the PR accomplish, why was it needed?
Address CVE-2023-29409 & Introduce patch from upstream to fix http regression
Change Log

Bump Go 1.19.11 -> 1.19.12
Introduce patch to permit requests with invalid header

Does this affect the toolchain?

No
Associated issues

Related PR:

https://github.com/microsoft/CBL-Mariner/pull/5936

Links to CVEs

https://nvd.nist.gov/vuln/detail/CVE-2023-29409

Test Methodology

Pipeline build id: Local build.

BuddyBuild:

AMD: link
ARM: link

I see that four spec files require a golang version lower than we will have going forward. I don't think this exactly matters, but it feels wrong. Please either confirm it doesn't matter or change it. The files are:
helm.spec
influx-cli.spec
influxdb.spec
terraform.spec

Looked at the OpenSSL bits, seems ok, but this does look like something we should look at.

I see that four spec files require a golang version lower than we will have going forward. I don't think this exactly matters, but it feels wrong.

Absolutely, IMO it matters; if we try to build the specs with anything higer than 1.18.8; they are unhappy.

Please either confirm it doesn't matter or change it. The files are:
helm.spec
influx-cli.spec
influxdb.spec
terraform.spec

We specifically have go1.18.8 for these specs: golang-1.18.spec
Thank you for being vigilant. 🥇

This will also address https://nvd.nist.gov/vuln/detail/CVE-2023-39533
NIST "To protect one's application, it is necessary to update to these patch releases and to use the updated Go compiler in 1.20.7 or 1.19.12."


Swift2 visual API

Exposed some objects in order to be able to customise like the image:

PS: Updated the ObjC example to use the latest API

Thanks! Particularly for updating the ObjC example, that's very helpful.
Can you explain how you used the newly public contentView to customize your implementation?

@nickoneill This is what I 've been doing to the content view:
_multiPscope.contentView.backgroundColor = [UIColor darkBlueColor];
_multiPscope.contentView.layer.borderColor = [UIColor darkBlueColor].CGColor;
I just noticed the the ObjC example does not work as expected. If I use [NotificationsPermission new] it crash looking for an init() method. If I use [NotificationsPermission alloc] it somehow understands as ContactsPermission. This happens to every permission type. Any ideas what might be wrong?

Never mind. I just fixed it. The good old alloc/init did the trick. The problem was with NotificationsPermission which didn't carry an init() method but instead carried initWithNotificationCategories().
I tool the liberty to migrate every "new" call to alloc/init for consistency.

Nice. Pending: Update README about new properties.


markdown: Use html5  tag for timestamps.

Previously, we had implemented:

The new syntax is:
    <time class="timestamp" timestamp="ISO 8601 string"></time>
Since python and JS interpretations of the ISO format are very
slightly different, we force both of them to drop milliseconds
and use 'Z' instead of '+00:00' to represent that the string is
in UTC. The resultant strings look like: 2011-04-11T10:20:30Z.
Fixes #15431.

Testing Plan:
Manual and automated tests.
GIFs or Screenshots:
No UI changes.

The <time> element wants a datetime attribute, not a timestamp attribute.
We shouldn’t apply the timestamp styling to invalid times; that might lead to mistakes where something that looks like a timezone-aware time isn’t.
pytz.utc should be datetime.timezone.utc.

I think the if/else logic with the tag variable is a bad idea; it'd be a lot clearer in both markdown processors to just do repeat ourselves a little.  Especially since we don't actually want to use the same styling for errors.

OK I mostly rewrote this to apply my feedback.

Just pushed back a version that passes all tests.  I'm still not sure I'm 100% happy with the error handling:

The rendered_markdown.js logic makes a time object with the text "Could not parse timestamp" in the hopefully-never-happens case of invalid data in a <time> object.  That's probably fine; this shouldn't happen.  But maybe it'd be better to just remove the <time> element altogether and just display the original text?  Not sure.
We need some sort of CSS for span.timestamp-error.  I'm basically fine with it having relatively verbose error text, given that I imagine these will all be generated by humans.  Though it's unpleasantly odd in cases where bugdown can understand the format and marked cannot; you'll get  a locally echoed complaint about invalid format that then turns into perfectly happy rendering.
We wanted to look at changing the markdown input for this feature; that can be a separate PR.
We wanted to look at adding a clock icon decoration to the time tag styling.

We should update the commit mesaage to reflect the final syntax here.

Merged the two commits ending with 3ec068f10cdccc2076cb6555a33c2cd0d3cbc328.  We can work on additional improvements in this PR.

Merged a documentation update as 1094e533bf1a28c29d93586047191648804dab28.

I didn't comment here earlier, sorry. I'm happy with the error handling markup. That message on the other hand... I have some fundamental issues with having an error message in someone's message content, but that's not exactly a syntax issue.
I'm closing this PR.


Refactor coverage.rb

This pull request refactors coverage.rb and replaces the use of the Singleton mixin with a simpler singleton implementation that allows for instances to be created and tested. It also replaces the use of the at_exit Ruby hook with the RSpec after(:suite) hook since we're already working within the RSpec ecosystem.

:+1:


Tender: Identify Trainees on Slack

https://docs.google.com/document/d/1YYGO4VW1oXKLVZRmQ5rqdOG37tvVxAaODpzmpxCjsMI/
Submit your bids by commenting on this issue. You can see a template bid here https://docs.google.com/document/d/1_ZYRPX4JnDdY1F_C1CVyq99YPuV33eDs1Lk5A0wOBOE/edit#heading=h.tqtjzcfetl58
UPDATE:
We will leave this tender open until
OCTOBER 24 2023 23:00 UTC
and then give our decision by
OCTOBER 27 2023 23:00 UTC

Hi Sally.
If this is an automated solution, is there a database or spreadsheet available that contains volunteer data such as emails that maybe can be used to check against Slack user profile data to see who is a trainee and who is not?

@kfklein15 I think this was your tender. Can you provide some extra details?

Hi @haroon-ali-dev , I am not sure I understand the question about this being automated? Are you referring to new or current trainees?
We have the current applicants' e-mail addresses. However, they might not be using the same one. So we need to be creative and think about a solution for this, which I can give you, but I was hoping to be given other options - since mine might not be the right way.
Nowadays, we have no differentiation on Slack for staff, trainees, volunteers, etc.

@kfklein15 Hi. By automated I meant if the identification of all the trainees is going to be done with software or manually where we are looking through data in spreadsheets and then manually changing the profile info in Slack.
I did have a possible solution in mind but I wasn't sure whether to discuss it first or create a bid first. It's my first time dealing with tenders.
The solution I had in mind was a GitHub Actions cron job that runs daily/weekly. The Slack API is used to pull all users of the workspace. The Google Sheets API is used to pull a list of trainees from a spreadsheet. Their emails and names are checked and if they match then 'Trainee' is added to the Title field in the Slack profile of the user. But you're saying that the emails might not match so then maybe some names won't match either.
Another solution I was thinking of was manually sorting out all the information for current trainees then for new trainees use Zapier to trigger an action when a new user joins the workspace. The action for this trigger sends an email to that person asking them to add 'Trainee' to their profile if they are a trainee.

@kfklein15

This is now making me think about something happening in a Slack app. Like a Slack bot that might ask a new user if they are a volunteer/trainee etc. I wonder if that's possible.
With this I'm thinking along the lines of looking through Slack data and finding out who is who and manually changing their profile because why would software be built for that if it's going to run only once.

I hope my train of thought is going down the right route 😅.

@haroon-ali-dev

I would be even simpler: how do you not let people into Slack WITHOUT this information? It should be a barrier, not an afterthought.
Agree.

@kfklein15
I've looked around and wasn't able to find a way to allow the user to select roles when joining the workspace.
How about a Slack app that does two things:

When a new user joins it sends them a DM with role options to select. When they select one, their profile is updated.
Every time the user posts a message in a channel (or something else), the app is triggered and checks the user's profile info and if the role field is blank it sends them that same DM again with the role options.
This should enforce all users to select a role.

Could you maybe give a hint about the solution you have? It will help make things even more clearer.

@haroon-ali-dev, I don't have a clear solution. But nowadays we have lots of fields on the Slack profile (check your's out). Could one of them be what we need? And could it be a a blocker - without setting this up you cannot join Slack?

@kfklein15
Yes we have many fields available in the profiles of Slack users and we can also add custom fields. But we can only do something with them after the user gets into the workspace. I've had a thorough look. There doesn't seem to be a way to prevent users from joining the workspace in relation to these fields at all.
But after users have joined, custom Slack apps can be used to run actions based on triggers such as new users joining, users joining channels, sending messages etc. The users can be sent options to click, their profiles can be changed based on those options. So a lot can be done with those fields after the user joins using custom Slack apps. So maybe we need to think about the solution in that realm.

Qualified Supplier
Declaration
I am a qualified supplier, and I am tendering for this work.

I am a CYF-certified developer. I have graduated CYF Fundamentals.
I have a minimum of 4 kyu on my Codewars profile and I am happy to walk through any kata on my profile on request.
I have a portfolio website with at least two team and two solo projects. I have provided links.
I have right to work in the UK which I can prove with an online share code.

Portfolio

Portfolio website
Solo project 1
Solo project 2
Team project 1
Team project 2

Contact Details

Name: Haroon Ali
Email<EMAIL_ADDRESS>CV: View

Proposal
A custom Slack app built with Node.js that will automate the process of assigning and verifying user roles, facilitating better organization and ensuring the preservation of important user distinctions, particularly for trainees.
Components
1. User On-boarding

Upon a new user joining the Slack workspace, the custom Slack app will be triggered.
The app will send a direct message to the user, presenting them with three role options: Staff, Volunteer, Trainee.
The user will select their role from the provided options.
The app will then update the "role" field in the user's profile with their selection.

2. Role Validation

Whenever a user posts a message in a channel, the app will be triggered.
It will check the "role" field of the user.
If the field is blank, the app will resend the previous direct message offering role selection.
This process will continue until the user selects a role, ensuring that all users, both new and existing, have a designated role.

Benefits

Clear Role Assignment: The app will facilitate consistent and accurate role assignment for all users.
Improved Organization: The organization will have a better understanding of its members and their roles.
Trainee Preservation: The system will identify trainees, reducing the risk of their accidental removal during user cleanup operations.

Deliverables

Create custom Slack app - £250 - 27th October
Transition plan - £50 - 1st November

Considerations

The custom Slack app will need to be deployed. A good option might be to use in a server less function hosted on a platform like Vercel which gives a very generous amount of free usage every month. Not sure if the deployment will be done by the supplier. This isn't clear in the docs. If the supplier deploys, then will need access to Slack API keys.

Thank you, @haroon-ali-dev , for your tender submission.
The solution is good, but it is not achieving what I wanted, which is getting this information from the start. We have plenty of data. We ask our trainees or volunteers to add their profiles, but no one does it. And I am not sure that a bot will actually make people do it. Why do you think this would work?
I know you investigated quite a lot, but is there any other way we think we could do this from the start?

@kfklein15
I could not find a way to get the information before joining the workspace. Slack does not provide the ability to add custom fields to the workspace joining process which the user could use to select a role before they join. There doesn't seem to be a way to do this at all.
The reason why I think the solution would work is because the user will be prompted to give the information frequently on Slack. Surely every user reads their DMs. So when a Slack bot keeps sending them a DM to select a role, they will surely select because they want to stop getting the DMs. This is very different than posting a message in a channel and asking users to make sure that they select a role, or sending everyone an email, because a bot can keep doing it for us. Also, the bot will provide the roles to select within the DM itself instead of asking them to go into their profiles and do it which speeds up the process. Let's say in the end not every new user has selected a role, but I'm sure most will which should make the job of identifying users a lot easier for whoever has to do it.
I've been researching with Google, AI and also talking with one of my final project mentors and now I think there can't be a perfect solution to this, but only a solution which makes the job of identification easier.

@kfklein15
If someone has multiple roles then the options for selecting can be like:

Staff
Volunteer
Trainee
Staff & Volunteer
Staff & Volunteer & Trainee
Volunteer & Trainee
Trainee & Staff

That covers all options I think. Very simple instructions for selecting can be shown to the users by the Slack app. For example: "Please select an option that covers all your current roles and any roles you have had in the past."
Choosing to identify trainees only, with the Slack app won't make my solution/project smaller because the options for selecting other than 'Trainee' would just be part of the options list in the code so we're talking about removing just a few lines of code.
You could also go with not having any software solution at all and just do a manual edit for all current trainees. Then when the time comes near for deactivating inactive users, or some other need, you create a new tender for manual editing. And you keep repeating this.

Thanks, @haroon-ali-dev
Doing it manually every year or 6 months means incorrect data for that period, which means we can never automate our offboarding process.

🤝 Contract Awarded
I am awarding this contract to @haroon-ali-dev pending these amends:

Only trainees should be identified, not staff and volunteers
The "Manually edit Slack data to identify all current trainees" deliverable to be kept at the original price of £100
Please schedule transition in the New Year so it doesn't get lost in the Christmas holidays

~ @kfklein15
👣 Next steps

Finalise your proposal with any amends.
Book a call with @kfklein15 , link in my Slack Profile

🎉 Thank you.

@SallyMcGrath Hi Sally. Does the code for the custom Slack app (or any code for any tender) need to be reviewed by volunteers? Do we need to create tests?
Or is it that if the code works fine then that is enough?

@SallyMcGrath
Does the code for the custom Slack app (or any code for any tender) need to be reviewed by volunteers?
Do we need to create tests?
Or is it that if the code works fine then that is enough?
@SallyMcGrath
Also, is deploying the bot part of my work? Or do I hand the repo off to someone in dev ops and then they do it?
@kfklein15 @SallyMcGrath
There is an issue with the manual-editing part.

After messing about with my own Slack workspace, I saw that Slack doesn't let us change custom fields at all through the dashboard. Even on paid plans. We can only change them through the API on paid plans.
Karen has informed me that CYF has a non-profit pro plan, so that means that the fields can be changed with the API. But for this, a field's data source has to be changed to 'API' which means that it cannot be manually changed. Also this approach will require building two scripts where one is used to search the user by email to get the user ID and one is used to update the profile field using the user ID.
I'm not sure now if manual editing can be done. What do you advise?

Thanks, @haroon-ali-dev, for looking into this another time. Let's remove the manual side from the tender and focus on the automated bot on welcome and when active again.

@kfklein15 I've removed the manual side from the tender. I think the next step is for me to sign a contract so I'm waiting to receive that from you.

Does the code for the custom Slack app (or any code for any tender) need to be reviewed by volunteers? I suggest you get a review by asking the community. Do your work in public.
Do we need to create tests? Yes you need to write tests.
Or is it that if the code works fine then that is enough? Tests please. 😄

@haroon-ali-dev, as per your point above, I am adding the following to the tender:
Deliverable
£50 Deployment of the system into CYF infrastructure
Definition of Done
Deployment is done by using the services that CYF uses, such as Netlify/AWS, etc, using the DevOps channel in Slack for guidance and approval
Let me know if you have any queries.

@kfklein15 thank you, I've added the deliverable to my bid.


Create graph & row for JTI firewall filter

Need to create a graph and row under data_streaming_collector  for JTI  firewall filter
Structure below as a reminder
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"foo_filter_output-et-19/0/3.1-o","filter_timestamp":1477327298,"filter_counter_name":"c300-et-19/0/3.1-o","type":"filter_counter.packets","value":121340}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"foo_filter_output-et-19/0/3.1-o","filter_timestamp":1477327298,"filter_counter_name":"c300-et-19/0/3.1-o","type":"filter_counter.bytes","value":6795040}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"match-dscp","filter_timestamp":1477327298,"type":"memory_usage.HEAP","value":3464}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"BACKBONE-SAMPLE-V6","filter_timestamp":1477327298,"type":"memory_usage.HEAP","value":1496}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"LOOPBACK-V6","filter_timestamp":1477327316,"type":"memory_usage.HEAP","value":54280}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"LOOPBACK-V6","filter_timestamp":1477327316,"type":"filter_counter.packets","value":86392,"filter_counter_name":"DENIALS-LO-V4"}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"LOOPBACK-V6","filter_timestamp":1477327316,"type":"filter_counter.bytes","value":6220224,"filter_counter_name":"DENIALS-LO-V4"}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"__default_bpdu_filter__","filter_timestamp":1477402764,"type":"memory_usage.HEAP","value":2424}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"chash-test","filter_timestamp":1477402935,"type":"memory_usage.HEAP","value":9856}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"__default_arp_policer__","filter_timestamp":1477327298,"type":"memory_usage.HEAP","value":1584}
2016-11-03 00:43:10 +0000 jnpr.test: {"device":"BAR-re0:<IP_ADDRESS>","filter_name":"__flowspec_default_inet__","filter_timestamp":1478106192,"type":"memory_usage.HEAP","value":707600}
2016-11-03 13:25:20 +0000 fluent.debug: {"message":"Extract sensor data from BAR-re0:<IP_ADDRESS> with output structured"}

done


Rarity

It doesn’t work when I add the key _sr, _r etc. I tried to find the way to give rarity to the different layers but it doesn’t with just rename the files. Please Help

The rarity system has not been implemented on Main branch.
I've created a pull request, you can find my fork here


Provide a compatible method to conditionally import between WASM and regular Web with also backward compatibility

Dio is embedded with the Web HTTP request support based on dart:html. It also widely supports Dart SDK >=2.18.0 applications.
During the release of the WASM support, the package:web requires Dart SDK >=3.3.0 which is way smaller than our supported range.
Many contributors including ourselves have tried to find a static way to split the Web support into packages without making any breaking changes, but that seems impossible because we need a compatible way to conditionally import libraries based on the environment, which is not provided by Dart if you applied the above restrictions.

https://github.com/cfug/dio/pull/2215
https://github.com/cfug/dio/pull/2218
https://github.com/cfug/dio/pull/2223

Please consider supporting the request or kindly provide a workaround, because there are millions of applications that are using Dio. Huge thanks for any input.

What if you do something like:
import 'stub.dart'
  if (dart.library.html) 'html_impl.dart'
  if (dart.library.js_interop) 'js_interop_impl.dart';

If the evaludation is short circuited, then the html implementation should be used on old web platforms and the js_interop implementation should be used on WASM.

What if you do something like:
import 'stub.dart'
  if (dart.library.html) 'html_impl.dart'
  if (dart.library.js_interop) 'js_interop_impl.dart';

If the evaludation is short circuited, then the html implementation should be used on old web platforms and the js_interop implementation should be used on WASM.

This could work if the condition is not running into any compile issues.

Potentially we could treat this as a duplicate of https://github.com/dart-lang/sdk/issues/55694?
cc @osa1

Potentially we could treat this as a duplicate of https://github.com/dart-lang/sdk/issues/55694?

Yes, I think this is a duplicate #55694.

I'll close this so we can track the idea of the new condition in the other issue.
I forgot to mention - the workaround suggested by @Rexios80 above is also what we recommend for distinguishing runtimes under the current available conditional import conditions.
dart:html is supported in JavaScript compilers (DDC and dart2js) and not Wasm, so dart.library.html  can be used to distinguish between the various JS runtimes, similarly dart.library.js_interop helps distinguish wasm from the VM runtime.
In the past some people have used js and js_util, but it's likely those will not be supported in Wasm in the future, and as such, we normally recommend using .js_interop instead.

This could work if the condition is not running into any compile issues.

Can you clarify what's your concern? I'm not sure I follow


Complete event model migration for pageNavigation, linkExposure, and socialMediaLinkSharing

In the early versions WebScience, we implemented a measurement and storage model for these modules (i.e., each module ran an individual measurement and stored records of that measurement). We have partially transitioned to a WebExtensions-like event model for these modules, and there are some remaining items to complete.
Remaining items for pageNavigation:

[ ] Implement support for adding and removing multiple pageDataCallback listeners with differing PageDataOptions values.

Remaining items for linkExposure:

[ ] Implement support for adding and removing multiple linkExposureCallback listeners with differing LinkExposureOptions values.
[ ] Integrate untracked link exposures into linkExposureCallback and LinkExposureDetails, rather than using different (and undocumented) callback and detail types for untracked exposures. Note that the meaning of an untracked link will vary by listener, depending on its match patterns.

Remaining items for socialMediaLinkSharing:

[ ] Implement support for adding and removing multiple socialMediaShareCallback listeners with differing SocialMediaShareOptions values.
[ ] Correct the documentation for SocialMediaShareOptions.
[ ] Add documentation for SocialMediaShareDetails.

A stale list of items for socialMediaLinkSharing: https://github.com/citp/news-disinformation-study/issues/64


Clean up tests

Add helpers where able
Make use of shared Mocha behaviors

https://github.com/721labs/partial-common-ownership/issues/14


Comment crashes parser

Description
This YAML file crashes deserializer:
test:
  # comment
  - item: 123

Reproduction steps
std::ifstream ifs("test.yaml");
fkyaml::node root = fkyaml::node::deserialize(ifs);

Expected vs. actual results
The YAML file is valid according to the YAML validator.  The parser shouldn't crash on it.
Minimal code example
No response
Error messages
No response
Compiler and operating system
Linux, clang version 16.0.6
Library version
0.3.1
Validation

[ ] The bug also occurs if the latest version from the develop branch is used.
[ ] I can successfully compile and run the unit tests.

@acfsmith
Thanks for your report! I'll take a look.

@acfsmith
I've fixed the crash in the PR #273.
Can you confirm if the issue is resolved on your environment in the develop branch?

I have checked the latest version.  It doesn't crash anymore.  Thank you very much!

@acfsmith
Thanks for your quick confirmation!
I'll close this issue then.


Selecting 1 required number of referees throws an error

In gitlab by @bjoplin on Aug 23, 2016, 23:02
...stating that you need at least 1 referee.  Selecting 2 required allowed,l however in the digest of facilities states that 1 is required.

In gitlab by @A-Lawrence on Aug 23, 2016, 23:22
Removed ~98471 label

In gitlab by @A-Lawrence on Aug 23, 2016, 23:23
mentioned in commit 5ff1ea5cc4ade742d8f18d94a8f80c27a7675d5a


Support to include to_json and from_json methods within structures (C++)

Hi! Is it possible within quicktype C++ generation to move the from_json and to_json methods from being after all of the defined structures to being contained within each respective structure? The output format I'm looking for is something along the sorts of:
struct Example
{
    std::string key;

    std::string toJSON()
    {
        nhlohmann::json _j;
        _j["key"] = key;
       return _j.dump();
    }

    void fromJSON(const std::string& json)
    {
        nhlohmann::json _j = nhlohman::json::parse(json);
        key = _j["key"].get<std::string>();
    }
}

Essentially, the generalized format I'm aiming for is:
struct NamedType1
{
    <named type properties>
   std::string toJSON() {}
   void fromJSON(const std::string& json) {}
}

struct NamedType2
{
    <named type properties>
   std::string toJSON() {}
   void fromJSON(const std::string& json) {}
}
...

Is this possible? Thanks!

If the main reason is to just have those member functions for convenience, then it should be very easy to keep the current from_json and to_json functions and implement the member functions by calling those.
I don't think that replacing the global functions is possible in non-trivial cases, because they need to be able to call each other to serialize and deserialize their members, and doing that operates on json objects, not on strings.
Also, are you sure you want a member function from_json that overwrites an existing instance with new data?

Having the member function overwrite the existing instance is part of the spec I've been provided, so that's what I have to do. I have managed to implement the member functions by using the from_json and to_json generator function.
How are the global functions called to serialize and deserialize the members? Does the nlohmann JSON get<>() method require the from_json and to_json methods in the nlohmann namespace to return custom-defined structures?

It's been a while since I implemented the C++ target language, but yes, I'm pretty sure that the get requires from_json and to_json.
The easiest thing you can do to get the member functions that you wrote manually to be generated automatically is to modify the emitClass method and just add them there:
    private emitClass = (c: ClassType, className: Name): void => {
        this.emitDescription(this.descriptionForType(c));
        this.emitBlock(["struct ", className], true, () => {
            this.forEachClassProperty(c, "none", (name, jsonName, property) => {
                this.emitDescription(this.descriptionForClassProperty(c, jsonName));
                this.emitLine(
                    this.cppType(
                        property.type,
                        { needsForwardIndirection: true, needsOptionalIndirection: true, inJsonNamespace: false },
                        true
                    ),
                    " ",
                    name,
                    ";"
                );
            });
        });
    };

Yep! It's pretty much just adding the member function additions as the last lines in the emitBlock() that generates the struct, right?

To do things "properly" we'd have to make the C++ target language and renderer classes more friendly for subclassing so that they can be extended like the C# example.

@daichij Correct, it seems that's all you'd have to do.

Awesome! Thanks for the help!
What sort of changes would you make the the C++ target language and renderer classes to make them more friendly for subclassing?

Many of the methods in there that are private should be made protected for a start.  Then at least a subclass of CPlusPlusRenderer could override emitClass.  To make your particular case easier (and others like it) I would move the part inside the emitBlock into its own method emitClassMembers.  Your subclass would then only override emitClassMembers, first calling super, and then emitting the two new member functions.  We'd be happy to accept that as a PR if you're interested.

What all would you look for in that PR? Just the body of that emitBlock to be moved into a protected method, emitClasMembers, and have it replaced by the method call?

That would be one thing.  I think all currently private methods should also be made protected.

Sounds good, I've submitted a PR for that. Let me know what I need to modify!


Add support for a configurable value for max_report_size

Closing since, annotations API cannot handle more than 1MB of data.


My Favourites adds div tag to Description and hyperlink

Category

[x] Bug

Expected or Desired Behavior
In My Favourites, when displaying or editing an item, the Description field should show plain text, and selecting the hyperlink opens the page in the browser.
Observed Behavior
The Description field shows the text inside a div tag.  Example -
Test2

When clicking the link in My Favourites, creates error, likely because link also has div tag.  Example - https://sitename.sharepoint.com/testsub1/SitePages/%3Cdiv%20class=%22ExternalClassEAAE154620D849B3ABCB3EFE9537BF7E%22%3Ehttps://sitename.sharepoint.com/testsub1/SitePages/TestPage1.aspx?loadSPFX=true&debugManifestsFile=https://localhost:4321/temp/manifests.js&customActions={"5c715e3b-fb6f-43d7-a8a7-52bb5db34e50":{"location":"ClientSideExtension.ApplicationCustomizer","properties":{}}}

Steps to Reproduce
Ran all the steps to install My Favourites sample on Windows 7 PC.
Freshly installed all tools for npm and related.
Manually created the Favourites list in SharePoint site.
Ran 'gulp serve' at command line, launching SharePoint site in IE 11 browser.
Used Add to My Favourites to create the item.  Used Show My Favourites to display item.  Saw div tag in Description.  Clicked link and saw page error.

I resolved the issue.  To fix, I edited the list columns for Description and ItemUrl and made sure that the setting "Specify the type of text to allow" was set to "Plain text."  It had defaulted to "Enhanced rich text."
The instructions to manually create the list did not mention that setting.  It said to make it of type "Multiple lines of text" but it did not specify any changes to what type of text.


storm over gearpump drops at-least-once support

The previous solution is flawed and the new idea is illustrated with the graph below.

The previous solution only keeps one pending message, acking the following messages and ack the pending message when the global minAckTime exceed its timestamp. This is incorrect since KafkaSpout will update the store offset to the latest message when the pending message is acked since we already ack the messages behind the pending message.
The new idea is to keep a next pending message when the pending message's timestamp is exceeded by the global minAckTime and only ack the pending message when the next pending message's timestamp is exceeded. Then KafkaSpout will advance the stored offset to that of next pending message. This is safe since we know that messages before next pending message have been processed by the system.
Then, next pending message becomes the new pending message and loops.


mne.datasets should not require tqdm and should not require pooch if datasets are already present

Closes #10022.

[x] Harmonize keyword arguments among all datasets: force_update, update_path, download and verbose are now keyword-only everywhere and not only on a subset.
[x] Remove tqdm dependency for progress bars.
[ ] Suppress progress bars if verbose is set to False.
[ ] Remove pooch strict dependency if datasets are already downloaded.

Keyword only arguments
If the proposal doesn't work for you, I can revert and at least set verbose as keyword-only everywhere.

tqdm
I've set tqdm as a soft dependency. If it is not installed, progress bars are disabled. I think this is simpler than checking for the verbosity level and defining a limit for the progress bars.

pooch
This one is a bit tricky, I will do it later.
The dependency is checked both in fetch_dataset and in _download_mne_dataset.
https://github.com/mscheltienne/mne-python/blob/37356f26cb765d74dc0b5fcf46c78bbbcbf22854/mne/datasets/_fetch.py#L130-L131
https://github.com/mscheltienne/mne-python/blob/37356f26cb765d74dc0b5fcf46c78bbbcbf22854/mne/datasets/utils.py#L151-L152

Have you seen https://github.com/mne-tools/mne-python/pull/10199 and the issue linked there?

I did see the issue earlier, but not the PR! I didn't think you would change it so quickly.
If both pooch and tqdm are included by default with mne, then #10022 is outdated.

Maybe we should make both a soft dependency, not sure.

I think this is simpler than checking for the verbosity level and defining a limit for the progress bars.

Triaging based on level whether or not to show a progress bar was intentional by design. To me if you're in verbose=True (info) mode you should see a bar and if warn or above, you shouldn't. This gives people direct call-by-call level control, rather than just deciding based on whether or not a package (pooch) is present.

I think we need to decide which packages are core dependencies and which are optional dependencies. Fetching data sets should be considered a core functionality, and therefore I'd keep both tqdm (which doesn't have any dependencies itself other than colorama on Windows) and pooch (which does require some packages itself, most notably requests) as required packages.
I have also been thinking about our recurrent discussion on packages that are not pure Python. I don't think this should be a criterium for us to include a package as a dependency or not (otherwise we need to drop NumPy, SciPy, and Matplotlib). It is much more important if a package is actively maintained and provides a broad variety of binary wheels on PyPI.

Triaging based on level whether or not to show a progress bar was intentional by design. To me if you're in verbose=True (info) mode you should see a bar and if warn or above, you shouldn't.

I was going to add it too; but I started with checking if tqdm was installed or not, this way dataset fetching will at least work regardless of tqdm.
I'll wait for the discussion/PRs on core dependencies to be merged before I continue this PR.

Suppress progress bars if verbose is set to False.

I don't think you need to make any changes for this to be the case, it already works this way for me on main -- is it different for you? If so then we're doing something wrong there...

Remove pooch strict dependency if datasets are already downloaded.

Since I'm going to be messing with this in #10199 a bit, I might get to this there. I think your * changes in this PR might still be worthwhile, though, since I think it will help users not do something bad even if it's technically a backward compat break...

I don't think it works as intended. After more testing using main and the sample dataset:

If tqdm is not installed, you can not download a dataset, regardless of verbose. The download should work regardless of tqdm.
mri\brain-neuromag\sets\COR.fif' has a problem and is downloaded/untar in a protected mode on my windows PC. I can not redownload and untar the dataset because a PermissionError is raised.

After deleting the dataset and using sample.data_path(verbose=False), the path is returned, BUT the dataset is not downloaded. But indeed, there is no progress bar displayed.
If both force_update and download are set to True with verbose=False, some funny thing happens. I had both times where I had the progress bar, and others when I didn't (legit, just using up-arrow to re-run the same command in the python terminal..)

To be confirm on macOS and Linux..

For the * change, I know it's backward-incompatible, it's just a proposal. IMO, this change reduces the risk of users making mistakes and is worth the backward-incompatibility. Open for debate 😄

For the * change, I know it's backward-incompatible, it's just a proposal. IMO, this change reduces the risk of users making mistakes and is worth the backward-incompatibility. Open for debate 😄

Here we go! I'm not a big fan of restricting what users are allowed to do in many situations. IMO the only advantage of kw-only args is that devs can freely shift around the order of parameters.
Regarding tqdm, I'd just make it a hard dependency. It's a small Python-only package so why bother trying to download a file without a progress bar?

Regarding tqdm, I'd just make it a hard dependency. It's a small Python-only package so why bother trying to download a file without a progress bar?

Yes, for that if it becomes a core dependency, this problem is solved. For any related change, I will wait for #10199 and eventual follow-ups.

mri\brain-neuromag\sets\COR.fif has a problem and is downloaded/untar in a protected mode on my windows PC. I can not redownload and untar the dataset because a PermissionError is raised.

We should maybe see if our code can force-remove even write-protected files. If it can't, we could update the dataset to have chmod a+w or so on the files... but it would be nice not to have to do this. Can you see if there is some more aggressive way to delete files on Windows? For me it works on macOS despite the strict permissions:
$ ls -al ~/mne_data/MNE-sample-data/subjects/sample/mri/brain-neuromag/sets/COR.fif
-r--r--r--  1 larsoner  staff  16903824 Nov 18  2013 /Users/larsoner/mne_data/MNE-sample-data/subjects/sample/mri/brain-neuromag/sets/COR.fif
(base) bunker:mne-python larsoner$ python -c "import mne; mne.datasets.sample.data_path(force_update=True, verbose=True)"
Downloading file 'MNE-sample-data-processed.tar.gz' from 'https://osf.io/86qa2/download?version=5' to '/Users/larsoner/mne_data'.
  2%|▋                                    | 29.6M/1.65G [00:04<13:22, 2.02MB/s]

If both force_update and download are set to True with verbose=False, some funny thing happens. I had both times where I had the progress bar, and others when I didn't

Do you mean like this as a "did not have progress bar" (second call below)?
$ python -c "import mne; print(mne.datasets.misc.data_path(verbose=True))"
/Users/larsoner/mne_data/MNE-misc-data
$ python -c "import mne; print(mne.datasets.misc.data_path(verbose=True, force_update=True))"
Downloading file 'mne-misc-data-0.23.tar.gz' from 'https://codeload.github.com/mne-tools/mne-misc-data/tar.gz/0.23' to '/Users/larsoner/mne_data'.
0.00B [00:00, ?B/s]
Untarring contents of '/Users/larsoner/mne_data/mne-misc-data-0.23.tar.gz' to '/Users/larsoner/mne_data'
/Users/larsoner/mne_data/MNE-misc-data

If so, there is not much we should do I think -- sometimes GitHub will give you the file size, sometimes it won't (same might be true for osf.io, not sure?). We could make multiple requests but I think just having some updating display here is good enough -- when it was running, I could see the file size actually updating, so it was clear something was happening. And each person doesn't have to call these super often, so I'd say it's a low priority bug at least.

After deleting the dataset and using sample.data_path(verbose=False), the path is returned

This I cannot replicate on macOS, assuming you did something like this:
$ rm -Rf /Users/larsoner/mne_data/MNE-misc-data
(base) bunker:mne-python larsoner$ python -c "import mne; print(mne.datasets.misc.data_path(verbose=True))"
Dataset misc version 0.0 out of date, latest version is 0.23
Downloading file 'mne-misc-data-0.23.tar.gz' from 'https://codeload.github.com/mne-tools/mne-misc-data/tar.gz/0.23' to '/Users/larsoner/mne_data'.
24.9MB [00:08, 5.93MB/s

All we do is look for the existence of the MNE-sample-data directory to decide if the dataset is there or not, so make sure you delete that directory (in addition to its contents).

I spoke too soon -- even on macOS there is a problem with COR.fif:
$ python -c "import mne; mne.datasets.sample.data_path(force_update=True, verbose=True)"
Downloading file 'MNE-sample-data-processed.tar.gz' from 'https://osf.io/86qa2/download?version=5' to '/Users/larsoner/mne_data'.
100%|█████████████████████████████████████| 1.65G/1.65G [00:00<00:00, 5.76TB/s]
Untarring contents of '/Users/larsoner/mne_data/MNE-sample-data-processed.tar.gz' to '/Users/larsoner/mne_data'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<decorator-gen-529>", line 22, in data_path
  File "/Users/larsoner/python/mne-python/mne/datasets/sample/sample.py", line 15, in data_path
    return _download_mne_dataset(
...
  File "/Users/larsoner/opt/miniconda3/lib/python3.8/tarfile.py", line 2178, in makefile
    with bltn_open(targetpath, "wb") as target:
PermissionError: [Errno 13] Permission denied: '/Users/larsoner/mne_data/MNE-sample-data/subjects/sample/mri/brain-neuromag/sets/COR.fif'

But after a chmod a+w ~/mne_data/MNE-sample-data/subjects/sample/mri/brain-neuromag/sets/COR.fif, the subsequent call to data_path(force_update=True, verbose=True) works, so I think we should make this change at the MNE-sample-data end.

COR.fif should be fixed at MNE-sample-data end; but I can look for a more aggressive deletion policy just in case.

Do you mean like this as a "did not have progress bar" (second call below)?

No, I did twice: python -c "import mne; print(mne.datasets.sample.data_path(verbose=True, download=True, force_update=True))"; first call I had just the path, and second call I had a functioning progressbar.
In the first call, I did not get any incomplete verbose from the download; just nothing. Anyway, this is the less important bug here.

After deleting the dataset and using sample.data_path(verbose=False), the path is returned

This I cannot replicate on macOS, assuming you did something like this:

This issue will be fixed when COR.fif is fixed. I got it when I used the same process for a second call to .data_path() after a permission error is raised. I guess a worker/processor from pooch did not close and is causing the issue. (c.f. below, the second call did not download anything; even if the dataset is removed from another terminal).

Also, I am getting progress bars with verbose=False.

This issue will be fixed when COR.fif is fixed.

Pushed a commit to fix this.

Also, I am getting progress bars with verbose=False.

Argh, yes I can replicate this. I think I understand why it's happening -- we used to triage based on the current verbose level, but in the switch to pooch we lost this functionality. It seems like a bug to me that it no longer behaves nicely.
#10199 should be in soon, so feel free to wait for that or continue fixing bugs here and rebase/merge later @mscheltienne !

Let's see if it's green:

Fix MNE-sample-data/subjects/sample/mri/brain-neuromag/sets/COR.fif file.
Fix progressbar display based on verbosity level
Removes soft dependency/imports for pooch

Proposal:

Keyword only argument * for dataset.data_path() functions.
-1 from @cbrnr for now.
At least; I will put verbose as key-word only.

At least; I will put verbose as key-word only.

+1 for this, simple enough (and more backward compatible)

@larsoner Clearly I broke something by removing the _soft_import(); and I don't exactly understand what or what this test is checking for:
=================================== FAILURES ===================================
_____________________________ test_module_nesting ______________________________
mne/tests/test_import_nesting.py:44: in test_module_nesting
    assert code == 0, stdout + stderr
E   AssertionError:
E     Found un-nested import(s) for ['pooch', 'pooch._version', 'pooch.core', 'pooch.downloaders', 'pooch.hashes', 'pooch.processors', 'pooch.utils', 'tqdm', 'tqdm._dist_ver', 'tqdm._monitor', 'tqdm._tqdm_pandas', 'tqdm.cli', 'tqdm.gui', 'tqdm.std', 'tqdm.utils', 'tqdm.version']
E   assert 16 == 0

@larsoner Clearly I broke something by removing the _soft_import(); and I don't exactly understand what or what this test is checking for:

It's testing to make sure module imports are nested. We try not to increase the number of modules we import at import mne time because it makes it take longer and longer. So don't use _soft_import but do nest the import pooch and import tqdm calls.

@larsoner Oh OK, that makes sense. Until now I believed it wasn't a good habit to nest the import in a function, but that's a very good point! I'll fix it.

Thanks @mscheltienne !


path/filepath: ensure Glob() returns in lexical order and comment about that

glob()'s comment mentioned the order.

Why should we specify the order?
It's quite possible that we can speed Glob up by not
sorting the filenames.
If you want a specific ordering, you should sort the
result manually.

I thought the order was intentionally maintained per the comment of glob() (it's called only by Glob). The only other method in the same package that deals with multiple files, filepath.Walk, maintains lexical order too.
If the order is intentionally unspecified, adding a line of comment would be good to avoid callers from depending on the current implementation.

I've had Glob() heavy code which profiles showed the call to Sort() is what was dominating the runtime.    (I switched to a large in-memory index to virtually eliminate the calls to Glob() for a significant performance win.)

I don't think we want to guarantee any particular order. I read glob's comment as specifying the order in which it appends items to the argument, which is not at all guaranteed to be the order that Glob returns. I also don't think we need to, ahem, specify that it is unspecified in a comment. There are many things all over the standard library that have undocumented additional behavior. (See also the recent changes to the implementations of Read in standard library types.)


Common wrapper generator mechanism

This patch adds support for common mechanism to use in different autogenerated wrappers (java and python for now).
OPENCV_MODULE_<module>_WRAPPERS
This variable indicates the list of wrappers which the module supports. To initialize it, one should use new syntax of ocv_add_module and ocv_define_module macros:
ocv_define_module(mymodule REQUIRED opencv_core OPTIONAL opencv_imgproc WRAP java python)

New module folders
Subfolder misc is intended to store all additional files needed by wrapper generator to support this module.

misc/java/src/cpp and misc/java/src/java folders contain manually written files
misc/java/test folder contains tests for this module
misc/java/filelist file contains the manual list of headers to parse for this module

Java tests reorganized
Now tests package is generated from 4 different sources:

modules/java/common_test contains common test files (resources)
modules/java/android_test contains android specific files and tests
modules/java/pure_test contains desktop specific files
modules/<module>/misc/java/test contain tests for specific module
Thus, files are copied from sources 1+2+4 for android and from 1+3+4 for desktop.

Other changes

one java test has been fixed (Imgproc.testArcLength)
obsolete ml java test stubs has been removed
several blocks in java module cmake scripts has been extracted as macros to improve readability

@mshabunin, that's a huge patch! excellent job! let's merge it while it's fresh :+1:

:+1:


Automated cherry pick of #11849

Cherry pick of #11849 on release-7.7.
/cc  @michelengelen
NONE

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.

Trying to auto merge this PR.

Pull Request successfully merged
SHA: 38e074d4bba7aa61dfdfea6b47c82d2e507c4abb