Rationalize test selection

Describe the challenge
Test selection is "novel and exciting" whereas it should instead be boring and straightforward. Let's fix that.
@drewbanin to complete this issue w/ info about what to change and how to change.
See also #2172, #2167

How does  test selection work?
Somewhat surprisingly, test selection is basically identical to model/snapshot/seed/etc selection! When selecting tests, dbt will:

select all nodes based on the supplied selectors
include all tests that reference those selectors
filter for only nodes that match the specified resource types and tags (ie. data or schema tags)

Step two here is really surprising in the general case! dbt will do extra work to find all tests that depend directly on selected nodes only to either:

include the tests and throw away the nodes for dbt test invocations
include the nodes and throw away the tests for dbt run|seed|snapshot invocations

Forgetting the implementation details for a moment, the big takeaway here is that test selection occurs fundamentally at a node-level, and then dbt does one extra hop to find the tests that depend on those selected nodes directly.
This is reflected in the name of the --models flag for dbt test. Fundamentally, we are selecting models to test, not tests themselves!
How could test selection work
Part of this is a naming/UX problem. The --models flag will select from all nodes, not just models! I propose:

The current behavior of --models is remapped to a new --select flag
A --models flag is provided which only runs tests on selected models
--seeds and --snapshots flags are also added to the dbt test command

Beyond that, there are some substantive and wonky thinks about test selection that we should address:
Tags
Selecting schema vs. data tests by tags is a bad idea. We should add a first-class property to test nodes indicating if they are schema tests or data tests, and perform selection based on that property. Test nodes should no longer be supplied with these auto-tags. We could also make these proper selectors instead of CLI args. Coupled with #2167, I think we could replicate the existing functionality in dbt test.
Selectors
dbt should support selecting tests by:

The models that they reference
Their "path" -- that is the path of the source file containing the tests, not the path of the models they reference
The test type (for schema tests). It's probably less likely that someone would want to select all unique tests, but this could make a lot of sense for custom schema tests
The configured test severity

Ideally, these would all be supported via the same system. Take this as a vignette and not a spec, but I'm picturing selectors like:
dbt test --select type:unique
dbt test --select severity:warning
dbt test --select references:my_model

If no selector is provided, dbt would default to the references: selector.
One big consideration is that the graph modifiers + and @ are typically applied after the selector is applied. For example, dbt run --models my_model+ will select my_model as well as all of its descendants. This is pretty insensible in test-land, as something like:
dbt test --select references:my_model+

would, semantically, select all of the tests that reference my_model, as well as all of the descendants of those tests. That's not really the desired behavior here though. Instead, we want to select all of the tests that reference my_model, or a descendant of my_model.
So, what do we do about this?? One "fix" could be inject tests inline into the DAG. Picture something like:
+-----------+    +-----------+    +------------+
| my_model  +--->| my_test   +--->| other_model|
+-----------+    +-----------+    +------------+

A key benefit here is that a selector like
dbt test --select references:my_model+

would "just work". In this example, tests applied to other_model would indeed be descendants of my_test. While this would be more internally consistent, I don't think it's a justifiable change for that reason alone. Instead, the much larger benefit here would be that we can (finally) combine dbt run and dbt test into a single command which runs models and tests in DAG-order. There is a lot more to consider there, but I do want to call out that it would be a good and helpful change of behavior for other reasons not totally related to test selection.
I think that our ultimate answer to test selection must include at least one of the following:

A big change to how tests are embedded into the DAG (shown above)
A realization that tests are special, and acceptance of the fact that dbt must make them a special case (the "do-nothing" approach)
A verbose and user-unfriendly syntax for specifying test selectors

An example of a user-unfriendly test selector which is unambiguous and consistent:
# select tests on my_model, or descendants of my_model
dbt test --select references:(my_model+)

I think this syntax is decidedly worse than the current implementation dbt test --models my_model+, but I could be convinced that it's not so user-unfriendly, and I bet we could add some sugar for more typical use-cases.
There's certainly more thinking to do here, but I wanted to open up this line of inquiry to discussion before going further down the rabbit hole. @beckjake, @jtcohen6, what say you?

On its face, without a full appreciation for how hard it would be to implement, I support the change to how tests participate in the DAG. Most implications of this feel intuitive:

Each schema test is a child of the model on which it's configured
Each data test is a child of the models it references (potentially one or many)

N.B. There are some schema tests (e.g. relationship, equality) which reference other models. Those are more like data tests in this sense.

If model_a is a parent to model_b, a schema test on model_a is also a parent to model_b:

+-----------+    +-----------+    +------------+
| model_a   +--->| unique    +--->| model_b |
+-----------+    +-----------+    +------------+

A hypothetical dbt build --select model_a model_b would execute steps in the following order:

Materialize model_a
Run the schema test unique_model_a
Materialize model_b

If the schema test failed, it would skip all downstream nodes, including model_b.
Some implications feel a little less intuitive.
In the same scenario above, let's say a data test references both model_a and model_b. If we have a model_c that depends on model_a, I suppose model_c would have to be a child of the data test, and thereby also a child of model_b. That means the build order for the DAG of all nodes would be model_a --> model_b --> data test --> model_c:
+-----------+    +-----------+    +------------+    +-----------+
| model_a   +--->| model_b   +--->| data_test  +--->| model_c   |
+-----------+    +-----------+    +------------+    +-----------+
      |                                 ^
      |                                 |
      +---------------------------------+

But the build order for a DAG that includes only model nodes could run model_b and model_c in parallel:
+-----------+    +-----------+
| model_a   +--->| model_b   +
+-----------+    +-----------+
      |
      |          +-----------+
      +--------->| model_c   |
                 +-----------+

@jtcohen6 I think your last example actually reflects the desired behavior!
+-----------+    +-----------+    +------------+    +-----------+
| model_a   +--->| model_b   +--->| data_test  +--->| model_c   |
+-----------+    +-----------+    +------------+    +-----------+
      |                                 ^
      |                                 |
      +---------------------------------+

If data_test fails here, we'd indeed want to skip running model_c in the sort of default/general case if we're doing a combined run + test. I also think that if we build a subdag that only includes models, dbt would automatically generate the ideal DAG:
+-----------+    +-----------+
| model_a   +--->| model_b   +
+-----------+    +-----------+
      |
      |          +-----------+
      +--------->| model_c   |
                 +-----------+

We do this by building a transitive closure over the DAG and then pruning nodes that we want to exclude from the run.
Some other thoughts around edge-cases:

I think ephemeral models would work just fine in this approach. They would get baked into the DAG, but more or less "skipped" at run-time. Tests on the ephemeral model would run before models that depend on the ephemeral models.
We could totally make the build failure behavior configurable. By default, nodes will skip their descendants if they fail to build, but we can pretty easily turn that on/off for tests

@beckjake curious what you think about all of this

I really don't love the idea of tests being inline in the DAG. I agree that it solves a number of problems around dbt run+test, but it makes the oft-requested #2052 behavior really hard to implement. If someone says dbt ls --select model_a+1, does that mean we only select model_a's tests? Even worse, does that mean that if you add tests to a model that didn't previously have them, the semantics of model_a+1 changes from model_b to model_a's tests? That sounds really undesirable!
I think a more compelling thing would be to note tests as "special children", and in the dbt run+test case we just require running all the test nodes of a given node before marking it "done". Basically, treat node + its direct test descendants as a single node in the graph. To handle the case @jtcohen6 described, I'd say that model_a -> model_b -> data_test would get treated as a "single node" by the part of the code that manages the thread pool. Since there's a strict dependency ordering enforced anyway, it's not big deal to force all three of those nodes to be handled by the same thread.
That's a bit more convoluted internally as we'd have to rework some of the "node done" completeness stuff, but I think it will result in a more intuitive/consistent outcome. This would be an opportunity to revisit how dbt handles model/job scheduling, and potentially make it less opaque.

Ok - i think that's fair - direct child selectors are a really good consideration! I'm happy for tests to remain special -- it just sounds like we need to make them more special than they currently are.
So, is the goal for this issue to change the current implementation of how tests are selected out of the DAG? While I'm in favor of the change you're describing, I don't think this will necessarily have any semantics of the command line interface for selecting tests!

Sorry for the essay, this took me a long time to think through and write about "intelligently".
I guess I've become convinced that tests are "special", but I do believe their specialness should get changed.
Let me lay out my reasoning here. In broad strokes, these are what I see as a representative sample of the key considerations:

Someday we want to implement dbt run+test, which I will summarize as: During run+test, test nodes must run after all the nodes they depend upon are finished, but before any nodes they don't depend upon.
dbt test --select my_model should work basically like it does now: everything that tests my_model is run.
we want users to be able to select more fine-grained slices of tests than just "tests for this model" when they're using dbt test - "all uniqueness tests"
We want to someday implement a parameter for model selection like #2052. An assertion on my part: This parameter applying to tests themselves is not really very valuable.
You should be able to do something like dbt test --select source:my_source.* and have it only run your source tests. Bonus: Being able to say "only run any tests that whose requirements are exlclusively in the selected set". The benefit there would be skipping tests that assert relationships between sources and refs, for example. Maybe this is more of an "advanced node selection syntax" problem.
dbt ls should reliably be able to tell me what nodes will be run (bonus: in what order!) for all of dbt run, dbt test, dbt run+test. Requiring parameters and type restrictions is acceptable (so dbt ls --resource-type=test is a fine way of saying whatever "dbt test" will do).
It should be possible to implement #2172 and #2167 in any new design
It would be nice if there was a simple, consistent explanation of how selecting your nodes worked. It would be ok (though less great) if that explanation started with "it depends if you're running or testing or both".

Thinking about all that, it seems to me that node selection should work similar to how it currently does, except it should be more context-dependent. I'd like to conceptually split it into two parts:

"node selection". I'm keeping this name!
"task-based graph creation". Horrible name!

"Node selection" would behave exactly like node selection today (barring any changes in #2172 and #2167), except:

it would not do the collected.update(tests) it currently does
it would not do an extra adding-in of ephemeral nodes.
That means that in the most common case node selection wouldn't pick up any tests. Node selection in this case returns the set of selected unique IDs.

"task-based graph creation" would be implemented per-task.

"dbt test" would find all the test descendants of the selected nodes that descended from all the selected nodes, and return those.

there would be a flag here for exclusivity, probably (filter out tests that descend from any models not selected)
note that the graph dbt test generates has no edges, because tests can't depend upon one another!

"dbt compile" / "dbt run" / "dbt snapshot" would create a graph the same way they currently do.
"dbt run+test" would create a graph like I describe below. I'm not sure this is exactly the transitive closure as there are some edges that don't exist in the regular transitive closure... see below

The wrinkle in my eyes is ephemerals. I know this is pretty vague and hand-wavy, but we should either lazily compile them or compute them up-front and rip through compiling them all immediately before we even start running. Either way, ephemeral nodes shouldn't be special-cased in node selection.
implementing dbt run+test's graph creation
Instead of the usual graph queue, you'd use a special one that inserts tests into the graph like this. Imagine you've selected nodes A, B, C, and D from the graph with your query, and constitute a subgraph that looks like this:
A -> B ->D
|        ^
---> C --|

And then you have tests on:
A: T(A)
B: T(B)
C: T(C)
A and B: T(A, B)
A and C: T(A, C)
A and D: T(A, D)
B and C: T(B, C)
B and D: T(B, D)
C and D: T(C, D)
A -> T(A) -> B -> (T(B), T(A, B)) -> T(B, C) -> D -> (T(A, D), T(B, D), T(C, D))
      |                                 ^
      -----> C--> (T(C), T(A, C)) ------|

I'm sure there's a fancy graph theory word for this, too!

Thanks for this writeup @beckjake - it's very helpful!!

"task-based graph creation" would be implemented per-task.

Yep, I totally buy this. Let's call these things:

node selection
DAG creation

They are very related, but I agree, they're different tasks and we should not conflate the two!

The wrinkle in my eyes is ephemerals. I know this is pretty vague and hand-wavy, but we should either lazily compile them or compute them up-front and rip through compiling them all immediately before we even start running. Either way, ephemeral nodes shouldn't be special-cased in node selection.

I agree it would be great to not special case ephemerals in node selection. I'm open to either of the approaches you've outlined here.
Last, the run+test DAG creation diagram you shared makes sense to me. I'll get @jtcohen6 to read some books on graph theory so that we can pick an appropriate name for this construct :)

@beckjake to open a new issue to split out node selection from DAG creation

great discussion! Closing in favor of https://github.com/fishtown-analytics/dbt/issues/2328


Disable already selected items

How do I lock/disable selected items in Multiple tag mode?
Just like hide-selected hide the items, I would like them to be visible but non-removable.

Use option slot and set the custom style to the option you want, it will be like disabled.
pointer-events: none;
opacity: 0.6;

@lehoaibaokg This only covers mouse events and won't prevent users from selecting options via keyboard arrow keys. Use undocumented $isDisabled: true on option object: https://github.com/shentao/vue-multiselect/issues/176#issuecomment-307638151


Explore Python bindings for Godot and godot_voxel

Godot Python: «The goal of this project is to provide Python language support as a scripting module for the Godot game engine.»
What kind of support do we have?

GDNative interface to use Python 3 as alternative scripting language in Godot

Why do we want Python vs GDScript for scripting? Because we can reuse the full Python ecosystem.
One of the goal of this ticket is to check that we can access the godot_voxel API from Python also (so we can create the McThings renderer for Godot). One of the first goals will be to replicate the one vovel creation using godot_voxel #11 but from Python scripting.

Beta release of Python support

So we can create a script in Python that it is called from McThings to render scenes in Godot?
Not, the approach is that the Python script is called from Godot. From this script we can call McThings to build the scene and render it in Godot using the godot_voxel API, which should be accesible from Python.

There are cool videos about Python Godot support. Probably the first one was this one in EuroPython: https://www.youtube.com/watch?v=SK0BtxHROdg

No more plans to use Python in Godot


Add -fetchRepositoryReadme:reference: method to `OCTClient+Repositorie...

...s` category.

Thanks! Just one note. :pencil2:

Of course, I have fixed it.

:shipit:


[Gally]: master <- dev

Automatically created by Git-Ally

:tada: This PR is included in version 1.17.11 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add SubSquare integration proposal

A clean governance platform is needed for SORA and I do like Subsquares UI/UX. I  suggest to add any roadblocks whether technical or not.

A clean governance platform is needed for SORA and I do like Subsquares UI/UX. I suggest to add any roadblocks whether technical or not.

Thanks, tried to add 2 items at ddfa8d067463b52fbc0f9a2edfebf287eabd405c.

We definitly need a tool like this. It definitly has better UI and better timeline then polkaassembly. Its better then forking polkaassembly for my opinion. I suggest we make a remark proposal and see if community wants it. After that Subsquare team can confirm is it possible to implement Sora or not.

There are some blocks current polkadot.js + sora types can not decode correctly. See block #412.

After that Subsquare team can confirm is it possible to implement Sora or not.

@Nolan730 I think we can still implement SORA though the upper decode problems, but I hope there are developers who can solve the decode problems.

I assume some types changed over time and you get error because of it. Probably you should use different types for older blocks and i think Subscan handles it like that.  Subsquare team should be able to handle this too

I assume some types changed over time and you get error because of it. Probably you should use different types for older blocks and i think Subscan handles it like that. Subsquare team should be able to handle this too

Hi, as I replied in the telegram, It will be perfect if SORA developers can make tweak the types file to make the polkadot apps show the history blocks correctly. All external apps/external developers will benefit a lot from this work.

I think sora versioned types maybe misconfigured, and we can correct this to make all history blocks can be decoded. How can I contact the developers, so maybe we can co-work to solve this?

I think sora versioned types maybe misconfigured, and we can correct this to make all history blocks can be decoded. How can I contact the developers, so maybe we can co-work to solve this?

Please message https://t.me/soradevs


The following assertion was thrown building _FocusMarker(dirty):

I don't know what happened. After suspending the APP in the background for a long time, I entered the APP again. After hot loading, the console prompted the following information. I am confused
I/flutter (29136): The following assertion was thrown building _FocusMarker(dirty):
I/flutter (29136): 'package:flutter/src/widgets/framework.dart': Failed assertion: line 4838 pos 14: '() {
I/flutter (29136):         // check that it really is our descendant
I/flutter (29136):         Element ancestor = dependent._parent;
I/flutter (29136):         while (ancestor != this && ancestor != null)
I/flutter (29136):           ancestor = ancestor._parent;
I/flutter (29136):         return ancestor == this;
I/flutter (29136):       }()': is not true.
I/flutter (29136):
I/flutter (29136): Either the assertion indicates an error in the framework itself, or we should provide substantially
I/flutter (29136): more information in this error message to help you determine and fix the underlying cause.
I/flutter (29136): In either case, please report this assertion by filing a bug on GitHub:
I/flutter (29136):   https://github.com/flutter/flutter/issues/new?template=BUG.md
I/flutter (29136):
I/flutter (29136): The relevant error-causing widget was:
I/flutter (29136):   MaterialApp file:///E:/graduationDesign/project/app_mobile/bju_information_app/lib/main.dart:41:20

Hi @H-Mongo
can you reproduce it?
would be helpful if you could post a complete minimal reproducible code sample together with your flutter doctor -v

Now that abnormal information has been closed. The reason for the closure is not clear. thank you for your help!


Bot Configuration file format could benefit from a JSON schema to light up intellisense and other tooling

Tool
Name: Msbot
Describe the problem
While there is some light documentation/samples of the .bot file format, you still are pretty much editing raw JSON when it comes down to it unless you stick to using the msbot CLI.
Describe the solution you'd like
Create a JSON schema for the .bot configuration file format to help developers who may prefer to hand edit the files in development environments that understand JSON Schema such as VS Code or VS full.
Describe alternatives you've considered
Just edit raw JSON and refer to docs/samples for schema.
[enhancement]

with the de-emphasis of .bot files, I don't see us addressing this.


Test results errored

Summary
This change updates the reporters API so test status can be 'failure' or 'error'.
A failure classification would be for assertions and known errors. An error classification would be if a test failed to run or for uncaught/unexpected errors.
Test plan
Existing tests pass.
Looking for feedback on how to properly test this with automated tests. I think a reporter test will work if we can have a jest-circus only reporter test.

Interesting change! One thing to note, this couples a user to only being able to use expect as their assertion library.
Currently you can use anything that throws, I.e. chai, and it will correctly report the test as failed.
I like the idea behind this of distinguishing between errored tests and failures, I’m just not sure how best to separate the two 🤔

GitHub doesn't allow me to leave comments...

will try again later

Agreed on all fronts.
I'm not 100% sure the best way to maintain a list of known failures (chai throw, node assert, etc). Any suggestions? Basically if there is an alternative to instanceof check that will give the same level of confidence.
One thing I am considering is adding something to the jest object for this - something like:
jest.failureTypes = [Error classes]
So then we could push a few things there and provide a mechanism in user-land to push additional assertion throwables into.
I'll continue looking at this today and try to get the tests passing. "yarn test" passed locally for me.

Seems like chai uses assertion-error: https://github.com/chaijs/chai/blob/26c8a794ec6da695f004bdbd6362a466dc3d098f/lib/chai.js#L19 (interesting choice to call the module a "standard" when chai itself has more weekly downloads... But whatever). We might be able to detect that? Without adding an explicit dependency on the module

@SimenB there are other assertion libs out there (other than chai) that people may want to use too - so maybe it would be better to give a way of plugging Failure Errors into Jest? 🤷♂

@SimenB I like that idea!

@palmerj3 thoughts on this? 😀

@SimenB I'll close and reopen in the future


transfer app support multi select

Is your feature request related to a problem? Please describe.
like title
Describe the solution you'd like
can multi select app
Describe alternatives you've considered
No response
Additional context
No response

Also if possible add in Select All tick


Fixed secret type of synced secret

This fixes #233  #176

Codecov Report

Merging #277 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #277   +/-   ##
======================================
  Coverage    2.16%   2.16%
======================================
  Files           5       5
  Lines         462     462
======================================
  Hits           10      10
  Misses        451     451
  Partials        1       1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3334eb9...455c7fd. Read the comment docs.


Include element in "Equipment" for the uncertainty evaluation procedure

Include element in "Equipment" for the uncertainty evaluation procedure.

Moved to https://github.com/wmo-im/wmdr/issues/16


Publish Atlas to Maven

It will be easier manage the dependency of Atlas via Maven. Please publish Atlas to Maven.

Atlas is now published to maven as of 0.2.10


BUG: CSRF Cookie Generated

The CSRF Cookie that is Generated and Stored are Unmasked Token, instead of Masked Token.
Yii2 Request Class validateCsrfToken() function does an Unmasked function on the token and compares.
Since the CSRF Cookie is already stored as Unmasked, it runs the unmasked function again on the value of the CSRF Cookie when the validateCsrfToken is executed, resulting in the value to be corrupted and not the same. Hence submission cannot be verified.
Below is what i've updated in Request.php for a quick fix:
public function getCsrfToken($regenerate = false)
{
if ($this->_csrfToken === null || $regenerate) {
$token = $this->loadCsrfToken();
if ($regenerate || empty($token)) {
$token = $this->generateCsrfToken();
}
//$this->_csrfToken = Yii::$app->security->maskToken($token);
$this->_csrfToken = $token;
}
return $this->_csrfToken;

}
protected function generateCsrfToken()
{
//$token = Yii::$app->getSecurity()->generateRandomString();
$token = Yii::$app->security->maskToken(Yii::$app->getSecurity()->generateRandomString());
if ($this->enableCsrfCookie) {
$cookie = $this->createCsrfCookie($token);
Yii::$app->getResponse()->getCookies()->add($cookie);
} else {
Yii::$app->getSession()->set($this->csrfParam, $token);
}
return $token;

}
What steps will reproduce the problem?
Validating CSRF Cookie.

Retrieve the CSRF Cookie Value
Run the Value Check with Request::validateCsrfToken()

What is the expected result?
Expected Result to Return As Verified Submission
What do you get instead?
Result Returned as Not Verified Submission
Additional info
| Yii version      | 2.0.49
| PHP version: 8.x |
| Operating system: Linux |

This is weird, I don't remember changes done there recently and we did not hear about problems with that functionality. Could you provide a quick way to reproduce your issue on a clean application?

Closing because of no response.


Compiler crash on unqualified import for unexposed value

To reproduce, go to the commit before 3ee8a1c306317e42ac0701b2f5922d7677549581 (linking to that commit because it illustrates the fix) and run:
cargo run -- check examples/parser/examples/parse-movies-csv.roc

The problem is that nat is not exposed at all from that module.
This should result in a nice error, not a compiler crash!

@agu-z is this reproducible in your new import branch?


Empty user PATH causes scoop install to throw

I bumped into this yesterday. When scoop is installed on a brand new virtual machine image, it turns out that the emptiness of the user's PATH variable causes $path.split(';') to complain:
PS> [environment]::getEnvironmentVariable('path', 'User')
PS> [environment]::getEnvironmentVariable('path', 'User').split(';')
You cannot call a method on a null-valued expression.
At line:1 char:1
+ [environment]::getEnvironmentVariable('path', 'User').split(';')
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
etc

This shot me like this: (when I was testing my install scripts on a fresh windows 10 vm image)
PS> scoop install 7zip
installing 7zip (9.38)
checking hash...ok
extracting...done
% : You cannot call a method on a null-valued expression.
At .......etc.

It turns out that the culprit was here:
function find_dir_or_subdir($path, $dir) {
    $dir = $dir.trimend('\')
    $fixed = @()
    $removed = @()
    $path.split(';') | % {
        if($_) {

A possible, untested, quickfix for this situation:
 function env($name,$global,$val='__get') {
     $target = 'User'; if($global) {$target = 'Machine'}
+    if($val -eq '__get') { [string][environment]::getEnvironmentVariable($name,$target) }
-    if($val -eq '__get') { [environment]::getEnvironmentVariable($name,$target) }
     else { [environment]::setEnvironmentVariable($name,$val,$target) }
 }

Thoughts?

Second thought: this shouldn't happen normally because scoop adds the shim to the user path anyway. It's only happening to me because I removed the shim from the path after installing scoop (in the spirit of portability).


Portfolio final setup and mobile

Added section in portfolio.

creating CSS part for the new HTML added sections.
-New sections added to the Html file:
"Works" section
"About myself" section

[ ] I apologize for this repetition, kindly consider the comment in the first summary as [OPTIONAL]


Add progress bar and make runnable

Create some runnable module that performs the crawling operation.
Currently, there is a test function that does this, but it feels
awkward to run a pytest to do work.

Implement some kind of progress meter when crawling.

Closed by 0d75a6582ba09a5c8e59cae53913b0fcb72e55f6


[$500] Chat - In bracket() email entered $ is shown like URL link

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join
Slack channel!

Version Number: 1.3.87-1
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Expected Result:

Actual Result:

Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos

Android: Native

https://github.com/Expensify/App/assets/78819774/95a5da0b-04f7-4a6e-ba7a-6878e85a0f3a

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01b86aa5e0856bfe6c
Upwork Job ID:<PHONE_NUMBER>733097472
Last Price Increase: 2023-10-19

This is not a bug as we support alias text for email as well/

closing


Adding "Use offer" tool into DC's sidekick.

Vicky Morel needs Use offer tool for dc
https://workfront.slack.com/archives/C03B0BUA75E/p1701735683207409

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (f095a63) 99.40% compared to head (d3ae040) 99.44%.

Additional details and impacted files
@@            Coverage Diff             @@
##            stage     #495      +/-   ##
==========================================
+ Coverage   99.40%   99.44%   +0.03%
==========================================
  Files          26       26
  Lines        2878     2878
==========================================
+ Hits         2861     2862       +1
+ Misses         17       16       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


New API docs

API doc generation no longer works after #8345. We need to find a new tool chain to generate API docs.
Possible options:

A while ago, @tschaub and I started an effort in the site branch to use the output from tasks/generate-info.js as starting point for a Gatsby generated API documentation. This turns out to be a huge effort, almost equivalent to writing an API doc generator from scratch.

Investigate tools like ESDoc, TypeDoc or not-yet-available tools that may come out of Microsoft's TSDoc effort.

Comments and thoughts on these options, or ideas for other options, are much appreciated.

I have investigated TypeDoc a bit. I did not find a way to make it work with *.js files. Maybe it works if we are able to generate d.ts files? If so, how can we create those?

I am currently investigating ESDoc. Will report here how that goes, but currently I'm stuck with Cannot read property 'charAt' of undefined errors.

Actually there is hope that we can get things to work with TypeDoc. See TypeStrong/typedoc#643.

I am currently working on making typedoc work with typescript@next: TypeStrong/typedoc#824

Looks like even with the above pull request, TypeDoc does not work properly with @typedef and @enum annotations.

I am trying to make typedoc output something but I only get an empty documentation page:
git checkout ts
npm link typedoc  # to get latest version and I use typescript@next there
npx typedoc --out tsdoc src

@ahocevar, do you have a branch somewhere so I have a look to how you did it?

@gberaudo See https://github.com/openlayers/closure-es-modules/ for an example project. You need to set allowJs: 'true' in tsconfig.json.

Thanks for your input @ahocevar, it helped a lot.
Strangely my issue was coming from the way I invoked typedoc.
I was passing the source directory like this:
npx typedoc --out tsdoc src  # matches the documentation but produces empty output

When omitting the source directory it does something (it is specified in tsconfig.json anyway):
npx typedoc --out tsdoc  # does something

In the end, after way too much time spent trying to make TSDoc work for us, I went with JSDoc again. See #8847.

Have you ever considered using https://documentation.js.org ? I didn't check the features / configurability but we have been using in some smaller projects and we are quite happy with it.

@marcjansen I have, but it does not work well with modules, and not at all with TypeScript types unfortunately.


Automated cherry pick of #6403

Cherry pick of #6403 on release-5.27.
/cc  @fmunshi

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.


[CSL-458] Delegation: MPC threshold support

@georgeee

Implement possibility to store "permanent" certificate (or PSK) on chain

Stakeholder should be able to post only one permanent certificate per epoch
Certificates should be distributed via Inv/Req/Dat mechanism, which is to be abstracted under CSL-351
Only last delegation posted to stake is considered in MPC, FTS
Delegated stake should be not less than 0.01% of stake
No more than 1000 certificates should be put into block

@gromak
It's called heavyweight delegation.

@gromak

0.01%
1000
Put it into Constants.hs please (cap).


Preemptively close chunks before big messages to isolate them

Changelog
Close chunks when big messages are written to isolate them, leading to much better read performance when going over small-message channels since those isolated chunks can be skipped over entirely.
Docs
None
Description
A very common situation is to have a few topics with very large messages (images, point clouds, etc.) and many topics with very small (and frequent) messages (state, transforms, statuses, etc.). When writing all those into a single ("merged") mcap file, it naturally leads to chunks typically containing a series of small messages and ending with one big message which fills the rest of the chunk and "closes" it. The result of this is that nearly all chunks are "mixed" (i.e., messages from all channels appear in each chunk). During playback / reading, some of the client code (e.g., C++) implement logic to skip over chunks that don't have any message of interest (topics being read). That logic is completely thwarted if most chunks are mixed in this pathological pattern.
So, this PR adds a very simple change. When an incoming message is very large (larger than the chunk size), instead of writing that message to the current chunk (mixing with prior smaller messages), it closes the current chunk and writes the big message to a fresh new chunk (which will be closed immediately after). The end result is that big-message channels get isolated into their own series of chunks, enabling the playback / readers to skip over them without loading those chunks to RAM or decompressing them.
In theory (but I have not seen this), compression could also perform better on the isolated channels, where it matters most. All I have seen (see below) is that incompressible data (such a protobuf wireformat) does not get uselessly compressed along with the larger "bytes" data (images, point-clouds, etc.). This does not affect the resulting file size but speeds up reading times. (BTW, it would be nice to be able to configure the hardcoded to 1KB minimum compression size in the writer)
The only slight drawback that I can see is getting some smaller / partial chunks. I guess there might be some rather unrealistic pathological cases (like interleaving every big and small message) where this could lead to a noticeable difference in file size (bigger chunk index). If you want a writer option to disable that behavior, that's fine, but I believe this should be the default.
I ran some tests on some simple mcap files with mixed channels, the result are seen below.
BeforeAfter
Running mcap-cli info on the recorded mcap file (2.7 GB):
library:   libmcap 1.4.0
profile:
messages:  31870
duration:  3m50.316691925s
start:     2024-08-08T13:03:44.098069201-04:00<PHONE_NUMBER>.098069201)
end:       2024-08-08T13:07:34.414761126-04:00<PHONE_NUMBER>.414761126)
compression:
    zstd: [6724/6725 chunks] [3.30 GiB/2.52 GiB (23.58%)] [11.21 MiB/sec]
    : [1/6725 chunks] [516.00 B/516.00 B (0.00%)] [2.00 B/sec]
channels:
    (1) [redacted_topic_0]           2121 msgs (9.21 Hz)    : foxglove.Grid [protobuf]
    (2) [redacted_topic_1]           2121 msgs (9.21 Hz)    : [redacted] [protobuf]
    (3) [redacted_topic_2]           2297 msgs (9.97 Hz)    : foxglove.PointCloud [protobuf]
    (4) [redacted_topic_3]           2297 msgs (9.97 Hz)    : [redacted] [protobuf]
    (5) [redacted_topic_4]          11517 msgs (50.01 Hz)   : foxglove.FrameTransforms [protobuf]
    (6) [redacted_topic_5]          11517 msgs (50.01 Hz)   : [redacted] [protobuf]
attachments: 0
metadata: 1

Running time mcap-cli cat --topic [redacted_topic_4] on the recorded mcap file (2.7 GB):
real  0m3.063s
user  0m4.179s
sys   0m0.662s

Running mcap-cli info on a recorded mcap file (2.7 GB):
library:   libmcap 1.4.0
profile:
messages:  31870
duration:  3m50.316691925s
start:     2024-08-08T13:03:44.098069201-04:00<PHONE_NUMBER>.098069201)
end:       2024-08-08T13:07:34.414761126-04:00<PHONE_NUMBER>.414761126)
compression:
    : [2121/8839 chunks] [276.02 KiB/276.02 KiB (0.00%)] [1.20 KiB/sec]
    zstd: [6718/8839 chunks] [3.30 GiB/2.52 GiB (23.58%)] [11.21 MiB/sec]
channels:
    (1) [redacted_topic_2]           2297 msgs (9.97 Hz)    : foxglove.PointCloud [protobuf]
    (2) [redacted_topic_3]           2297 msgs (9.97 Hz)    : [redacted] [protobuf]
    (3) [redacted_topic_4]          11517 msgs (50.01 Hz)   : foxglove.FrameTransforms [protobuf]
    (4) [redacted_topic_5]          11517 msgs (50.01 Hz)   : [redacted] [protobuf]
    (5) [redacted_topic_0]           2121 msgs (9.21 Hz)    : foxglove.Grid [protobuf]
    (6) [redacted_topic_1]           2121 msgs (9.21 Hz)    : [redacted] [protobuf]
attachments: 0
metadata: 1

Running time mcap-cli cat --topic [redacted_topic_4] on the recorded mcap file (2.7 GB):
real  0m0.070s
user  0m0.059s
sys   0m0.024s

I think this makes sense and is a good improvement for most users. I'm inclined to believe it should be possible to disable this behavior with a writer option. I also think it changes the documented behavior of the chunkSize writer option, so that docstring should be updated. What do you think?

Sounds good, I updated the comment for the chunkSize option and added a "noHugeMessageChunk" option to disable this behavior if needed.

This makes me wonder if we should instead close any chunk when writing a message if the current chunk size + new message size > chunkSize? That is, why check if a message itself is larger than a whole chunk, rather than just closing chunks when they are about to become larger chunkSize (rather than when they are already larger than chunkSize)?

FYI to @jhurliman in case you have any opinions on this change.

FYI to @jhurliman in case you have any opinions on this change.

This seems like good default behavior, I’m not sure it needs an configurable option?


feature req: for version 1.2.3 --> show all control addresses and what they are used for

version 1.2.3 should include the new control address feature.
would be nice to have them in the dashboard as soon as possible!

changes to control addresses are merged into master and working (i cannot say that about the latest master itself :D)

I will see what I can do

it's not implemented as you describe but you should be able to see your adresses in the dashboard and give them specific names. Could you confirm it answers your request.

I didn't find the link in the API to attach a contro address to an usecase.
So what I suggest is to use the address.toml  file to set the name accordingly.


Enable recommended eslint checks and fix resulting errors

This triggers a few errors for unused variables, trailing commas in
function arguments, but most importantly a missing yield in
core/sagas/inputs.js (see issue #151). Tone down that error into a
warning for the time being, until this gets fixed properly.

I stumbled over these when trying to build welder-web in Cockpit. That has these eslint errors on, so the build failed due to these.

Codecov Report

Merging #152 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #152   +/-   ##
=======================================
  Coverage   41.87%   41.87%
=======================================
  Files          73       73
  Lines        2252     2252
  Branches      296      296
=======================================
  Hits          943      943
  Misses       1114     1114
  Partials      195      195

Impacted Files
Coverage Δ

core/sagas/inputs.js
39.13% <ø> (ø)
:arrow_up:

core/constants.js
100% <ø> (ø)
:arrow_up:

core/utils.js
60% <ø> (ø)
:arrow_up:

core/store.js
75% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87dbdda...2cf89a5. Read the comment docs.

@henrywang or @jkozol, do you have any comments for these changes?


About plans and subscriptions

About plans and subscriptions
Is tables are created built in the central database? Or in the tenant database?
As well Financial transactions!
How shopify.com works
Please explain how you can manage tenant subscription plans!

It's up to you what data you put into the central & tenant database.
I don't see a single reason why you'd want to store this in the tenant database so I'd put this into the central database.


yt-dlp stops working after being compiled with pyinstaller

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
I am trying to compile a python program with the yt-dlp API, using pyinstaller.
The program works perfectly before being compiled but it does not work after being compiled.
yt-dlp has trouble accessing every website that i have tried it on, and keeps timing out.
The code is very short and simple because i was running a test to see if yt-dlp would work after being compiled with pyinstaller
Here is my code:
`from yt_dlp import YoutubeDL
URLS = ['https://www.youtube.com/watch?v=zKhDNNvoBEg']
yt_opts = {
'verbose': True
}

with YoutubeDL(yt_opts) as ydl:
ydl.download(URLS)
`
I have tried to run this with the newest version of yt-dlp and the newest versions of the nightly and master builds.
I have also saved the output of pyinstaller if that helps:
pyinstaller master build.txt
pyinstaller nightly build.txt
pyinstaller normal build.txt
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
My output for the normal build:

[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp [615a84447] (pip) API
[debug] params: {'verbose': True, 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.3 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev (setts), ffprobe 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.40.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1803 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=zKhDNNvoBEg
[youtube] zKhDNNvoBEg: Downloading webpage
WARNING: [youtube] Unable to download webpage: (<urllib3.connection.HTTPSConnection object at 0x000002626C760510>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C762710>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C764110>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C761610>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (3/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] Unable to download API page: (<urllib3.connection.HTTPSConnection object at 0x000002626C765F10>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)') (caused by TransportError("(<urllib3.connection.HTTPSConnection object at 0x000002626C765F10>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')"))
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C766C90>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C768210>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C761B10>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (3/3)...
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] Unable to download API page: (<urllib3.connection.HTTPSConnection object at 0x000002626C76A310>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)') (caused by TransportError("(<urllib3.connection.HTTPSConnection object at 0x000002626C76A310>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')"))
[youtube] zKhDNNvoBEg: Downloading iframe API JS
WARNING: [youtube] Unable to download webpage: (<urllib3.connection.HTTPSConnection object at 0x000002626C761D90>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')
[youtube] zKhDNNvoBEg: Downloading web player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C7646D0>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading web player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002626C7637D0>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading web player API JSON

My output for the master build:

[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-master-builds [32abfb00b] (win_exe) API
[debug] params: {'verbose': True, 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.3 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev (setts), ffprobe 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.40.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1807 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=zKhDNNvoBEg
[youtube] zKhDNNvoBEg: Downloading webpage
WARNING: [youtube] Unable to download webpage: (<urllib3.connection.HTTPSConnection object at 0x000002150D469610>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002150D46B390>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002150D475150>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002150D46AE50>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (3/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] Unable to download API page: (<urllib3.connection.HTTPSConnection object at 0x000002150D476C90>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)') (caused by TransportError("(<urllib3.connection.HTTPSConnection object at 0x000002150D476C90>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')"))
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002150D4777D0>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading android player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x000002150D499290>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading android player API JSON

My output for the nightly build:

[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>from yt-dlp/yt-dlp-nightly-builds [86e3b8226] (win_exe) API
[debug] params: {'verbose': True, 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.18 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}}
[debug] Python 3.11.3 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev (setts), ffprobe 2023-04-10-git-b18a9c2971-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, sqlite3-3.40.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1807 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=zKhDNNvoBEg
[youtube] zKhDNNvoBEg: Downloading webpage
WARNING: [youtube] Unable to download webpage: (<urllib3.connection.HTTPSConnection object at 0x00000223A7D15E10>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)')
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x00000223A913AA10>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (1/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x00000223A91444D0>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (2/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON
WARNING: [youtube] (<urllib3.connection.HTTPSConnection object at 0x00000223A9139910>, 'Connection to www.youtube.com timed out. (connect timeout=20.0)'). Retrying (3/3)...
[youtube] zKhDNNvoBEg: Downloading ios player API JSON

Check if your exe is able to make any requests at all by adding this to the top:
import requests
print(requests.get('http://www.youtube.com').status_code)

If this fails, issue is somewhere in your environment and is unrelated to yt-dlp

Check if your exe is able to make any requests at all by adding this to the top:
import requests
print(requests.get('http://www.youtube.com').status_code)

If this fails, issue is somewhere in your environment and is unrelated to yt-dlp

I compiled it with only those two lines, and it does not work.
Thanks for your help, now i know that it's not a problem with yt-dlp


removed included default material; cleaned up surface and material networking

Perfect, this is pretty much what I was gonna do. Going to be a fun merge with my local code though :worried:


Limited number of fields displayed for user records in rails console

After upgrading to Devise 4.2.0 when displaying a user record in the rails console, for instance by running 'User.first', we are only seeing 'id', 'email', 'created_at', 'updated_at' and the custom fields we have added ourself. So for instance 'encrypted_password' is not displayed, but we can still access it directly eg. 'User.first.encrypted_password'.
We're on Rails 4.2.7, upgraded from 4.1.1.

Issue persists only with 4.2 only. On 4.1 it displays all fields. I am using rails 4.2.5 and mongoid 5

👍

This was added on #4140 to prevent sensitive data to leak on exceptions or log messages that use the record instance directly.


Play backwards

For some effects/use cases it would be great if this module also
allows playing backwards (either automatically or each frame).
Alternatively a frame index/counter which can be incremented/changed and passed.
Edit: Would a "negative" easing be the right approach (https://github.com/gre/kenburns/blob/master/API.md)?

For that, I would suggest you use the lower level API (animateStep), not the promise one (animate).
animate is just an helper to call animateStep many times.
when you run animate(source, from, to, duration, easing) it will call animateStep(source, from, to, progress) by varying progress from 0 to 1.
e.g.:
animate(source, from, to, 2000)

animateStep(source, from, to, 0)
animateStep(source, from, to, 0.01)
...
animateStep(source, from, to, 0.98)
animateStep(source, from, to, 0.99)
animateStep(source, from, to, 1) // after 2000 millis

see animate code:  https://github.com/gre/kenburns/blob/master/src/Base.js#L75-L89
you can inspire from it to do a backwards.
Also, i'm pretty sure you can workaround the easing to do this: p => 1-p and it will do backwards, but also if you want more control, just make your own version of that render loop.


Having invalid options:version value on explicit provider causes conversion to produce incorrect output

What happened?
When attempting to use pulumi convert to generate additional examples from yaml to other languages, the other languages were missing the explicit provider.
Example

Source YAML
name: base
runtime: yaml

plugins:
  providers:
    - name: base
      path: ../../bin
resources:
  provider:
    type: pulumi:providers:base
    properties:
      accessToken: "accessToken123"
    options:
      version: 0.0.0-alpha.0 #Appending a "v" to the front of this actually makes it behave correctly
      pluginDownloadURL: https://testfakedownload
  my-account:
    type: base:Account
    properties:
      name: myaccount
    options:
      provider: ${provider}

outputs:
  myoutput:
    fn::secret: ${my-account.secretValue}

Command used
pulumi convert --cwd ./examples/simple --language typescript --out ../test --logtostderr --generate-only --non-interactive
Generated Typescript
import * as pulumi from "@pulumi/pulumi";
import * as base from "@pulumi/base";

const provider = new base.Provider("provider", {accessToken: "accessToken123"});
const myAccount = new base.Account("my-account", {name: "myaccount"});
export const myoutput = pulumi.secret(myAccount.secretValue);

Output of pulumi about
CLI
Version      3.143.0
yaml         converter  1.12.0

Additional context
I actually think this maybe should fail some sort of validation rather than succeeding incorrectly. In my testing, removing version or appending v to the front of the string actually makes it behave as expected. For example version: v0.0.0-alpha.0
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@tgummerer like the schema or what do you mean here?

I used the following to test with
https://github.com/zbuchheit/provider-base/blob/main/examples/simple/Pulumi.yaml

This is exactly what I was looking for, and I can repro the issue now.  Thanks!


Change CLI to use clap

Changes CLI setup from manual to use the clap crate. Makes it easier to maintain and more fool proof.
!! Need further recommendations for "short names" for certain CLI flags, like dir which is currently -r.

Hey @Vaimer9 thank you for this pr!
I reviewed your code and here's my feedback:

Using dir instead of EntryType is more efficient. (I like it)

if config.dir == is_dir && config.is_match(&name) {
    result.push(entry.path());
}

Flags are missing their descriptions.
It looks like clap is slowing down the program.

Could remove clap and leave the dir field?


Add from sat checked

Replaces https://github.com/rust-bitcoin/rust-bitcoin/pull/3698

Adds a checked variant of from_sats
Explicitly panic in function from_sats

With these two changes, we can add another commit to change MAX to MAX_MONEY next.  Once that is done, remove #[allow(clippy::absurd_extreme_comparisons)].

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

10 of 12 (83.33%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.002%) to 82.872%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

units/src/amount/signed.rs
8
10
80.0%

Totals

Change from base Build<PHONE_NUMBER>8:
-0.002%

Covered Lines:
20201

Relevant Lines:
24376

💛  - Coveralls

What is the benefit of panicing instead of returning an error?

What is the benefit of panicing instead of returning an error?

Are you asking what's the benefit of having an unchecked variant?

If you want to do something like from_sats(42) and you don't want to bother handling an error because you know by looking at the call that it won't panic or error.
It makes the upgrade path simple because we already provide an unchecked variant from_sats.

I wonder though if it would be wiser to deprecate from_sats and make a new method with the panic behavior called from_sats_unwrap to explicitly opt in to behavior change.  I had named it from_sats_unchecked in a previous PR but its doing a check and then panic so that name doesn't make sense I think.

What is the benefit of panicing instead of returning an error?

Are you asking what's the benefit of having an unchecked variant?

I was referring to this line: "Explicitly panic in function from_sats"
The question stands.

I was referring to this line: "Explicitly panic in function from_sats"

@tcharding I'll try to be more precise with what was meant by "Explicitly panic in function from_sats" to hopefully dispel any confusion.  What is meant is: Explicitly panic if the value passed exceeds max.  In otherwords, don't overflow and panic no matter which mode.
As for your earlier question "What is the benefit of panicing instead of returning an error?".  The benefit of panicking is that you would only call this function ideally when you either don't care if it panics or if you know it won't.  That way, you don't need to worry about handling the error case upon return.  Also, there are a number of functions in the test suit that do not expect an error to be returned, and it would be a rather large and disruptive change to add that now, not to mention for those who depend on this downstream.
Lastly, there is also the checked version from_sats_checked for the case where you want to know if max is exceeded and gracefully handle the functions return.  I return an Option instead of Result in that case since it seems consistent with other functions in this library to do so.  Although, I'm open to suggestions if it seems like I'm overlooking something.

Lastly, there is also the checked version from_sats_checked for the case where you want to know if max is exceeded

Cool, we got to the bottom of it. IMO from_sats should return an error and there should be not be a  function called from_sats_checked

Cool, we got to the bottom of it. IMO from_sats should return an error and there should be not be a function called from_sats_checked

@tcharding I mean I guess from_sats could return an error, and then we don't need from_sats_checked.  This is counter to your comment on https://github.com/rust-bitcoin/rust-bitcoin/pull/3692 I think:

The unwraps are ugly, perhaps we should add a from_sat_unchecked function that doesn't enforce the invariant?

If we change this to return a result, there's going to be a lot of ugly unwraps(), right.

Fuck, you are using up a lot of my time these days @yancyribbens

That's because I contribute a lot to this project, more than anyone else except yourself and @apoelstra, so I take this as a compliment.

closing in favor of https://github.com/rust-bitcoin/rust-bitcoin/pull/3730


Feat/add certification process2

Explain SCS DigiSov and Certs. How to get into compliance.

- Explain the motivation for the SCS certification levels rooted in
  the levels of digital sovereignty.
- Give a practical example how to achieve the SCS certification by
  running the tests.
- Hints on how to get listed, why we have the HM there and how to
  react to failed runs.

Note that I have pushed the changes to the staging branch, so we should see the results in https://docs-staging.scs.community/ in a few minutes.

This one supercedes PR #251.
Please review the end results, I unfortunately went through some trial and error for getting docusaurus to do what I need.
Note that the Markdownlint check fails with or without this patch. It barfs on the generated index files, not sure whether the generation process can be tweaked or whether we'd need to filter things out for that check. In any case, that's unrelated to the contribution from this PR.

From Compliant cloud environments overview to 'Compliant Clouds'

This page was called overview before and the document's name is still overview (and should stay so, as it is linked to from various places). That's why I kept the word "overview" in the title. A compromise could be 'Compliant Clouds overview' to keep the relation with the link name.

3. Move "The taxonomy of digital sovereignty" from Certification Overview Page
Starting with a short introduction and the list of scs certification levels helps engage visitors, reducing the chance of overwhelming with too much text at in the beginning. Introduction sentence could be:

Agreed. While I think we should explain how our understanding of digital sovereignty drives the levels of SCS certification, not everyone wants to read through it, so splitting out additional background info into a separate page makes sense.

I suggest to shorten the sidebar item texts to keep them one liners for better readability such as:
* From `Getting SCS-compatible certification (for operators)` to `Certification Process` or `Getting Certified`

* From `Example testing and adjustment for SCS IaaS-compatible compliance` to `Testing Compliance' or `Compliance Testing`

* From `Compliant cloud environments overview` to 'Compliant Clouds'

Do I need to change the titles in the pages? Or is there a straight-forward way to do it in sidebarStandards.js?

I'm not sure I am in a position right now to really review this.
Just a few observations:
* The "welcome page" (top-level page) regarding certification is now a big wall of text. I wouldn't bother to read all of that or try to find the things that interest me. I suggest to make the welcome page quite short and just provide a big-picture overview (2--3 paragraphs) plus a list of links to the pages about specific topics for those who are interested.

Did you have a look after https://github.com/SovereignCloudStack/docs/pull/253/commits/e23e99cccd18d19469d3acd8e503f346138bad14 ?
I tried to address this, as Max had pointed this out as well and I agree.

The "Getting certification" page repeats the (authoritative) standard scs-0004-v1. It may be warranted to translate the standard into a more accessible language, but (a) I would be prefer to have the standard accessible as-is, and (b) the redundancy is a flood-gate for divergence and contradictions. There is a reason why I merely linked to the standard.

The example page is great, but I would make that a blog article and then link to the blog article from the welcome page. To me, the example shows that the scripts are still too cumbersome. For instance, why do we need -V v4? Also, we could use scs-test-runner.py, whose CLI syntax is simpler, but that one immediately sends the report to the compliance monitor (because we don't want to make cherry-picking results a thing).

Hmmm, I wanted to write a blog article first and then opted for improving the documentation.
So, maybe we do the following:

Keep the new intro (after e23e99c !) in the docs, hiding the wall of text (explanation of DigiSov) in a separate page.
We have another look at scs-0004 and check whether it can be made easier to read
Move the example into the blog article -- additional hints on how to get certification could probably be added there if they don't fit into scs-0004.

Thoughts?

PR for blog article still has to be done (cloning the website repo takes ages over satellite internet).
PR for standards is in https://github.com/SovereignCloudStack/standards/pull/781

The PRfor the blog article https://github.com/SovereignCloudStack/website/pull/1029


Support to OR'ed predicates in tupleDomain

Quoting @electrum 's comment in this post https://groups.google.com/forum/#!searchin/presto-users/david/presto-users/gw63DLA86BU/naVRaK6m2uEJ
"Unfortunately, it is not yet possible to pass an OR expression into a connector with the current connector interface. We do plan to support passing arbitrary expressions (and even full queries) into connectors."
There has been some changes around TableLayout and ConnectorPartition. Does presto support predicate pushdown for OR filters in tuple domain yet?
This is significantly important when you need to run a query like (SELECT * FROM foo WHERE foo.id IN  (x, y,z .....)) against huge amount of data. Currently presto fetches entire data in cluster to run this kind of queries which is impractical.
Creating this issue to track development around this.
Thanks.

Closing this.. as I can see OR filters as well in the tuple domain.

To clarify, we support this:
(a = 1 OR a = 2) AND (b = 3 OR b = 4)

Which is the same as this:
a IN (1, 2) AND b IN (3, 4)

But not this:
a = 1 OR b = 2

Thanks for replying so quick @electrum
Do you guys have plan to support the last type of filters [ a = 1 OR b = 2 ] ?

A tuple domain for a = 1 OR b = 2 is not possible (it only supports AND between different columns).  This will be supported when we have push down for arbitrary expressions to connectors.


Multiple recaptcha on the same page

The element should be modified to add a callback function, it would not be the ideal way of solving it however we would be able to have multiple recaptcha on the same page without changing any helpers or components.
I will submit a PR with the change we are using for our application when using this plugin. I have tested that it verifies independent of any other recaptcha on the page.
Best regards

Thank you


[firebase-database] CocoaPods could not find compatible versions for pod "Firebase/Database"

Hi,
I updated Nativescript from 6 to 8 and I have now an error I was wondering if you can help.
I have the following error when trying to run " ns run ios " :
Pre-downloading: `SwiftyCam` from `https://github.com/NathanWalker/SwiftyCam.git`, branch `feature/swift-4.2`
[!] CocoaPods could not find compatible versions for pod "Firebase/Database":
  In Podfile:
    Firebase/Database

    Firebase/Database (~> 9.0)

None of your spec sources contain a spec satisfying the dependencies: `Firebase/Database, Firebase/Database (~> 9.0)`.

You have either:
 * out-of-date source repos which you can update with `pod repo update` or with `pod install --repo-update`.
 * mistyped the name or version.
 * not added the source repo that hosts the Podspec to your Podfile.

[!] There are duplicate dependencies on `Firebase/Analytics` in `Podfile`:

- Firebase/Analytics
- Firebase/Analytics (~> 9.0)

[!] There are duplicate dependencies on `Firebase/Database` in `Podfile`:

- Firebase/Database
- Firebase/Database (~> 9.0)

[!] There are duplicate dependencies on `Firebase/Crashlytics` in `Podfile`:

- Firebase/Crashlytics
- Firebase/Crashlytics (~> 9.0)

[!] There are duplicate dependencies on `Firebase/Storage` in `Podfile`:

- Firebase/Storage
- Firebase/Storage (~> 9.0)
'pod install' command failed.
For more information on resolving CocoaPod issues in NativeScript read.

I tried ns clean , npm i .. didn't work.
Infos:
ns --version 8.2.3 up to date
node -v v16.15.0
package.json:
{
  "description": "NativeScript Application",
  "license": "SEE LICENSE IN <your-license-filename>",
  "repository": "<fill-your-repository-here>",
  "dependencies": {
    "@angular/animations": "~13.2.0",
    "@angular/common": "~13.2.0",
    "@angular/compiler": "~13.2.0",
    "@angular/core": "~13.2.0",
    "@angular/forms": "~13.2.0",
    "@angular/platform-browser": "~13.2.0",
    "@angular/platform-browser-dynamic": "~13.2.0",
    "@angular/router": "~13.2.0",
    "@nativescript/angular": "^13.0.0",
    "@nativescript/background-http": "~5.0.0",
    "@nativescript/camera": "^5.0.12",
    "@nativescript/core": "^8.2.3",
    "@nativescript/datetimepicker": "2.1.9",
    "@nativescript/firebase": "^11.1.3",
    "@nativescript/firebase-core": "^2.0.1",
    "@nativescript/firebase-crashlytics": "^2.0.1",
    "@nativescript/firebase-database": "^2.0.1",
    "@nativescript/firebase-storage": "^2.0.1",
    "@nativescript/geolocation": "~8.0.0",
    "@nativescript/imagepicker": "~1.0.0",
    "@nativescript/mlkit-barcode-scanning": "^1.0.0-alpha.8",
    "@nativescript/mlkit-core": "^1.0.0-alpha.8",
    "@nativescript/mlkit-face-detection": "^1.0.0-alpha.8",
    "@nativescript/theme": "^3.0.1",
    "@ngx-translate/core": "^11.0.1",
    "@ngx-translate/http-loader": "^4.0.0",
    "@nstudio/nativescript-camera-plus": "^4.1.0",
    "@nstudio/nativescript-loading-indicator": "~4.0.0",
    "@nstudio/nativescript-pulltorefresh": "~3.0.0",
    "@nstudio/nativescript-snackbar": "~2.0.0",
    "nativescript-imagecropper": "~4.0.0",
    "nativescript-nfc": "~5.0.0",
    "nativescript-permissions": "^1.3.8",
    "nativescript-sound-kak": "^1.2.0",
    "nativescript-theme-core": "~1.0.4",
    "nativescript-ui-chart": "~9.0.0",
    "nativescript-ui-sidedrawer": "~10.0.0",
    "nativescript-webview-interface": "^1.4.2",
    "rxjs": "^7.4.0",
    "tns-platform-declarations": "^6.5.15",
    "zone.js": "~0.11.4"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^13.3.6",
    "@angular-eslint/builder": "13.0.1",
    "@angular-eslint/eslint-plugin": "13.0.1",
    "@angular-eslint/eslint-plugin-template": "13.0.1",
    "@angular-eslint/schematics": "13.0.1",
    "@angular-eslint/template-parser": "13.0.1",
    "@angular/compiler-cli": "~13.2.0",
    "@nativescript/android": "8.2.2",
    "@nativescript/ios": "8.2.3",
    "@nativescript/schematics": "~0.5.0",
    "@nativescript/types": "~8.2.0",
    "@nativescript/webpack": "~5.0.0",
    "@ngtools/webpack": "~13.2.0",
    "@types/jasmine": "^3.5.11",
    "@typescript-eslint/eslint-plugin": "5.3.0",
    "@typescript-eslint/parser": "5.3.0",
    "eslint": "^8.2.0",
    "typescript": "~4.5.5"
  },
  "readme": "NativeScript Application",
  "main": "./src/main.ts",
  "scripts": {
    "lint": "ng lint"
  }
}

If you need anything else let me know, thanks !

Remove  "@nativescript/firebase": "^11.1.3",

I also used @nativescript/firebase to be able to use mlkit @triniwiz

If you are using the new mlkit ("@nativescript/mlkit-core") then you don't require the old firebase plugin, also you can upgrade those to the 1.0

Now I got this @triniwiz
Pre-downloading: `SwiftyCam` from `https://github.com/NathanWalker/SwiftyCam.git`, branch `feature/swift-4.2`
[!] `GoogleAppMeasurement` requires CocoaPods version `>= 1.10.2`, which is not satisfied by your current version, `1.10.0`.
'pod install' command failed.

Yeah my bad I removed it but I got a new error now @triniwiz

It works now I finally succeeded to launch my app, thank you SO much @triniwiz


Run snapshot test using corresponding branch

MOB-3716
What was solved?
This PR extends clone-snapshots-ci make command by running checkout.sh script, which tries to make checkout to the source branch in Snapshots if it exists, otherwise tries to make checkout to the destination branch if it exists, otherwise makes checkout to master branch
Release notes:

[ ] Feature
[x] Ignore
[ ] Release notes (Is it clear from the description here?)
[ ] Migration guide (If changes are needed for integrator already using the SDK - what needs to be communicated? Add underneath please)

!squash


RLock server when checking shard groups

This fixes a race detected by the race detector. "Create shard groups"
commands must be broadcast across the cluster without holding the server
lock so the commands are created under lock, and then processed after
releasing the lock.

Actually, need to make small change to this.

Green build, ready to merge.

:+1:


targetSdkVersion="26" Notifications not received on  some android device

If the app target targetSdkVersion="26". Notifications wont be displayed on Android devices running android 8. I think this is due to how application in the background should handle notifications. Could you please check ?

Same issue with receiving notifications in background on Android 8.
0-02 11:48:44.700  1969  3885 W ActivityManager: Background start not allowed: service Intent { act=com.google.android.c2dm.intent.RECEIVE flg=0x1000010 pkg=ua.com.mhp.agro.dev cmp=ua.com.mhp.agro.dev/ti.goosh.IntentService (has extras) } to ua.com.mhp.agro.dev/ti.goosh.IntentService from pid=14856 uid=10165 pkg=ua.com.mhp.agro.dev
10-02 11:48:44.704 14856 14856 D AndroidRuntime: Shutting down VM
10-02 11:48:44.706  3263  3263 W GCM     : broadcast intent callback: result=CANCELLED forIntent { act=com.google.android.c2dm.intent.RECEIVE pkg=ua.com.mhp.agro.dev (has extras) }
10-02 11:48:44.717 14856 14856 E AndroidRuntime: FATAL EXCEPTION: main
10-02 11:48:44.717 14856 14856 E AndroidRuntime: Process: ua.com.mhp.agro.dev, PID: 14856
10-02 11:48:44.717 14856 14856 E AndroidRuntime: java.lang.RuntimeException: Unable to start receiver ti.goosh.BroadcastReceiver: java.lang.IllegalStateException: Not allowed to start service Intent { act=com.google.android.c2dm.intent.RECEIVE flg=0x1000010 pkg=ua.com.mhp.agro.dev cmp=ua.com.mhp.agro.dev/ti.goosh.IntentService (has extras) }: app is in background uid UidRecord{9df5545 u0a165 RCVR bg:+1m2s763ms idle change:uncached procs:1 seq(0,0,0)}
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ActivityThread.handleReceiver(ActivityThread.java:3228)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ActivityThread.-wrap17(Unknown Source:0)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1696)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:106)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:171)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:6633)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:547)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:823)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: Caused by: java.lang.IllegalStateException: Not allowed to start service Intent { act=com.google.android.c2dm.intent.RECEIVE flg=0x1000010 pkg=ua.com.mhp.agro.dev cmp=ua.com.mhp.agro.dev/ti.goosh.IntentService (has extras) }: app is in background uid UidRecord{9df5545 u0a165 RCVR bg:+1m2s763ms idle change:uncached procs:1 seq(0,0,0)}
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ContextImpl.startServiceCommon(ContextImpl.java:1531)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ContextImpl.startService(ContextImpl.java:1487)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.content.ContextWrapper.startService(ContextWrapper.java:660)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.content.ContextWrapper.startService(ContextWrapper.java:660)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.support.v4.content.WakefulBroadcastReceiver.startWakefulService(WakefulBroadcastReceiver.java:99)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at ti.goosh.BroadcastReceiver.onReceive(BroadcastReceiver.java:32)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	at android.app.ActivityThread.handleReceiver(ActivityThread.java:3221)
10-02 11:48:44.717 14856 14856 E AndroidRuntime: 	... 8 more

Same problem here...
Any solution?

Has anyone figured out a solution for this?

I am using the appcelerator sdk 7.3.1.GA with the ti.goosh 4.1.1 and it's OK
but with the appcelertor sdk 7.4.X is failing

I highly recommend using this plugin: https://github.com/hansemannn/titanium-firebase-cloud-messaging
It works perfectly and no crashes.

Still no news on a fix for this?
Play Store now demands that updates has targetSdkVersion=26 so this is important.

Good news .... there is a new push whit the fix 👍
Merge branch 'feature/oreo'
I am whating for the dist version

Let's hope that someone who make the a new release sees this (today) 🤞

We've merged that fix to try and make the module work with Android Oreo, but there are still things to do. As stated in this comment, there's been radical changes in the way Android handles services. You can find the details here: https://developer.android.com/about/versions/oreo/background.
Basically, methods like "GcmReceiver.startWakefulService()" will throw exception when they are called while the app is closed or flagged as "in background". This means that the current master of Ti.Goosh doesn't show notifications in those cases.
We're trying to migrate the module to the Firebase library to solve these issues.

I have compiled the module ...
ti.goosh-android-4.1.1.zip
I am testing android 8 and 7 (sdk 25,26) and i have problem with the custom sound on android 8 ... and  i have only this issue

The good news: This fix solves getting push when app is backgrounded.
Bad news: The push does not show up when the app is dead. Furthermore our app stalls and eventually crashes when starting up from dead after having sent a push to the device.

do you have ?
// subscribe

TiGoosh.subscribe({
	topic: "/topics/myTopic",
	success: function(e) {
		console.log("unsub done " + e);
	},
	error: function(e){
		console.error("error")
	}
})```

@furthurr  So we need to subscribe to a topic now ? Can be anything ?

I think so, I had the same problem that @andersdp  says, so this way was solved xD.

@furthurr It seems unlikely that subscribing to a topic solves the problem of creating a background service from a backgrounded app.
Have you tried sending a notification to the app while it's closed? Is the notification displayed in the tray?

this build have te last modification
Uploading ti.goosh-android-4.1.1.zip…

thiz is the las build example;
https://www.youtube.com/watch?v=4b8IyU3YbNU

@furthurr  will integrate  the build and keep you guyz updated. Thx :)

Seems that build above doesn't have the latest changes.
Works for me only after I built module manually from branch.
Tested when app is in background mode or closed (Android 8.0).

We've just released a new version of Ti.Goosh that should solve this problem. You can find it here.
Let us know if you're still experiencing the same problem with Ti.Goosh 4.2.0.

I tried the new version (4.2.0) and it worked just fine on Android 9.


feat: Added Support for Bigquery for staging

This PR to discuss these changes:

Adds support for the BQ staging macro
Moves the hash algorithm selector to the top-level (outer edge of the system)
simplifies the hash_columns function

@DVAlexHiggs re: future config, If you're worried about passing all the config down N layers for them to apply, what do you think about specifying a config folder with "get config" macros for the main configuration?
for example a config__get_hash_separator() which looks up the config on the vars and returns '||'.

Since this is way outdated now and there's other works in progress to get BQ up and running, I'm closing this

Since this is way outdated now and there's other works in progress to get BQ up and running, I'm closing this

I was leaving it open to bring your changes in for the new future as a base to work against for our work on BigQuery. It would be great to have your contributions, however you're right that it may be too outdated. Maybe we can work something out?


Handle continue_train semantics in correct way

https://github.com/nnstreamer/nntrainer/pull/720#discussion_r521073037
when training.

readModel() should be called internally iff continue_train is true and hidden from users side

when inferencing

readModel() should be called if file does not exist raise an error.

@kparichay Please review 👍

:octocat: cibot: Thank you for posting issue #736. The person in charge will reply soon.

For inferencing - yes
For training - IMO readModel must be called to load the model weights but load training weights iff continue_train is true.
To not call readModel so as to training from scratch, either the save path file must be different (can add another property for it but I think file already exists/not is good enough)

This is handled with adding load/save api. The user requires to load explicitly if he wants to initiate weight with a certain weight file. Save_path only cares about saving


Twilio v1.0.0-beta3 fails

Hi,
I tried to install the video-quickstart-swift projects, but when i run pod install (pod version = 1.1.0.rc.3) as explained in the readme file, cocoapods displays this error log :
Analyzing dependencies
[!] Unable to satisfy the following requirements:

TwilioVideo (= 1.0.0-beta3) required by Podfile
TwilioVideo (= 1.0.0-beta3) required by Podfile
TwilioVideo (= 1.0.0-beta3) required by Podfile
TwilioVideo (= 1.0.0-beta3) required by Podfile

None of your spec sources contain a spec satisfying the dependency: TwilioVideo (= 1.0.0-beta3).
You have either:

out-of-date source repos which you can update with pod repo update.
mistyped the name or version.
not added the source repo that hosts the Podspec to your Podfile.

Note: as of CocoaPods 1.0, pod repo update does not happen on pod install by default.

Hi @fpoisson,
Thanks for reaching out to us.
Did you also try pod update ?  As of CocoaPods 1.0, pod repo update does not happen on pod install by default.
-Piyush

@fpoisson You need to invoke pod repo update twilio to grab the latest specs.


error trying to update supporters page

I changed the Varnish configuration to wait slightly longer for WordPress to start serving the page.
Try now?


throughput drop when train resnet50

Hi, @rwightman.
When I run resnet50 on my A100, I notice throughput will drop once in a while.

I use the following command to run resnet50 on A100 with NGC22.04:
python train.py /aisw/datasets/latest/datasets/vision/imagenet -b 512 --sched cosine --epochs 1 --lr 0.05 --amp --channels-last --model resnet50  -j10 --pin-mem --log-interval 1
And if I run with benchmark.py, I can get good result, throughput always 2600 img/s.

@wchen61 It's normal, there are hiccups in the instantaneous throughout, you should be trying to improve the avg which is only 1923. The gap between last step and avg is fairly high so there is likely a bottleneck. Not a timm bug. Make sure you've got pillow-simd installed and no normal pillow in the env. Use a fast ssd for dataset or use TFDS.

@wchen61 It's normal, there are hiccups in the instantaneous throughout, you should be trying to improve the avg which is only 1923. The gap between last step and avg is fairly high so there is likely a bottleneck. Not a timm bug. Make sure you've got pillow-simd installed and no normal pillow in the env. Use a fast ssd for dataset or use TFDS.

After I use the pillow-simd, I can get throughout of 2500 img/s. Thank you so much for saving my day! @rwightman


20181220 - meeting

@dpaez and I exchanged via mail about having a online meeting to share what we do about DAT. This is community building and as such we want to open it to other attendees.
First meeting:
Thu, 20 Dec 2018 at 15:00 GMT  - Online.
Not sure how we are meeting yet, but if you are interested please raise your hand here.

What about https://meet.jit.si/ ?

https://meet.jit.si/DATcomm-comm

Thanks for taking initiative on this!


A file is not found: no module named 'src.evalcap.coco_caption'

Hi
When I implemented 'run_caption_VidSwinBert.py' to train a new model, an error message appeared: no module named 'src.evalcap.coco_caption'.
I found that your code lacks relevant files. Can you share them?
Thanks!

The relevant files are pre-installed in the container. If you launch the container as instructed, you should be able to find the files.

Closing as duplicate https://github.com/microsoft/SwinBERT/issues/3

The relevant files are pre-installed in the container. If you launch the container as instructed, you should be able to find the files.

Due to the server permissions, I can't install the container, so if You can, please transfer the relevant files to the GitHub.


refactor(admob): upgrade quickstart to admob 20.0.0

The latest version of AdMob (20.0.0) contains breaking changes. This PR should upgrade our admob quickstart to that version.

Notice that I deleted the androidTests because:

They test features from the previous AdMob version.
I'm not sure if they even run correctly since #1247

@rosariopfernandes thank yoU!

@djj1109 please stop approving random PRs on this repository.


swin definition mismatch with weight

Dear author:
I found the naming of  class SwinTransformer in mmdet/models/backbones/swin.py can not match with weight file in https://github.com/open-mmlab/mmdetection/tree/main/configs/swin, but when the config file set "init_cfg=dict(type='Pretrained', checkpoint=pretrained)", the weight can be used to initialized, where is the functionn be implemented? It seems in the baseModule, but i cann't found there, can you help me?

class definition is backbone.patch_embed.projection, weight file is patch_embed.proj

the function is implemented in class SwinTransformer's init_weights function.


Add Two Numbers

Leetcode Question no. - 2
Problem name - Add Two Numbers
Instructions

Create a new folder for your script and file/folder name should be appropriate.
Create a README.md in your folder for program Instructions
add requirements.txt if needed

Programming Language

[x] Python

Happy Coding

I would love to do this. and also I have very good knowledge in python.


General issue of scope: open source vs software engineering in general?

(Opening at @david-a-wheeler's suggestion.)
Is this about overall effort about open source specifically, or is it about software engineering in general?
A lot of the requirements, particularly around security (reactive & preventative), are really generic software engineering concerns. Once that door is open, though, there is no limit to the scope.
I think it might help to divide criteria strictly between things that are really about open source, and things that are about software engineering in general. Then for every item in the latter category, its inclusion must be justified on stricter grounds -- for example, it's some general software engineering practice that open source projects tend to be systemically weak at for some reason, so its inclusion as a criterion here would be to give projects an opportunity to demonstrate that they do not suffer from that systemic weakness.

The original impetus of these criteria was to create more secure FLOSS, but there aren't simple security-only criteria or FLOSS-only criteria that you can realistically apply in isolation.  If you want to counter vulnerabilities well, you need to look at things more holistically - that is, look at the project as a whole, and include criteria that are generic software issues (not just security, and not just FLOSS-unique).
To get to the heart of your comment: I think it's important to have a set of criteria specifically for FLOSS projects, and that it include general software engineering practices.  Many "generic" software engineering "best practices" are biased against FLOSS (presumably unintentionally), and instead tend to focus on criteria for corporate proprietary software.  Be it ISO 9000 or the Joel test, many criteria presume that the software is proprietary and directly funded by a commercial entity.  If nothing else, they often presume you can have expensive processes, physical workspaces, hired personnel, and other trappings that are often not true for FLOSS projects.  Even the Common Criteria, which has been applied to FLOSS, doesn't give any credit for some good FLOSS practices. For example, the Common Criteria gives no credit for mass peer review, even though that is the standard way that mathematics and the sciences create and encourage high-quality results.  I think it's valuable to have a "one-stop shop" with all the key criteria that apply to FLOSS projects if the project wants to produce good secure software.
Indeed, per the background, we intentionally include a larger scope to achieve our objectives:

The criteria are focused on best practices for FLOSS, particularly those that are more likely to produce secure software, and not on proprietary software. Clearly developers of proprietary software should also strive to develop secure software. That said, FLOSS software is sufficiently different that it's appropriate to have a set of criteria specific to FLOSS. For example, a key approach for developing secure FLOSS is to enable and encourage widespread review of the software. Many of the criteria focus on how to best enable and encourage this review. This kind of information availability and worldwide detailed review would be unacceptable to most businesses that develop proprietary software. As a result, there's a need for criteria that focus specifically on FLOSS.

We could in principle divide criteria in terms of "FLOSS or anything" or "relevant to security or not", but it would never be a clean division.  There'd be a lot of overlaps, and frankly, a lot of unhelpful arguments on where things go.  We do have a "security" section, but even so, many of the criteria in other sections are relevant to secure software.
We have a similar issue with include "security" vs "not security" criteria.  It's not that clean a division there either:

Heartbleed was instructive - as far as I know, every significant static analysis tool in existence that can handle C has been used on OpenSSL. Yet none of the static analysis tools found the Heartbleed vulnerability, nor did any of the widely-used dynamic analysis tools of the time. It was only found later by manual review (Google) and a special new fuzzing tool (Codenomicon).
There's also academic support for the claim that "generic quality" helps security.  For example, ["Predicting Software Assurance Using Quality and Reliability Measures" by Dr. Carol Woody, Dr. Robert Ellison, and Dr. William Nichols,  December 2014 (CMU/SEI-2014-TN-026)](http://resources.sei.cmu.edu/asset_files/TechnicalNote/2014_004_001_428597.pdf] looked at various kinds of evidence and concluded, "Our research suggests that implementing systems with effective operational security requires incorporating both quality and security considerations throughout the lifecycle."

In short, I think we need to include the more general software requirements, so that there's a simple single "one-stop shop" for FLOSS project criteria.  It's hard to separate, and because we want to be a one-stop shop, we don't want to separate the criteria.  If a project meets the set of criteria, whatever they are, then that should be an excellent indication that they're doing a good job.

I'm closing this.  Clearly different scopes are possible, but we've chosen one, and I think it's reasonable :-).  That said, thanks for the question - that means we've documented the answer here.

Yup; I can buy the above :-), and agree that having it documented here is useful -- thanks for taking the time.


Picker item polish

Testing #14367

Enter the URL of the running JupyterHub Server... could be shortened to Enter URL of running JupyterHub server in the quickpick item name and placeholder
Invalid JupyterHub URL is pretty vague and hard to troubleshoot (could be because there isn't a server at that URL, or because of some other connectivity error), would be nice to either include the actual error message or provide a command link to open the extension output channel for logs

If the username and password are wrong we show the username input again and say Invalid username or password., there shouldn't be a period at the end of that as it's inconsistent with the other validation messages that we show, and ideally it should say something more actionable like Your username or password did not match (even better if we can say which part was incorrect)

even better if we can say which part was incorrect)

As a best practice in security this is never done, else anyone trying to hack your account know they have made progress and need to only focus on the other

error message or provide a command link to open the extension output channel for logs

To my knowledge cors errors cannot be detected,

could be shortened to Enter URL of running JupyterHub server in the quickpick item name and placeholder

Leaving this as is, as this is what we have in the Jupyter extension for standard urls.


pass metadata through in pdf.ocr.correct

details can be found here: https://support.prodi.gy/t/pdf-ocr-image-annotation-metadata-feature-suggestion/7211
just a suggestion, in order to be able to link text transcription and bounding boxes to PDF data in order to train a layoutLM

@koaning & @magdaaniol
Should this warrant a version bump in setup.cfg?
[metadata]
- version = 0.2.1
+ version = 0.2.2
description = Recipes for PDF annotation
url = https://github.com/explosion/prodigy-pdf
author = Explosion
author_email =<EMAIL_ADDRESS>
[options]
zip_safe = true
python_requires = >=3.8
install_requires =
    pypdfium2==4.20.0
    Pillow<11.0.0
    pytesseract==0.3.10

[options.entry_points]
prodigy_recipes =
    pdf.image.manual = prodigy_pdf:pdf_image_manual

[bdist_wheel]
universal = true

[sdist]
formats = gztar

Thanks @PaulBFB!

Yes, @it176131 - I'll take care of it!