USB HID Detach Kernel Driver fix for FreeBSD.

This change catches all exceptions during kernel driver detach try.
It was blocking pyOCD on FreeBSD where HID was attached to DAPLink.
More generic kernel driver detach exception handling fixes crashes
on systems where user is not permitted (or even welcome) to detach
HID driver on the whole system just to use DAPLink.
On FreeBSD user is not allowed to detach HID kernel drivers because
that would impact whole OS and the rest of the workstation services,
while DAPLink works fine even with those HID drivers attached :-)
I have added one Debug message before detach and Warning after it fails.
Signed-off-by: Tomasz 'CeDeROM' CEDRO<EMAIL_ADDRESS>

Cool, thanks @cederom !

/morph test

/morph test

/morph test

sorry!

/morph test

test

TANK U SIR =)


Fix the test case.

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[ ] New feature provided

[ *] Improve performance

Related issues

Bug fix

Bug description.

How to fix?

New feature or improvement

Describe the details and related test reports.

Ci haven't reported there is test case failure. Couls you explain why need this pr? What happens?

In fact, no error was reported for this test. I just think the test info here should be modified so that the developers will know the data structure clearly when they receive the warning info through webhook.
:-P

Sure. This should harm nothing. :)


Update KF_TechTree.cfg

Adding the rest of the polyspatial tech tree stuff. Why can't I merge this from the get-go? Is GitHub fracking up permissions again?

No, it is a protected branch and needs review

Done

Alright, so GitHub permissions were fracked up. No problem - I've fixed that already. You should be able to commit directly from here on out.


Planetary Approach Suite indicator now looks janky in Safari

Bug introduced in v0.17.0.
Somehow it's not rendering a circle correctly >.>

The fix for this is now in and will go out in the next update.

Fixed in v0.18.2


Fix: SAM not working with FSDP/DeepSpeed and LR scheduler.

Description
I'm tracing this bug back to Composer coming from using LLMFoundry after we noticed that our models didn't learn anything when running distributed training with FSDP alongside SAM and a LR scheduler. If running the same setup but w/o distributed training (on a single GPU), everything works fine.
Reproducibility
Just checkout the source branch at the commit a0b33a85, which contains only the tests (without the fix), and run:

## All tests pass (no distributed training with 1 GPU)
make test-dist-gpu WORLD_SIZE=1 EXTRA_ARGS="tests/algorithms/test_sam.py::TestSAMParamGroups"

## All tests failed (FSDP and DeepSpeed tests)
make test-dist-gpu WORLD_SIZE=2 EXTRA_ARGS="tests/algorithms/test_sam.py::TestSAMParamGroups"

Problem (FSDP)

When using SAM with a LR scheduler that resets the starting param_groups[0]['lr'] to 0.0 (such as  CosineAnnealingWithWarmupScheduler),  it varies the learning rate at SAMOptimizer.param_groups[0]['lr'],  but not the one at base_optimizer.param_groups[0]['lr'] (which is also reset to 0.0).
So the latter remains equal to 0.0 for the whole training process, which is why we don't see any learning.

Cause (short answer)

SAMOptimizer.param_groups[i] and base_optimizer.param_groups[i] are not referencing the same params because of the way Composer handles the sharding.

Cause (long answer)

Right when triggering Event.INIT, SAMOptimizer binds references to the param groups in base_optimizer., so up to this point both optimizers reference the same param groups.
Later (after initializing optimizers and anything triggered by Event.INIT) Composer initializes FSDP (here), which is weird since the FSDP docs warns about initializing the optimizer after doing the sharding, not the other way around.
Composer does the sharding by clearing the param groups from the SAMOptimizer and bounding to it the new sharded params.
At the end of the whole Trainer initialization, SAMOptimizer and base_optimizer reference different param groups:

SAMOptimizer the sharded ones.
base_optimizer the original ones.

During training, state.schedulers[0].step() is done on the 'lr' param  of SAMOptimizer, which are not the ones referenced by base_optimizer, but the real optimizer step is done by the base_optimizer, not SAMOptimizer. So no learning.

Problem (DeepSpeed)

As far as a I debugged, the problem above is just related to PyTorch's FSDP and not DeepSpeed (it handles the sharding process on its own).
But a problem arises when allocating the internal optimizer state when initializing the DeepSpeedEngine (here).

SAMOptimizer does the optimizer step iff a closure is passed, while BF16_Optimizer.initialize_optimizer_states() needs to do an optimizer step without any closure.

Solution

Let any sharding-related initialization be done on the param groups of the would-be base_optimizer optimizer instance, and just after all of this wrap the resulting optimizer with a SAMOptimizer instance, binding references to the sharded param groups.
For DeepSpeed this works, even though the underlying base_optimizer is a DeepSpeed Zero BF16_Optimizer instance, since it uses the same underlying parameter groups as the original PyTorch optimizer (see here and here).

This fix is achieved by triggering the SAMOptimizer wrapping later in the code. After this, distributed training works as expected.
Before submitting

[x] Have you read the contributor guidelines?
[ ] Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
[ ] Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
[ ] Did you update any related docs and document your change?
[x] Did you update any related tests and add any new tests related to your change? (see testing)
[x] Did you run the tests locally to make sure they pass?
[x] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

Thanks for the PR! Once the tests pass, you can tag me for review.
Also, the PR description is really well done 🙏 -- appreciate the detailed writeup. Fix makes sense to me! Once tests pass I'll approve

@mvpatel2000 Hi!
I fixed the failing tests.
Now the SAM algorithm always set the state.scaler to ClosureGradScaler(). This is safe since Trainer._use_grad_scaling() takes care of enabling/disabling the grad scaling step during training.

@mvpatel2000 yapf formatting tests are failing, but they are not related to files I changed in this PR.
How should I proceed?

@mvpatel2000 yapf formatting workflows are failing, but they are not related to files I changed in this PR.
How should I proceed?

I can help out with linting :) will take a look soon!


Restore lost entries in the CHANGELOG

Some changelog entries are lost at ea2de20a4e8775edf5b75ad67fd9acebb4d0f339.

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.
[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Run bundle exec rake default. It executes all tests and RuboCop for itself, and generates the documentation.

@marcandre Looks like some bug with the rake task.

@marcandre Looks like some bug with the rake task.

I'd say there was a bug in the existing Changelog itself, with two sections with the same heading, something that the task didn't expect. Is it worth fixing guarding against that?

@marcandre It's not a big deal, but I guess it'd be nice if the task failed in such cases, as that's a problem that's relatively easy to miss.


Conform to the official hashicorp vault service broker

Currently the spring-cloud-vault look forward for just one backend, which by default is secret, and tries to reason the following mounts out as per your documentation.

/secret/{application}/{profile}
/secret/{application}
/secret/{defaultContext}/{profile}
/secret/{defaultContext}

The above is configurable like the following as well, which is good
spring:
  application:
    name: recommendation-service
  cloud:
    vault:
      generic:
        application-name: ${spring.application.name}
        default-context: application
        backend: secret
      token: ${vault_token}
      scheme: https

But the problem arise when we try to conform to the official vault service broker , when we can not fully take advantage  of the mounts and convention they provide

When a new service instance is provisioned against the Cloud Foundry HashiCorp Vault Service >Broker, it will perform the following operations in the Vault server:
Mount the generic backend at /cf/<organization_id>/secret/
Mount the generic backend at /cf/<space_id>/secret/
Mount the generic backend at /cf/<instance_id>/secret/
Mount the transit backend at /cf/<instance_id>/transit/

Realistically we can have 4 backends with their broker.
2 static/fixed backends under /cf/<organization_id>/secret/ and /cf/<space_id>/secret/
and
2 ephemeral backends mounted under  /cf/<instance_id>/secret/ and  /cf/<instance_id>/transit/
$cf org my-org --guid
9a9fb216-42ae-4bfb-865e-69b44e256a98

$cf space my-space --guid
d0cae1cd-e4e2-44f4-9718-a1f6d54d605b

$cf app recommendation-service --guid
4fb00cd1-51cf-40c4-aa16-7e9d542bdcb6

Currently I am improvising with the below, but it is not very efficient. A bit ugly as well IMHO
$ vault write cf/d0cae1cd-e4e2-44f4-9718-a1f6d54d605b/secret/shared app.token=space-foo

spring:
  application:
    name: recommendation-service
  cloud:
    vault:
      enabled: true
      generic:
        application-name: ${spring.application.name}
        default-context: shared
        backend: ${vcap.services.aws-vault.credentials.backends_shared.space}
      token: ${vcap.services.aws-vault.credentials.auth.token}
      host: blah.compute.amazonaws.com
      port: 8200
      scheme: https

Another issue like you see, from the above config, is that we have to give explicit tokens for looks for host, port and scheme. Though the broker service provides well defined url under ${vcap.services.aws-vault.credentials.address}, I can not make use of it.
Please go through the https://www.hashicorp.com/blog/cloud-foundry-vault-service-broker/
and https://github.com/hashicorp/cf-vault-service-broker#architecture-and-assumptions

I created #115 to configure the Vault endpoint via URI. Cloud-autoconfiguration and reading from dedicated secret backends will happen in https://github.com/pivotal-cf/spring-cloud-vault-connector.

Connector support added at https://github.com/pivotal-cf/spring-cloud-vault-connector so closing this ticket.

@mp911de - how are you dealing with the default-context when you consider the list of generic backends in your spring-cloud-vault-connector  ? I think you are making all the generic SecretBackendMetadata with that context as well.
The shared generic backend mount points are meant to be constant and shared across and used likewise. With the context concept in the spring-cloud-vault it becomes slightly like a variable, from the clients perspective. IMFO I believe it becomes irrelevant and should be deprecated.

The default context originates in Spring Cloud Config and is supported in all config (Spring Cloud Config, Spring Cloud Consul) modules. It gives you an additional way to model your configuration layout. From my perspective, the default context makes a lot of sense with the dedicated backend and the application name based context makes a lot of sense with the shared backends. Not saying the other combinations are useless, the mentioned come with a lot more value.
If you have questions regarding Spring Vault and PCF, please open tickets in https://github.com/pivotal-cf/spring-cloud-vault-connector instead of commenting closed tickets.


Problems to setup embedded Spring Config Server

I'm on my way to setup Spring Cloud Config to allow one of our applications to change some configuration properties while runtime. As for now it is only one application to be managed by the Spring Config Server and I want to keep deployment simple, I would like to deploy the Config Server and the Application as one. As far as I got this is supported as "Embedding the Config Server".
To get this running, I used https://spring.io/guides/gs/centralized-configuration as base to setup a separated Config Server and a client. I also setup a config repository in file:/Users/mibutec/projects/config-repo containing
message=Hello, Mibutec

This setup works fine, and when calling curl localhost:8888/message I see my message Hello, Mibutec. Everything fine!
Now I tried to get the Config Server also running within the client application.
I changed the application.properties of the client application as follows:
server.port=8888
spring.application.name=a-bootiful-client
management.endpoints.web.exposure.include=*

# spring.config.import=http://localhost:8888/
spring.profiles.active=composite
spring.cloud.config.server.composite[0].type=git
spring.cloud.config.server.composite[0].uri=file:/Users/mibutec/projects/config-repo
spring.cloud.config.server.bootstrap=true

I enabled the Config Server by adding @EnableConfigServer to the Main-class.
I didn't do anything with some bootstrap.properties, as I goy this is only required Spring Boot < 2.4 and I'm beyond.
However, I don't get the client to update it's properties. Whenever I call localhost:8888/message I see my message Hello default
The Config Server itself seems to work fine: Calling curl localhost:8888/a-bootful-client-composite.yml returns
message: Hello, Mibutec

I also tried to refresh the application via actuator, which also does not change anything.
So my questions:

Did I get the idea of "Embedding the Config Server" right? Should it be able to configure my application directly?
Is it correct to remove spring.config.import from the client-config? I got it is replaced by spring.cloud.config.server.bootstrap. I also tried to configure it to the address of my internal Config Server, but than I get while startup Config data location 'http://localhost:8888/' does not exist which makes sense as the port is seems not started yet.

Because the configuration needs to be fetched during bootstrap in order to configure the config server you need to take the properties that you have in application.properties and put them in bootstrap.properties and include spring-cloud-starter-bootstrap on the application classpath.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


Axis range number showing -0.0

When zooming-in, the Axis Range numbering is showing -0.0, as shown below.

[d3.scale.linear] computes the ticks values in that way, I'm quite sure the number is something like -0.000000038923472 but due to rounding it becomes -0.0, I don't want to add special code for this special case on function-plot


README.md: support & feedback: first 2 links are not accessible

isc.devexpress.com is not accessible, documentation can not be viewed.
applies to:

dashboard-extension-online-map-item
dashboard-extension-webpage-item
dashboard-extension-funnel-d3-item

Thank you for your report.
The issue is now fixed.


Quoter

What is the vendor's name?
Quoter
What is the vendor's pricing site?
https://www.quoter.com/pricing/
What is the base pricing? Use the lowest tier that looks sane for a small business customer, not free or personal tiers.
199/month
What is the minimum pricing for SSO support?
299/month
Does this pricing info come from a quote or other non-public source?
all public
Are there any caveats we should list in the footnotes?
No

Hey @bitjbullock ,
as this repository was not maintained for a long time, I decided to start a new initiative under https://ssotax.org/. The data is already way more up2date than in this repository! Moreover I integrated feedback from the community like a Friends of SSO page.
Feel free to contribute there. I will merge changes quickly, I promise!


Fix dto

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- src/users/models/dto/user.dto.ts  1
- src/users/models/user.entity.ts  2
- test/tracks.e2e-spec.ts  1

See the complete overview on Codacy


Vagrant package fails

Currently running vagrant package step for AWS setup fails with following exception:
$ vagrant package SpectrumScale_base --output SpectrumScaleBase.box

==> SpectrumScale_base: Burning instance i-049ac1cd75ae1f46b into an ami
There was an error talking to AWS. The error message is shown
below:

Malformed => AMI names must be between 3 and 128 characters long, and may contain letters, numbers, '(', ')', '.', '-', '/' and '_'

@morazow I can re-create that issue with the CentOS8 AMI and the below Vagrantfile modification to make that work for the prep-ami:
diff --git a/aws/prep-ami/Vagrantfile b/aws/prep-ami/Vagrantfile
index d09f443..0d16917 100644
--- a/aws/prep-ami/Vagrantfile
+++ b/aws/prep-ami/Vagrantfile
@@ -14,7 +14,19 @@ EOT
 load File.expand_path('../../Vagrantfile.aws-credentials', __FILE__)

 # Load RPMs that need to be installed on top of the CentOS base image
-load File.expand_path('../../../shared/Vagrantfile.rpms', __FILE__)
+load File.expand_path('../../../shared/Vagrantfile8.rpms', __FILE__)
+
+
+class Hash
+  def slice(*keep_keys)
+    h = {}
+    keep_keys.each { |key| h[key] = fetch(key) if has_key?(key) }
+    h
+  end unless Hash.method_defined?(:slice)
+  def except(*less_keys)
+    slice(*keys - less_keys)
+  end unless Hash.method_defined?(:except)
+end

 # Customize configuration specific settings

Here is the output I get:
$ vagrant package SpectrumScale_base --output SpectrumScale_base.box
==> SpectrumScale_base: Burning instance i-0a13a3c4a393f54d6 into an ami
There was an error talking to AWS. The error message is shown
below:

Malformed => AMI names must be between 3 and 128 characters long, and may contain letters, numbers, '(', ')', '.', '-', '/' and '_'

At first glimpse I do not see any issue as all given names are conforming to the character set specified in the error message. Will try to dig a bit deeper to understand if this is a vagrant-aws issue.

As a side note, I was able to install vagrant-aws on an Ubuntu 20.04 system after installing
sudo apt install libcurl4-openssl-dev

Versions:
$ vagrant --version
Vagrant 2.2.6
$ vagrant plugin list
vagrant-aws (0.7.2, global)
vagrant-libvirt (0.0.45, system)

Hey @hseipp,
Thanks for looking into it!
Yes, I also could not understand why it fails. Since it says between 3 and 128 characters maybe it does not set the AMI name.
p.s. For now, I manually created AMI from instance, and the rest worked for me with minor issues (commented on #16).

I was able to identify the root cause - it is an issue with this line of code in the Vagrant-aws plugin.
The variable server.tags["Name"] contains a line break and blank characters that are not accepted by the AWS API.
The following patch fixes the vagrant-aws code so that vagrant package works again:
--- /home/user/.vagrant.d/gems/2.7.0/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/package_instance.rb	2021-06-23 13:58:10.612642358 +0200
+++ /home/user/.vagrant.d/gems/2.7.0/gems/vagrant-aws-0.7.2/lib/vagrant-aws/action/package_instance_fixed.rb	2021-06-23 13:59:10.136684461 +0200
@@ -39,11 +39,12 @@
           begin
             # Get the Fog server object for given machine
             server = env[:aws_compute].servers.get(env[:machine].id)
+            servername = "#{server.tags["Name"]}".chomp!

             env[:ui].info(I18n.t("vagrant_aws.packaging_instance", :instance_id => server.id))
-
+
             # Make the request to AWS to create an AMI from machine's instance
-            ami_response = server.service.create_image server.id, "#{server.tags["Name"]} Package - #{Time.now.strftime("%Y%m%d-%H%M%S")}", ""
+            ami_response = server.service.create_image server.id, "#{servername}-Package-#{Time.now.strftime("%Y%m%d-%H%M%S")}", ""

             # Find ami id
             @ami_id = ami_response.data[:body]["imageId"]

Hey @hseipp,
Thanks a lot for looking into it! This is helpful!
Would opening PR in vagrant-aws help? It does not seem to be very active though.

Hey @morazow indeed, getting a vagrant-aws PR accepted would be the ideal way, but looking at the PR queue there I have no hopes that such will happen soon. For now, I'm adding the above workaround to the README.

Hey @hseipp,
Yes, that is fine. I am going to close the ticket.
Best


Takes the test suite up to TOML 0.4

This is a big PR that does the following things:

Takes the suite from what it was before the last wave of PR merges up to 0.4, respecting the order of the TOML spec Changelog. That means that commits containing tests for language features are sorted as those features appeared in the TOML spec.
Tags a specific commit as 0.3, meaning that up to that point it contains tests for all languages features up to version 0.3 of the TOML spec and none of the more recent features.
Change the json comparison function to compare datetimes as datetimes instead of strings.
Adds a git attribute for toml files as "-text" so that git doesn't apply CRLF conversions.

Hi Sergio,
Thank you for the pull request!
My parser is failing the datetime test because it outputs a datetime with the timezone information instead of converting it to UTC for output. If it is not too tricky, could you fix up the datetime comparison to parse the strings into Go datetimes?

Oops, turns out you did do that and I forgot to recompile. The test also exposed some bugs in my toml implementation.
Thank you again! Merged.

Hello,
thanks for merging!
I plan to make a new PR soon to take the test suite up to 0.5, but I'd recommend creating a release for 0.4 as it is now.


Automatically find and include .fsi files, consistently

From fstarlang/fstar-mode.el#13:

I'd argue that it's great that F* found the fsi file automatically. OCaml also finds ".mli" files in the include path (lib directory, current directory), and I would be awful if we had to specify the list of all the .mli files we need when type-checking an OCaml file. I'll file an issue against F*.

So I'd say that this issue is about:

making the behavior consistent between interactive and batch
making sure we don't need extra configuration or voodoo in both these cases, because finding fsi's automatically is great

+1. Different behaviour between batch and interactive mode, and between verifying single and multiple files is source of a lot of confusion.
I don't see why this shouldn't work for .fst files too: look for a matching .fsi file in the include path first, and if there's none look for a .fst file. We probably want to admit .fsi's as if there was a --admit_fsi flag.

Please define "matching" fst/fsi file.

Here's what we did for Mezzo:

if you want to open module Foo, there must be a file whose name is foo.mzi in your include path
the definitions from foo.mzi are admitted when you "open Foo"
it is up to your build system to make sure that foo.mz type-checks against foo.mzi

I'd say having the same behavior would be desirable for F* (replace "mz" with "fs").

Speaking of build system, we had custom rules for ocamlbuild, meaning that you could build a mixed project with ocaml/mezzo files, and ocamlbuild would take care of: making sure mz's match against mzi's, extracting mzi's to mli's, extracting mz's to ml's, and compiling everything altogether.
So as long as you have a good solution for your build system, the recipe above is, imho, pretty user-friendly.

Seems like a nice, simple and usable setup, but quite different than what we currently have for F*, where a single file can export multiple top-level modules and where modules with the same name are defined by multiple files (usually they are replacements of each other). I guess in OCaml you can afford this simplicity by having a more complex module system ... for instance F* modules don't nest AFAIK.

Oh right I was unaware you could export multiple modules from the same file. What about, if file "foo.fs" contains "module bar" and "module baz", considering that it exports "foo.bar" and "foo.baz" (nesting at depth one)?
But then you could no longer switch between files easily in your build system.

There are more subtleties: the matching mechanism should be aware of namespaces. I'd say it's reasonable to map the FStar to F* lib directory. Other namespaces can be inferred from directory names on the include path, or explicitly mapped. Coq uses e.g. Add LoadPath "../contrib/CoreCrypto" as CoreCrypto Coq's LoadPath

The rest of this is now discussed in #438, closing.


Question on BTC fees.

In an omnicored wallet if address X has say MAID coins and address Y has some Bitcoin; when I use omni_send will it use the coins in address Y to pay the Bitcoin TX fee or does it have to have coins in address X? If so, what is the recommended procedure as a service provider when you assign user ABC a deposit address? Delay withdrawals by sending some Bitcoin to the address and waiting for a confirm before performing the omni_send? For payment processing I could see it being even worse since the address would only be used once, it seems like there would always be waste since you would have to add enough Bitcoin to cover the TX fee without knowing exactly how much the fee would be.

Hey @Crypto2

In an omnicored wallet if address X has say MAID coins and address Y has some Bitcoin; when I use omni_send will it use the coins in address Y to pay the Bitcoin TX fee or does it have to have coins in address X?

Omni Core will not use a different address for fees if the sending address has no BTC I'm afraid - Omni identifies the sending address by the inputs, so if we only used the inputs from address Y then the Omni transaction will be parsed as being sent from address Y (and not address X).

If so, what is the recommended procedure as a service provider when you assign user ABC a deposit address?

This is an interesting question, and if you don't mind I'll get a little technical on exactly what we're doing in this regard...
A little while ago we activated a new class of transaction (class C) which moves away from our old encoding mechanisms to instead use nulldata (OP_RETURN) outputs.  As part of this transaction class we also updated the sender handling so that the first VIN is always considered the sender, and any remaining inputs are ignored (from a sender identification perspective).
As such with a Class C transaction, only the first input has to come from the sender, and the rest can be from any other address.
When we apply that logic to the service provider deposit/sweep/withdrawal consideration, we can see that:

A user sends some tokens to DEPOSITADDRESS.  As part of this (and any Omni) send, a small amount of BTC (a touch over the dust threshold) is also sent to DEPOSITADDRESS for recipient identification.

Service provider attaches the output from the users' send in step 1 as the first input to a sweep/withdrawal transaction thus identifying DEPOSITADDRESS as the sender.

Service provider attaches outputs from any other transaction/address as additional inputs to this sweep/withdrawal transaction to pay the remainder of fees.

The idea here being that we can always use fees from other addresses as long as there is at least one input to attach to the transaction as first VIN (which there is if the address has received a send).
Hope that helps provide a bit of a base, and please don't hesitate to ask any followup questions.
Thanks :)
Z

I should also note that this is considered advanced usage and not currently done natively / automatically by the Omni Core wallet tx builder (eg omni_send and friends) but can be used via the raw tx interface (omni_createraw commands).

Are there any plans on adding that to Omni Core so it does it automatically? Possibly with a new RPC omni_send parameter or omnicored -option to not break backwards compatibility.
Of course I'm just a johnny come lately and only have a cursory understanding on the Omni Layer technical details, I was just thinking as an option instead of using the vin to determine the sender the Bitcoin address could be included in the the data (20 or 21 bytes I think) and signed by that address as proof of ownership. I guess this would substantially increase the amount of data embedded in the blockchain though, it has probably already been thought of and rejected for that reason.

Are there any plans on adding that to Omni Core so it does it automatically? Possibly with a new RPC omni_send parameter or omnicored -option to not break backwards compatibility. It seems like it would help a lot of people and make entering the Omni world easier.

There are indeed plans to add this functionality to Omni Core, but as with most projects it all comes down to prioritization.  This is functionality that would help integrators I completely agree and I'm very supportive of adding it sooner rather than later.

.....I was just thinking as an option instead of using the vin to determine the sender the Bitcoin address could be included in the the data (20 or 21 bytes I think) and signed by that address as proof of ownership.....

A signature is roughly 70 bytes (give or take) so in addition to the HASH160 of the address we're at 90+ bytes just for the sender identification.  Nulldata (OP_RETURN) outputs are currently limited to 80 bytes via default Bitcoin Core policy so it's probably not a strategy we'd look at, but thanks for the suggestion!

@zathras-crypto could you provide an example on how to populate the rawtransaction field for omni_sendrawtx method. I'm guessing one would use omni_createrawtx_opreturn with rawtx = null(?), but what should go in payload field?

could you provide an example on how to populate the rawtransaction field for omni_sendrawtx method. I'm guessing one would use omni_createrawtx_opreturn with rawtx = null(?)

@dexX7 I'm sure I've seen gist workups from you with examples for the raw transactions interface - am I remembering correctly?  If so can you please link for @bumbar1 ?  Otherwise we should do a guide for this.

...but what should go in payload field?

There are omni_createpayload calls but they hadn't been documented yet.  Here you go https://github.com/OmniLayer/omnicore/pull/385

I'm working up a patch right now, it's not super pretty but it's a starting point/proof of concept doing basically what you were talking about before. Basically was have one address, the Funding Address (FA) that holds BTC to pay the TX fees for all the transactions. The reason for a specific funding address is so it doesn't use inputs from addresses that have Omni Layer coins randomly and mess up sends from those later.

Modified the Omni SelectCoins function so it grabs only one input from the Sending Address (SA) and grabs anything else it needs from the funding address. (To minimize waste of SA inputs for later sends.)
I changed Coin Control to use a vector instead of set to maintain the order of the inputs, then in CreateTransaction it makes sure that the 1st item in the vector is the 1st in the new transaction.
Also in CreateTransaction, if there is more than 0.00005 BTC change it returns it to the FA and gives the small remainder back to the SA so if there is another send later it has an input to use.

Is there any importance to the order/amount of the outputs in the Bitcoin transaction? I know by default Bitcoin places the change randomly and that doesn't seem to have been altered.
Here is an example successful test with 12JYmnfYU2ghzjwUAspzJsSnmJtK9bZPYR as the FA:
https://blockchain.info/tx/2bf9592c84ec6c76a8cbeaf620de0545e6afe6fce046c3607daf1bbba0f0bf4c
http://omnichest.info/lookuptx.aspx?txid=2bf9592c84ec6c76a8cbeaf620de0545e6afe6fce046c3607daf1bbba0f0bf4c

I'm working up a patch right now

Awesome!!! I'm impressed how quickly you've got up to speed! Thanks for contributing! :)
The model seems good and I like the idea of an explicit funding address (I'm all for explicit behaviour).  We could get away without it by first evaluating the tally when selecting inputs to check if an address holds Omni properties & if not, consider it safe to use for an input - though I'm +1 on the explicit address if @dexX7 you're cool with that too?
The functionality could be exposed via a configuration call, for example:
omni_setaltfeesource false configures to use original model (default)
omni_setaltfeesource true 1BitcoinAddressToUseForFees configures to use new model with specified FA
I'd be very interested to see what you've done when you're happy to share :)

Is there any importance to the order/amount of the outputs in the Bitcoin transaction? I know by default Bitcoin places the change randomly and that doesn't seem to have been altered.

Actually yes - reference identification (the recipient) removes outputs to the sender as change but since the change output to the FA is not from the sender, it likely won't be detected as change.  The fallback in reference identification is always highest vout, so best bet is to always put the reference address as the last vout and then multiple change addresses shouldn't be an issue.

Here is an example successful test

That's fantastic :)  Though if I may give a quick suggestion - create yourself a property in the test ecosystem (or ask one of us to send you some Test Omni) and use those for your test transactions - even though it's unlikely I'd hate to see you lose any funds while testing different ways of building transactions etc.

Yeah, if someone wants to send a few Test Omni to 1GP8Stw1G6bX4AG55fayx8DoKxr71jFBaK that would be cool.
I updated it now so it will always put the FA and SA change(s) right after the payload so the reference address will always be last: https://blockchain.info/tx/ec3726be944e84ea8212f12b6a5e4f9560257571d7ba29de706637372d356b0b
I created a PR at https://github.com/OmniLayer/omnicore/pull/388 (mainly for easy readability of multiple commits.) Fair warning to any non-Omni devs: I've only sent a few test transactions and only used omni_send. Needs more testing and review before production use.

Hey guys,
I'm really happy to see more action, and thanks a lot for your contribution @Crypto2!
I generally see three options:

Transactions are manually created raw, with a combination of omni_createrawtx_input, omni_createrawtx_opreturn, omni_createrawtx_reference, omni_createrawtx_change, and the new payload creation calls. This can be done right now, but it involves fiddling with raw transactions on a lower level. I can provide an example in the next days.

We add a new set of API calls to facilitate the external funding explicitly, e.g. in form of omni_funded_send, omni_funded_xxx, ... When using these calls, the transactions are created by Omni Core, and funded by a specified source.

We use a global funding source, which can be configured, as proposed in the earlier posts and #388.

I could imagine crafting all transactions manually could be a bit annoying, but I could also imagine having a global funding source might be too inflexible for some cases.
Given that this feature was also requested by Tether, it would be interesting to learn more about the specific use cases.

The main reason I went with #3 which required mods to Bitcoin's CreateTransaction was because you don't know what the fee is anymore until then. Although #2 could be implemented the same way, just a little more work since you'd have to add parameters to several functions instead of setting a global variable. Depending on how multithreaded your application is being able to change the funding address on the fly via RPC in #388 does add some flexibility, in the end having omni_funded_send may be the way to go though.
I know manually crafting every transaction is a pain I don't want to bear :)

Awesome, thanks for posting the code @Crypto2 :)
I think option 1 is a no-go - we definitely want a raw tx interface to allow people to get creative, but alternative funding for fees is something many folks ask for, so it makes sense to make it as painless as possible.
Since the funding address can be changed during runtime I don't see a huge lack of flexibility here, but it does preclude the use of multiple funding addresses at the same time which might be an issue.
I think I'd agree it'd be great to understand where Tether stand on this - @CraigSellars can you add anything here?

@CraigSellars can you please feedback on Tether requirements?

So far so good with the code as-is; not exactly high volume but no problems so far.

@zathras-crypto and @crypto2 this is quite fantastic.  We have a large integrator who needs this functionality (managing the deposits for a huge number of users is far too cumbersome).
I am in favor of a modification of #2 and #3 as described… a config option that allows you to specify a funding address (and uses it when necessary), but I don’t think we need a new API call - I was thinking an alternate paramater for funding address at the end of the list of attributes would suffice.
If we have a working version of this, there are integrators who would use it immediately - what is the current status of this patch?

As far as the status we haven't had any trouble so far. The only major edge case left is if there aren't any inputs from the sending address to use it will still fail even if there are plenty in the funding address, no real way around that unless we auto generate another TX to the sending address and then use it in the new transaction.

Quick update nearly a year on, my patch is still working great with no issues. Not sure why it was never merged and integrated?

Your timing is perfect @crypto2 :)
Firstly please allow me to apologize that this got forgotten about - we have a small development team and are often attacking many things at once which sadly can cause issues to lose focus.
I actually raised with the team not two weeks ago that baaed on feedback we needed to assist our integrators with two functions; central fee addressing and batch sends (multiple sends in a single transaction).  There was unanimous agreement to shift focus to these two items next.
I'm just finishing up a rewrite of sending in the UI for our Omni Lite prototype after which I'll be looking at these items (central fee/batch withdrawals) & will definitely start with your PR.
TL:DR; great to hear your patch is working & be back to you re refining/merging soon :)

Any updates on this? Unfortunately with the latest hardfork/mandatory update I'm having to try to update the patch. Seems to be working for the most part but isn't signing the TX properly somehow; haven't figured that out yet even though the signing code is untouched.

Actually, while this still would be nice to have in the core daemon I ended up just doing the same thing on my end with the payload functions. Took less than an hour to do the same thing versus hacking on the wallet :)

Hey @Crypto2, I'm very sorry for breaking your implementation! How did you end up doing it? With the raw transaction build calls?

No worries :) Yep, I reimplemented with the raw transaction calls.

I have issues using method omni_funded_sendall.
For example I have balance 7.7 propertyid 1 on one address and minimum BTC (546e-5). I'm funding from another address that has 5+ BTC balance and plenty of propertyid 1 (even though none is necessary). But whenever I try to transfer I get invalid reason "Sender has no tokens to send".
On top of that the method doesn't work at all if the from address (which has 7.7 - not a funding address) has 0 BTC balance. In that case I get "Error choosing inputs for the send transaction".

I think there is a misunderstanding:

But whenever I try to transfer I get invalid reason "Sender has no tokens to send".

Tokens are only spent from the very first address.

On top of that the method doesn't work at all if the from address (which has 7.7 - not a funding address) has 0 BTC balance.

This is expected, because at least some BTC is needed.

I published instructions for the new funding RPCs:
https://medium.com/omnilayer/introducing-funded-omni-layer-transactions-or-how-exchanges-can-save-money-f3f04161a6af


Feature Request: change 'ONGOING' and 'NEXT CLASS' automatically without reload

It'd be nice if the timetable automatically update those two fields without the user having to reload.

the main.js file does all the stuff. it calculates the current time and figures out which slot the time lands in and thus the current and the next class. Is there some way to reload the script say every 5 minutes?

no way to do something like
while(1){
    print_table()
    sleep(5 mins)
}

in js?

That could be done but as far as I know most browsers make webpages static to save RAM.
An auto refresh command could come in handy but it needs testing

Tried something along these lines, works on my machine. merged the PR. Can you test for 2-3 days or so and close if this feature works fine for you now?

lgtm

Fixed by #9


I only get a random image being render everywhere with random sizes

changed https://github.com/juangallostra/augmented-reality/blob/master/src/ar_main.py#L20 to 200 and helped!

I find the tracking not very smooth, don't you apply any filter so that its a bit smoother the tracking of movement?

@pmgou yep, this is a known issue. Currently there is no tracking being applied, the reference surface is detected each frame without taking into account information from the previous frame.
Tracking is something that is on my to-do list though. I hope to find some time to work on it.

I get the same issue. Did you change the camera_parameters in https://github.com/juangallostra/augmented-reality/blob/19a49c47982df9009c7f6ec06a0eb3613050562b/src/ar_main.py#L29? Is there any necessity to do this when I use my own model and video?

@WiiliamC changing the camera parameters might help. However, I would expect it to work with the default parameters included in the code. I suggest to check each step of the process individually to try and find were the problem is.
The first thing I would check is how good the matches between the reference surface and a video frame are. If the obtained matches are not accurate that might be the cause of the issue. If they are, we can jump to the next step of the process.


No documentation

On readdocs there no configuration at all. so not possible to see what options are required

Strange, there was documentation. I'll investigate what might have happened.

FYI here is the documentation for the last release - http://django-security.readthedocs.io/en/0.9/

@aldarund my coworker spotted the bug: 5f8695104e5dcc4c1f9d04f73de4b5120c620aba


[Feature] Serde Serialization for U256 etc.

Component
primitives
Describe the feature you would like
This works:
#[derive(Debug, Serialize, Deserialize)]
pub(crate) struct TokenMint {
    pub token: Uint<256, 4>,
}

But this doesn't:
#[derive(Debug, Serialize, Deserialize)]
pub(crate) struct TokenMint {
    pub token: U256
}

It gives: the trait Deserialize<'_>is not implemented foralloy_primitives::Uint<256, 4>`
I'm sort of a rust beginner; but it seems I can't really implement those traits. myself either due to the orphan rule.
Additional context
No response

hmm, did you enable serde feature?

Oh no! I thought i checked for a feature, but somehow missed it. Thanks! 🙏


Update get-started.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Deploy preview for docsdocker ready!
Built with commit 67f6fed85981184010fb86696802fef514d21b17
https://deploy-preview-8713--docsdocker.netlify.com


⚠ gorzdrav.org has degraded performance

In 5ee229a, gorzdrav.org (https://gorzdrav.org) experienced degraded performance:

HTTP code: 200
Response time: 2235 ms

Resolved: gorzdrav.org performance has improved in 1e68ad8.


hidden file

Hello：
I used my own data set, the following error occurred;
FileNotFoundError: No such file or no access: '/home/zhengsc/Desktop/unet/3D/Task100_Colon/./imagesTr/colon_212.nii.gz'
I really don't have this hidden file.
I looked at the contest data and he had a hidden file.

How do I generate this hidden file？？

Hi @wxr521314
Can you verify if you have the file colon_212.nii.gz'
Try
ls -l /home/zhengsc/Desktop/unet/3D/Task100_Colon/./imagesTr/colon_212.nii.gz


Add notes on hosted PostgreSQL such as RDS, Heroku

As per this chat: https://gitter.im/debezium/user?at=59a090ee9acddb2407cc16d9

Thanks, @aviflax!


feat: added open dot arrow feature

started fulfilling request from #3167 to add more arrow ending shapes

Hey @caroleenchang, looking good, but we'll need to figure out and fix a few things:

I think it'll be better if the circle isn't intersected by the line. Not sure how to go about it, but probably the best way is to terminate the line earlier instead of pushing the circle further out, so as to ensure all the lines (well, especially the lines ending in circles) are visually the same length (below, the red line indicates where the line should terminate):

The arrowhead should be on the other end for this menu:

With this new arrowhead we're already hitting the width limit of the menu and in some circumstances it results in a horizontal scrollbar. We can either start wrapping the buttons onto a new line, or preferably make the arrowhead menu (<Popover/>) render via React Portal outside the menu container so it can overflow it.

Are there other arrow heads under consideration? Visually, it may be better to add these in sets of four (to preserve the 4xn grid) than waiting until we have eight to create a second row.

There were more ideas linked here https://github.com/excalidraw/excalidraw/issues/839 and discussion https://github.com/excalidraw/excalidraw/issues/616.

Yeah, we can wrap.
What about (1) https://github.com/excalidraw/excalidraw/pull/3477#issuecomment-826073450?

where are we with this?


(PUP-10716) Fix show action output formatting

Description of the problem: when puppet facts show action is called without indicating the render, the default render is json which is not pretty printed.
Description of the fix: The default render should be console (the default for find action) and in order to display the facts as a pretty json a new method (when_rendering :console) was implemented.

Verified default is pretty printed json:
$ bx puppet facts show ruby
{
  "ruby": {
    "platform": "x86_64-darwin18",
    "sitedir": "/usr/local/Cellar/rbenv/1.1.2/versions/2.7.2/lib/ruby/site_ruby/2.7.0",
    "version": "2.7.2"
  }
}

But it's possible to render as json or yaml:
$ bx puppet facts show ruby --render_as json
{"ruby":{"platform":"x86_64-darwin18","sitedir":"/usr/local/Cellar/rbenv/1.1.2/versions/2.7.2/lib/ruby/site_ruby/2.7.0","version":"2.7.2"}}
$ bx puppet facts show ruby --render_as yaml
---
ruby:
  platform: x86_64-darwin18
  sitedir: "/usr/l


Add dagger and dependency injection

This PR adds Dagger and converts most basic flow (sending notifications) to dependency injection. There is still a lot to do, but I feel this is good enough first step to merge it in and continue from there.
Beginning of https://github.com/crc-32/fossil/issues/12
Depends on https://github.com/crc-32/libpebblecommon/pull/1

If you agree, I will create bunch of issues on the repo on what still needs to be done to complete this change once this PR is merged.


Remove unneccessary vars generation

In the second ops file, the /variables do not need to be touched. The only 2 cases where they could be not there would be:

You run Step3 before step 1
You forget to remove the ops after step 5

This way in both cases the bosh deploy errors and will not cause all VMs to lose connection to BOSH. We internally use it like this for quite a while now and it prevented some incidents already so I think everyone should have that.

@jpalermo The only thing i remove is that the ops file will not generate the var here. If you don't persist the new nats CA then you are doomed anyways.
I kept the nats_2 in the ops for the filling out of templates. It loads fine from the bosh-secrets.yml unless its missing, which only happens in the 2 error cases I mentioned in the first message.

Oh yeah, my brain totally forgot about the vars file. 👍

Thanks! @schmidtsv


Throw exception early if  is required but not configured

Codecov Report

Merging #357 into develop will increase coverage by 0.48%.
The diff coverage is 62.5%.

@@             Coverage Diff             @@
##           develop     #357      +/-   ##
===========================================
+ Coverage    55.03%   55.51%   +0.48%
===========================================
  Files           62       62
  Lines         1819     1823       +4
  Branches       207      210       +3
===========================================
+ Hits          1001     1012      +11
+ Misses         752      743       -9
- Partials        66       68       +2

Impacted Files
Coverage Δ

.../maven/artifacthandler/ZIPArtifactHandlerImpl.java
64% <50%> (-4.19%)
:arrow_down:

...ure/maven/artifacthandler/ArtifactHandlerBase.java
80% <75%> (+1.42%)
:arrow_up:

...crosoft/azure/maven/appservice/DeploymentType.java
45% <0%> (+45%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 587a9cc...fff3671. Read the comment docs.


V1.3 GUI

I have refactored the GUI into the following.

There are two motivating factors for this:

Our commands are long. It would be good to have a tall output box to show the example syntaxes. So instead of having a long unreadable line of Example: add mn/Aspirin d/1 f/Daily sd/01/08/2023 ed/25/12/2023 exd/01/01/2024 ts/100 n/Test note, we can break the parameters down into lines and display on the LHS. These line breaks have not been implemented yet.
To allow for better readability of the prescription, I have broken the list down to the RHS and made it (hopefully) cleaner.

The consumption will automatically update to red or green depending on the status.
I have also added more sample data into SampleData.util to test the scroll bar.
I need yall to help pull this into your local and try run the GUI and test. JavaFX doesn't inherently support responsive UI that adjust properly to different screen sizes so I tried my best to work around it already. As an intended feature, you cannot resize or move the application at all. Everything is scaled to the centered and resized to your screen size upon start up.

We can ignore codecov in this case right? How do we even test the UI.

We can ignore codecov in this case right? How do we even test the UI.

Yup seems like other groups also not testing for UI.  Writing some test cases for other files to make up for the codecov


Mockup: Download feature

This issue was moved to OrgGrupoProgrameiros/Ambrosia#3

This issue was moved to OrgGrupoProgrameiros/Ambrosia#5


Adds Bootstrap 4.3.1 display template

Hello and thanks for the great library!
There were some minor style tweaks needed for Bootstrap 4.3.1, so I've added a display template to the repo.
Thanks,
-Nick

Merged, thank you very much!


signalr alpha 2: how to deserialize from camel to pascal with messagepack

I am having the following problem:
I have a typescript FE and a .net core backend.
my typescript objects are camelcase and my .net core objects are pascal case (the standard way for both languages).
When I use the JsonHubProtocol the ts/js object is deserialized properly to my POCO object in .net and also nicely serialized to camelcase when sending from .net to my ts frontend.
However, when I am using MessagePack this is not happening.
with
services.AddSignalR(o =>
{
    o.MessagePackSerializationContext.DictionarySerlaizationOptions.KeyTransformer = DictionaryKeyTransformers.LowerCamel;
});

I can make sure the POCO object is transformed into camel case. However, I can't find any way to do it the other way around. So having a camelcase ts object that will be mapped to a pascal case POCO object automatically.
Any thoughts on how to achieve this?

@anurse will take a look.

Looks like this was an issue in an older version of MsgPack-Cli that we were using. I'm updating to 1.0.0-rc and it seems to have resolved the issue. PR pending


Higher kinded classes

This implements what is proposed in https://github.com/tibbe/hashable/issues/114. It is not yet complete. I still need to add more instances. There are two deviations from the original proposal:

The method liftHash is not included. The reason for this are found in the documentation in Data.Hashable.Lifted.
The Hashable2 class was included as well. This is mostly for consistency with other packages that provide higher rank typeclass variants.

In some of existing Hashable instances, I call hashWithSalt1 to avoid duplicating code. You can look at the instance for [] to see this. I do not believe that this will hurt performance because the inliner should expand these.
The example instances I provide in Data.Hashable.Lifted have not been typechecked and are likely broken. I'll get to that soon.
Let me know at this point if there are any concerns so that the can be addressed before I write all the remaining instances.

Generics support for Hashable1?

@RyanGlScott I knew you would show up ;) Now that I think about it, that would be useful (especially if I'm working with Fix Expr, and Expr and I keep changing Expr). I'll attempt to implement this, and I would appreciate if you would review it.

I'll attempt to implement this, and I would appreciate if you would review it.

Sure thing! I recommend using a trick like aeson's ToArgs as a way to reuse the code between the code for the Generic- and Generic1-based machinery.

@RyanGlScott Thanks. That's a really good trick. I was just about to duplicate everything.

Generics for Hashable1 is now ready. Also, does anyone know why hashSum even takes size as a parameter? It looks like there some bitshifting arithmetic that happens on it, but ultimately, it's never used. Here it is as it currently exists: https://github.com/tibbe/hashable/blob/a5bb09f8ecd175e62118258ffd30ac4afaf35530/Data/Hashable/Generic.hs#L43-L61
I'm guessing that it's leftover from a different approach to hashing sums that was removed.

@andrewthad I think you're missing a GHashable case for (:.:) (i.e., how would you generically handle something like newtype Compose f g a = Compose (f (g a))?)

@RyanGlScott Thanks for the feedback. I've address both of your comments.

Not sure what's causing the build failure. The failing line is:
Preprocessing test suite 'tests' for hashable-<IP_ADDRESS>...

: cannot satisfy -package-id hashable-<IP_ADDRESS>-inplace:
hashable-<IP_ADDRESS>-inplace is shadowed by package hashable-<IP_ADDRESS>-85f768408d56687a048b7dee7b04a672
Other than that, I still need to add tuple instances. Any other thoughts or concerns?

Thanks for clearing that up. I've wondered about test suites that use other-modules like that before. Definitely won't try to fix that in this PR.
I'll go ahead and get the tuple instances ready.

Ben Gamari was able to help look into the differences in the generated core, and the new generic implementation of hashWithSalt surprisingly results in fewer allocations. I've added benchmarks the generic machinery. Here are the results:
---------------------------
NEW GENERICS IMPLEMENTATION
---------------------------
> cabal bench --benchmark-options='Generic'

Preprocessing benchmark 'benchmarks' for hashable-<IP_ADDRESS>...
Running 1 benchmarks...

Running 1 benchmarks...
 I think that I still need to correct some of the examples I gave in the haddocks.
@tibbe What were your thoughts on my response concerning Hashed?

Just wanted to check in and see if there is anything I can to do keep this PR going forward.


Normal users can't log in.

As it currently stands, normal users can't log in.

Fixed login issue with normal users.


Support multiple outputs for custom factors

Add an optional outputs parameter to CustomFactor allowing multiple outputs to be computed and returned from a single CustomFactor, each output of which is itself a Factor.

Coverage increased (+0.04%) to 84.308% when pulling 233602ce7d3c1a8bcc1099bf68520c27ee7735ea on multiple-factor-outputs into 675eff4f6d17345a159002b3cfe9c39d004e05af on master.

Coverage increased (+0.04%) to 84.309% when pulling 835af4eb7a6491611e0161ff33602427ed484e7f on multiple-factor-outputs into 675eff4f6d17345a159002b3cfe9c39d004e05af on master.

Coverage increased (+0.003%) to 84.274% when pulling 3fc69c1c2512b841de134382e958c49eb7a1d9f2 on multiple-factor-outputs into 3f2a5746f8364f4f1a09ccf1a3e092ce8ae1dc65 on master.

Coverage increased (+0.003%) to 84.274% when pulling 3fc69c1c2512b841de134382e958c49eb7a1d9f2 on multiple-factor-outputs into 3f2a5746f8364f4f1a09ccf1a3e092ce8ae1dc65 on master.

Coverage increased (+0.03%) to 84.297% when pulling d21ece4888ecdecac35c9dc395eb305e8222a49f on multiple-factor-outputs into 3f2a5746f8364f4f1a09ccf1a3e092ce8ae1dc65 on master.

Coverage decreased (-0.008%) to 84.263% when pulling a872419f43e4041f5745cefb876a108a868eab60 on multiple-factor-outputs into 3f2a5746f8364f4f1a09ccf1a3e092ce8ae1dc65 on master.

Coverage increased (+0.03%) to 84.297% when pulling a872419f43e4041f5745cefb876a108a868eab60 on multiple-factor-outputs into 3f2a5746f8364f4f1a09ccf1a3e092ce8ae1dc65 on master.

Coverage increased (+0.06%) to 84.294% when pulling 9fce2f87861ddd8e466f7d854dd20c0f42280d1d on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

Coverage increased (+0.09%) to 84.323% when pulling 6bf612bd657cc7e4a56b0aac8f3aab84483fbdb6 on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

Coverage increased (+0.09%) to 84.323% when pulling bfdb64822aadce6c8b35f1acc16bad0bf4cda16e on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

Coverage increased (+0.02%) to 84.252% when pulling 7df8544abb80aa480feb3cc8deb514e1cdbbc69e on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

Coverage increased (+0.01%) to 84.245% when pulling fb0d5214b45d6c55f5c44f00cf67216ca06ab6c3 on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

@ssanderson Ready for another look

Coverage increased (+0.01%) to 84.245% when pulling f525e6acce7e989d22890984a366a02b872d5fac on multiple-factor-outputs into d2ab5ed46ec579c72a92c69bee3d8decf2240886 on master.

Coverage increased (+0.007%) to 84.056% when pulling abbec1e1e1c04231914014cef5938b25967f9a99 on multiple-factor-outputs into 620d2a8b2294986003940ace7a06e019bdc911f9 on master.

Coverage increased (+0.06%) to 84.08% when pulling 05891bd1cab08f87735a62a802329b7639656d3e on multiple-factor-outputs into a5f7fc7d6d3cb01d3df21e6e968332727fde54eb on master.

@ssanderson Should be ready for another look

Coverage increased (+0.05%) to 84.245% when pulling ec68d04d2ee656412b106a1ec31fe18df2863ed5 on multiple-factor-outputs into 842c47b32861ca5d264ab5f0b2c7031aa3b27906 on master.

Coverage increased (+0.05%) to 84.243% when pulling c4d190cb867c9316a2a11f7d6522fd03ec53278f on multiple-factor-outputs into<PHONE_NUMBER>3009a38a7500324331c0d6fe91ac62 on master.

LGTM.  @dmichalowicz feel free to merge when appveyor finishes.


Fix AttributeError bug caused by using id in skip_row instead of pk

Problem
Crash caused by the problem described in this issue https://github.com/django-import-export/django-import-export/issues/1435 . A recent commit uses the id field of instances instead of pk which causes crashes.
Solution
Use pk field which universally works to designate the primary key of an instance, instead of id
Acceptance Criteria
I have added a comment to make sure no one introduces a regression on this.
Writing tests for this would require significant additions to the test infrastructure (adding a model with a different pk than id and associated Resource). I don't have the time resources to do this.

I added a test for this in the 'release-3-x' branch.  Feel free to add your name to AUTHORS and I will merge.

I've applied this fix (along with test) to release-3-x.  Thanks for the PR.


fix: test search correctly

close #19
test_read_record_with_different_type_of_redirected_pid is now failing on my machine... It actually raises the exception instead of serializing to a 500. I know pidstore and error handling have been modified recently. What am I missing?

Can be re-reviewed now.


Minor issues

Thanks for the great directive.  I've played with your library somewhat and found a few issues that I've fixed.  However, unfortunately my corporate firewall doesn't allow me to submit a PR, so I will highlight them here:

'rg-top' has an incorrect 'left' value.  'left' should change from 50% to 0.

Dragging can cause the selection of elements in the panel.  This can be disabled by adding the following to the bottom of dragStart:
         if(e.stopPropagation) e.stopPropagation();
         if(e.preventDefault) e.preventDefault();
         e.cancelBubble=true;
         e.returnValue=false;

If you have height or width animations on the resizable div, it won't work smoothly.  To fix this, add a no-transition class to the css:

.resizable.no-transition {
transition: none !important;
}
Then toggle it on in dragStart, and toggle it off in mouseup using the element.add/removeClass();

Gotta love corporate firewalls... Thanks for doing the hard work @jinder!
The issues have been fixed using your suggestions on the latest commit. Here they are in a live test as well http://codepen.io/Reklino/pen/yymMWN?editors=101


Delayed resolution

Generalize the delayed resolution of language items to arbitrary items.

#[extern] currently doesn't parse as an attribute because extern is a keyword

Another application:
An rng that needs to be initialized from another rng. In a freestanding environment there might be no standard rng available and so the first rng cannot be initialized from a default source and thus it cannot implement Default and thus it cannot be used as a default argument for a hashmap in a freestanding environment, etc., etc.

liblog could also really use this. It currently gets around this with boxing, trait objects, and at_exit; with this it could just use core https://github.com/rust-lang-nursery/log/issues/13 properly.
Should it work on functions too so we can replace more lang items with this?

Should it work on functions too so we can replace more lang items with this?

Everything can be simulated with types, static functions on those types, associated constants and statics or const fns and functions returning mutable references.

If A declares a public #[extern] type, B defines it (and depends on it of course), does B see the concrete type when it looks at A

#[extern]
type T: ToString;

#[extern = "T"]
type _Dummy = String;

When B looks at _Dummy it sees String. When it looks at T it sees some type that implements ToString.

Can #[extern]-declared types be used in the definition of #[extern = ...] definitions?

I don't see why not. At worst the compiler might crash.

What about specifying defaults? For example in the case of the allocator, I would expect most dependants don't really care about what is chosen.

I'd like to propose we move this RFC to final comment period (note that there is a change to the RFC acceptance process here to get more consensus from the team before moving to FCP).
Summary: the RFC proposes a mechanism for abstracting (at compile-time) the values of global variables, type aliases, and functions. Where these items are declared, they are only given bounds, rather than a concrete value and type checking, etc. is performed using these bounds. A downstream crate must provide the concrete value of the item, which is checked against the bounds. The motivation is as a more general alternative for mechanisms for supplying allocators, etc.
There has not been much discussion, you can probably just read it, rather than me summarising.
My opinion is that there is the seed of a good idea here, but the motivation is not strong enough for such a big feature, nor has there been enough of a positive reception to warrant further investigation for now. Therefore, I would like to propose that we move to FCP with a view to close this PR.
@rust-lang/lang members, please check off your name to signal agreement. Leave a comment with concerns or objections. Others, please leave comments. Thanks!

[ ] @nikomatsakis
[x] @nrc
[ ] @aturon
[ ] @eddyb
[ ] @pnkfelix (on vacation)

I recall this RFC being a bit underspecified, but I think the motivation is quite good. In various fora it's been discussed how at least log, panicking, compiler-rt/builtins, and allocators could benefit from this. Thats quite the list.

I agree that we have evolved a number of systems that seem to have this kind of "needs-require" relationship. However, I think that before accepting an RFC around this feature we'd want to address a number of interactions.

How/if this can be used to subsume allocators, panicking, and builtins.
A bit more thought into how this relates to the trait system. These items are basically associated items from some global trait -- maybe we can make that link more explicit? If nothing else, giving a "desugaring" to a trait mechanism might be nice.

In general, this direction of "global needs/requires" worries me, because it is so clearly non-composable. (The same holds for the existing instances.) In other words, if I have a library that supplies type X, I cannot link it with another library that does the same. Do we want to mitigate that risk in the design somehow? I think we should also discuss how we will describe the purpose of this in order to clarify that, basically, people shouldn't be using it very often.
Along those lines, I am very interested in pursing a kind of "functors-lite" approach that addresses similar problems (and maybe even serves as an alternative). The basic idea is that you can parameterize modules and crates with type/lifetime parameters; this could effectively be desugared to threading those parameters around on all the modules contained within, so it is not equivalent to (e.g.) ML functors, and in particular not "generative functors". (I should sketch out my preliminary thoughts here, I keep alluding to them; ah well, I'll do it again for now.)
An advantage of this trait-based approach would be better composability. A disadvantage would be that it doesn't necessarily apply to panic handling and other global codegen properties. Anyway, even if we wanted both, I think I would prefer to have some kind of appealing trait-based alternative in place before we offer these convenient, but non-compsable, global items as an alternative.

@nikomatsakis A side note regarding a "needs/requires" -kind of system: A way to globally have orphan impls without coherency issues by using such a system was also proposed: https://internals.rust-lang.org/t/another-proposal-for-overcoming-orphan-impl-limitations/3863 However, as you say, the globality makes such schemes non-composable.

Ah another usecase I forgot to enumerate a better foundation of main and entry points---libraries (std, "frameworks") can declare the entry point and then call a "needed" function, or provide that needed function and need another.

@nikomatsakis I love applicative functors too! We could still handle the truely global things by making some crates only includable once in the dependencies dag, even when applied to different parameters. My biggest worry is this will require a lot of Cargo work to get right---though even plain needs-provides still needs some Cargo work.

In addition to the other points made in comments above, relying on the attribute system to express (path-based) references a la #[extern = "alloc::Default"]  is going to run afoul of the same problems noted in other RFCs like https://github.com/rust-lang/rfcs/pull/1605#issuecomment-229434796 and https://github.com/rust-lang/rfcs/pull/1327#issuecomment-149302743 , right?

The cases are slightly different since here I think that #[extern = Foo] is mean to be providing the definition for Foo, while in the other two cases I listed above, the attributes are referencing paths that are defined elsewhere. That actually might mean that e.g. hygiene is not an issue? I am not sure.

🔔 This RFC is entering its final comment period. 🔔
The lang team's disposition is to close.

No comments since the end of the FCP. Per the summary comment, the feeling is that there is a good idea here, but that idea would require more elaboration before we could accept it (as described in the comments that come after that summary comment).

https://github.com/ghc-proposals/ghc-proposals/pull/5 is highly relevant for the applicative functors approach.


Update from commonjs to esm project

I expect more and more modules will have this as a requirement (eg: chai 5.x, chai-as-promised 8.x)
However it looks like making this change would require making all imports to have explicit .js suffix, which is kindof gross.
https://mikebian.co/converting-a-node-project-from-commonjs-to-esm/
eg https://github.com/chaijs/chai-as-promised/issues/301

Why? https://electerious.medium.com/from-commonjs-to-es-modules-how-to-modernize-your-node-js-app-ad8cdd4fb662

Ugly, but forward-looking PR merged.


Percentage metrics can result in "unsupported operant type(s) for /: 'decimal.Decimal' and 'float'.

For some datasets, using percentage metrics results in the error: unsupported operand type(s) for /: 'decimal.Decimal' and 'float'.
I'm using Postgres, and I've traced this back a few ways -

It doesn't happen if the column is of type INT (under dataset/columns/Data Type).
It does seem to occur if the column is of type BIGINT or NUMERIC.
It appears to come from: (https://github.com/apache/incubator-superset/blob/master/superset/viz.py#L780)

Changing that line to (df[percent_columns].astype(float).div(df[percent_columns].sum()).add_prefix("%")), seems to fix the issue for me, but it may have ramifications I'm unaware of (I'm not very familiar with this portion of the code-base).
Disclaimer: This is on yesterday's master branch.  (I'm not sure if we're supposed to report bugs on master or not.)
Expected results
Ideally it would show a table with data, including percentages.
Actual results
It shows the error message unsupported operand type(s) for /: 'decimal.Decimal' and 'float'.
Screenshots

How to reproduce the bug

Create a table and add some data to it:

CREATE TABLE decimaltest (
	"name" varchar NULL,
	value int8 NULL
);

INSERT INTO decimaltest ("name", value) VALUES
    ('one',	478696),
    ('two',	344274),
    ('three',	215815);

I specified int8 since that seems to result in a BIG INT classification.
2. Add the dataset to Superset.
3. Create a new chart on the dataset, of type Table.  Set GROUP BY to name, and PERCENTAGE METRICS to SUM(value).
4. Click RUN.
5. See error.
Environment
(please complete the following information):

superset version: 1.0
python version: 3.7.9
node.js version: v13.11.0

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.

ERROR:superset.views.base:unsupported operand type(s) for /: 'decimal.Decimal' and 'float'
Traceback (most recent call last):
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 143, in na_arithmetic_op
    result = expressions.evaluate(op, left, right)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/computation/expressions.py", line 233, in evaluate
    return _evaluate(op, op_str, a, b)  # type: ignore
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/computation/expressions.py", line 68, in _evaluate_standard
    return op(a, b)
TypeError: unsupported operand type(s) for /: 'decimal.Decimal' and 'float'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/newersuperset/superset/views/base.py", line 178, in wraps
    return f(self, *args, **kwargs)
  File "/home/ubuntu/newersuperset/superset/utils/log.py", line 125, in wrapper
    value = f(*args, **kwargs)
  File "/home/ubuntu/newersuperset/superset/utils/cache.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/newersuperset/superset/views/core.py", line 537, in explore_json
    return self.generate_json(viz_obj, response_type)
  File "/home/ubuntu/newersuperset/superset/views/core.py", line 433, in generate_json
    payload = viz_obj.get_payload()
  File "/home/ubuntu/newersuperset/superset/viz.py", line 488, in get_payload
    payload["data"] = self.get_data(df)
  File "/home/ubuntu/newersuperset/superset/viz.py", line 780, in get_data
    (df[percent_columns].div(df[percent_columns].sum()).add_prefix("%")),
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 651, in f
    new_data = self._combine_frame(other, na_op, fill_value)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/frame.py", line 5870, in _combine_frame
    new_data = ops.dispatch_to_series(self, other, _arith_op)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 275, in dispatch_to_series
    bm = left._mgr.operate_blockwise(right._mgr, array_op)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 364, in operate_blockwise
    return operate_blockwise(self, other, array_op)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/internals/ops.py", line 38, in operate_blockwise
    res_values = array_op(lvals, rvals)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 190, in arithmetic_op
    res_values = na_arithmetic_op(lvalues, rvalues, op)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 150, in na_arithmetic_op
    result = masked_arith_op(left, right, op)
  File "/home/ubuntu/newersuperset/venv/lib/python3.7/site-packages/pandas/core/ops/array_ops.py", line 92, in masked_arith_op
    result[mask] = op(xrav[mask], yrav[mask])
TypeError: unsupported operand type(s) for /: 'decimal.Decimal' and 'float'

[x] I have reproduced the issue with at least the latest released version of superset.

I've only tried on master, which... This may answer my earlier question about reporting bugs on master.  I'm going to submit it anyway, just in case, but apologies if that is not the accepted approach.

[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Though I am not the best at constructing search queries.  If something similar has been reported, apologies.

Additional context
None

(looks like this made it into 1.0)

(looks like this made it into 1.0)

Same problem!

Same problem!

@altef It seems you have a potential fix for this issue. Would you mind opening a PR and we can determine potential ramifications of the change there?

Same problem!

Tested in master, this is now working — the contribution percentage is no longer calculated in superset/viz.py, but in the post processing instead.


First version that can make a successful delivery API call

There might be a future where maybe the model package is auto-generated, perhaps the http client as well. But overall I'd lean toward flexibility and being idiomatic.


Allow egress all and ingress all without target set

What this PR does / why we need it:
Fixes bug where a blacklist rule is being created for allow all rules where a whitelist cannot be created
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run


Piad debt needs to be calculated correctly

At the start of a round, the paid debt is calculated, but that has to be done correctly (not more than money available, not more than needed to cancel debt).

The correct formula is:
newPr.setPaidDebt(Math.max(0, Math.min(0, newPr.getSpendableIncome()) - oldPr.getSpendableIncome()));

Updated.

Check that the calculation is correct:


FR: add qmplay2

https://github.com/zaps166/QMPlay2

recipe added, but build not succesfull

it didn't build against Qt6, but I built it succesfully against Qt5.


An unexpected error occurred

Hello, I have been using your tool for a year without problems, but since this morning I receive an error "An unexpected error occurred." and there is no way to make it work.
Any ideas?
Thank you so much.

root@homebridge:/home/pi/ABUSEIPDB# abuseipdb -C <IP_ADDRESS>
► Check IP: <IP_ADDRESS>
ERROR
✗ Detail:    An unexpected error occurred.
Request time: 0.08986496925354s | UTC time: 2023-06-12 09:28:15
Kristuff\AbuseIPDB v0.9.20 | Made with ♥ in France | © 2020-2022 Kristuff (https://github.com/kristuff)

It seems that the site has now the Cloudflare DDoS Mitigation.
I made a quick test and the problem is that the response to any api request is the temporary html page "Just a moment"
I didn't see related change in the api documentation so unfortunately, nothing I can do for now.(I just reported the issue on their site).

Hi.
it works again.

Good to know :) Thank you


Add support for soundex() function

add soundex scalar function.
it seems that sqlite did not enable soundex() function by default unless build it with SQLITE_SOUNDEX, while the sqlite in the ci workflow did not enable it. this pr skipped the test over soundex() temporarily in the scalar-function.test file.

@flaneur2020 As a follow up, we should do a soundex feature flag and wrap this function with that, following SQLite's lead.


allow extension factories to be asynchronous

Title: allow extension factories to be asynchronous
Context:

at the moment, extension factories must implement a synchronous create*(config) method, assuming that a given configuration itself is enough to return a ready-to-use extension instance or *FactoryCb. E.g.

envoy.access_loggers: AccessLog::InstanceSharedPtr createAccessLogInstance(config)
envoy.filters.http: Http::FilterFactoryCb createFilterFactoryFromProto(config)
envoy.filters.network: Network::FilterFactoryCb createFilterFactoryFromProto(config)
envoy.filters.listener: Network::ListenerFilterFactoryCb  createListenerFilterFactoryFromProto(config)
etc

Problem:

in the context of Wasm extensions, config object itself is not enough to return a ready-to-use extension instance or *FactoryCb, e.g.

Wasm code needs to be loaded from a remote location
Wasm extension is allowed to define _start / proxy_on_vm_start / proxy_on_configure callbacks that, in a general case, might need to use async Envoy API, such as HTTP Client / gRPC Client

right now, envoy.access_loggers.wasm, envoy.filters.network.wasm and envoy.filters.http.wasm implement create*(config) method in a way that lets Envoy to proceed and start serving requests even though Wasm extension has not been loaded, configured and warmed yet

Proposal:

change extension factory interfaces to return a Future/Promise instead of a ready-to-use object
it will give Wasm extensions flexibility in deciding when they are ready-to-use
it will make Envoy behaviour consistent - the enclosing object (e.g., Listener) will remain in the warming state until all extensions are actually ready-to-use
it will simplify user/operator experience with wasm extensions. It's a burden for users to decide whether it's ok to serve requests while the extension is not ready yet. Just don't serve the requests until configuration is complete.

I think the big question is what happens if configuration fails? There is no exit state from warming other than success.

eventually, config based soft-failure or hard-failure?  Presumably the filter owner can decide if they want the server to limp along without that functionality (500 on routes using WASM, fail to access log) or hard-fail (not start up / never listen)

Would it be sufficient to hook into the listener's Init::Manager and use that to defer finalizing the listener load until the filter factory is ready?

If a failure happens, it might be desirable to back track to a previously working listener, or fast-track to the next incoming listener update and interrupt the configuration future.

I feel very strongly that we have a holistic design for this issue which also covers https://github.com/envoyproxy/envoy/issues/12183. This is also highly related to the work that @kyessenov already did on the filter config discovery service implementation inside the HCM. At a high level the problem is very similar and we should be very consistent on how we handle it both in terms of config and implementation (like in FilterConfigDS we should have default config, skip warming, fail until success, etc.).
Can we please do a design doc that covers all of these different uses cases and possibly consolidate issues where possible? Thank you!
cc @ASOPVII @lambdai @kyessenov @lizan

@ASOPVII has switched to ondemand filter chain, which is independent from the proposal here.
The major efforts of ondemand filter chain is to trigger the initialization upon traffic at dataplane(on worker thread),
which can never be addressed by master thread tricks.

@yskopets
This looks reasonable except:

it will give Wasm extensions flexibility in deciding when they are ready-to-use
it will make Envoy behaviour consistent - the enclosing object (e.g., Listener) will remain in the warming state until all extensions are actually ready-to-use

The existing code doesn't introduce future concept but envoy does achieved in the warming state until all extensions are actually ready-to-use with init manager.
The callback embedded in the watcher of Init::Manager acts as if the Promise::set_value(). Could you look into it?

xref: https://github.com/envoyproxy/envoy/issues/12061 (a local init manager for extension config loading was cut out since it's complicated)

@ASOPVII has switched to ondemand filter chain, which is independent from the proposal here.

OK.
Please no coding on any of this until we have design documents for discussion. Thank you!

Thanks everyone for the feedback!
I will need some time to look into the related issues.
Hopefully, in the end of the week.

Here is the draft version of proposal.
It takes into account this issue, #12183 and FCDS.
If you find that the overall direction is correct, then I will it move into a doc to continue with more focused threads of discussion.

Objectives

a Listener/FilterChain should remain in the warming state until all its extensions have been warmed up

Proposal
Outline:

Hierarchy of Init::Managers
Use of Init::Manager by extension factories
Support for negative result in Init::Manager
Notes on initialization of wasm extensions
Notes on FCDS

1. Hierarchy of Init::Managers
Current behaviour:

a Listener has only one Init::Manager for everything under it, i.e. ListenerFilter, AccessLogger, NetworkFilter, HttpFilter extensions and their xDS

Proposed behaviour:

a Listener should have a hierarchy of Init::Managers

1 Init::Manager for Listener itself

this Init::Manager will be used by ListenerFilter and AccessLogger extensions (at listener level) and relevant xDS

+1 Init::Manager per FilterChain

these Init::Managers will be used by NetworkFilter, HttpFilter and AccessLogger extensions (at filter level) and relevant xDS

Motivation:

Hierarchy of Init::Managers is meant to address #12183
The goal is to let NetworkFilter/HttpFilter extensions to always register their init targets under the FilterChain-specific Init::Manager
behaviour of FilterChain-specific Init::Managers will depend on the "warming mode"

In the "default" warming mode (current behaviour):

Init::Manager of the Listener will be waiting for Init::Managers of FilterChains

which means that a Listener will be considered "warmed up" (ready to accept connections) only after all ListenerFilter,  AccessLogger, NetworkFilter and HttpFilter extensions have been warmed up and relevant xDS resources fetched

In the "on demand" warming mode (#12183):

Init::Manager of the Listener will not wait for Init::Managers of FilterChains

which means that a Listener will be considered "warmed up" (ready to accept connections) after

ListenerFilter and AccessLogger extensions (at listener level) have been warmed up and relevant xDS resources fetched
Init::Managers of all FilterChains have been initialized (kicked off but not ready)

connections accepted by worker threads will have to pause until Init::Manager of the respective FilterChain becomes ready

2. Use of Init::Manager by extension factories
To support async initialization, extension factories should use Init::Manager from the *Context object passed in as an argument (namely, ListenerFilter and AccessLogger extensions (at listener level) should use Init::Manager of the Listener, other extensions should use Init::Manager of the FilterChain).
E.g., in the case of HttpFilter extensions:

HttpFilterFactory implementations should return shared_ptr<Http::FilterFactoryCb> (instead of Http::FilterFactoryCb at the moment)
if the extension requires async initialization, HttpFilterFactory should

add init target to the Init::Manager (of  the FilterChain)
return shared_ptr(nullptr) (the assumption is that extension should not be used anyway until warming is compete)

if initialization completes successfully, HttpFilterFactory should

fulfil shared_ptr with the actual implementation
mark init target as ready(Success)

if initialization fails, HttpFilterFactory should

mark init target as ready(Failed)

Once Init::Manager of the FilterChain becomes ready

if there was at least one Failed init target, consider overall FilterChain configuration invalid

in the "default" warming mode, propagate ready(Failed) to the Init::Manager of the Listener, which should mean that overall Listener configuration is invalid and such Listener should be discarded
in the "on demand" warming mode, replace that filter chain with a stub rejecting TCP connections

if all init targets were Success

in the "default" warming mode, propagate ready(Success) to the Init::Manager of the Listener
in the "on demand" warming mode, consider FilterChain ready to accept connections

NOTE:

the error handling approach is coarse grained

in the "default" warming mode, none of the init targets is allowed to fail; otherwise the entire Listener will be deemed invalid and will be discarded
in the "on demand" warming mode, none of the init targets of a FilterChain is allowed to fail; otherwise the entire FilterChain will be deemed invalid and will be replaced by a stub rejecting TCP connections

this is motivated by the following reasons

technical reason: all extensions under a FilterChain are using the same Init::Manager. When one of the init targets calls ready(Failed), there is not enough context to understand which particular extension has failed and apply more local corrective action
logical reason: if initialization fails, it only means that overall configuration is not valid. The assumption is that it's better to warn the user asap and let him take a proper action rather than let the system keep doing something, which is unlikely what the user expects anyway

3. Support for negative result in Init::Manager
Current behaviour:

Init::Manager and Init::Targets have only 1 outcome - ready()

Proposed behaviour:

Init::Manager should be parameterized to support various outcomes

e.g., Init::Manager<T> where T is enum { Done } to represent current behaviour

ready() calls should be replaced with ready(Done)

e.g., Init::Manager<T> where T is enum { Success, Failure } to represent behaviour of this proposal

ready() calls should be replaced with ready(Success) and ready(Failure)

Init::Manager implementation should be parameterized with a strategy to aggregate outcomes

e.g., a strategy Done + Done = Done
e.g., a strategy Success + Failure = Failure
e.g., a strategy Success + Failure = Failure and notify the watcher immediately

4. Notes on initialization of wasm extensions

a wasm extension requires initialization on the main thread and each of the worker threads
to keep it simple and also reusable in the context of FCDS, factory of a wasm extension should complete initialization on all threads (main + all worker threads) and only after that consider the extension warmed

5. Notes on FCDS

every FCDS update will require a separate instance of Init::Manager

Hi yskopets, great proposal. I just want to introduce the current progress of the on-demand filter chain as well.
As you mentioned, in the default warming mode, the listener's init manager will wait for filter chain's init manager.
In the current implementation for on-demand filter chain, the filter chain will not pass its init manager to the transport_factory_context. Instead, we just provide an empty placeholder. Hence the listener's init manager will not include any dependencies of this filter chain as targets.
Instead, a PerFilterChainRebuilder which includes init manager has been implemented for passing init manager and getting callbacks from dependencies later when rebuilding the real filter chain.  This might be the Hierarchy of init manager you mentioned.

@ASOPVII Having per-FilterChain Init::Managers from the beginning (as opposed to only at a time of PerFilterChainRebuilder ) is meant to:

to do the most of Listener config processing while in the context of onUpdate of the LDS subscription (so that invalid Listener config could be rejected asap)
to avoid reprocessing FilterChain configs twice (once without Init::Manager and another time with Init::Manager)
to avoid the need in worker (thread) -> main (thread) communication in the "on demand" warming mode

assuming that every FilterChain has to be initialized anyway, main thread can kick off FilterChain initialization without waiting for an explicit request for that (and then handle deduplication and so on)
in the meantime, worker threads could use a thread local waiting list to hold paused connections
once FilterChain has been warmed, main thread can use Envoy's TLS mechanism to deliver new Listener state to every worker thread and resume connections in the local waiting list

I'm not saying that you shouldn't use PerFilterChainRebuilder and worker (thread) -> main (thread) communication. Just explaining what other options this proposal can give you.

@yskopets

assuming that every FilterChain has to be initialized anyway,

Sorry this is the core value of on-demand.
What's more. I am not sure if I get the story of init failure. Under the scenario of LDS, why should listener address the failure?
If LDS server cannot respond to a failure, the listener should not be aware of that failure.
A complete-or-block init manager is supporting the listener behavior that the listener is always fully initialized. You see the new
listener and drain the old listener, or you stay at the the old listener. What connection should you drain when listener is partial
ready when you make the init manager asynchronous?

@lambdai
From the description of #12183 I understand that the problem it is solving is how to enable individual FilterChains to start serving requests as soon as they are ready. I assume that you still need all FilterChains, just at a different speed.
Regarding why to handle configuration errors and discard invalid Listener configurations:

To release unused resources timely
wasm extensions have non-negligible overhead:

Each extension runs in separate Wasm VM, each worker thread needs a separate instance of Wasm VM
E.g., if memory overhead of 1 extension starts from 1MB per VM and you have 10x cores, you're spending 11MB just to keep one extension loaded in Envoy

There is no reason to hold this memory if we know that Listener will never become ready to serve requests.

To make sense of xDS protocol and metrics inferred from it

if Envoy ACKs a new Listener configuration after just 1 second but in the background is spending minutes trying to warm it up and silently fails in the end, it doesn't help understanding the status of the system and where to look for an error (in Envoy/in control plane/in user-defined policies/etc ?)

Regarding draining the old listener and making init manager asynchronous, I don't understand what you mean:

if warming up fails and a Listener get discarded, the old listener will not be drained
this proposal doesn't change the nature of init manager; it only says that init manger will now be used not only for fetching related xDS resources but also for making sure that Wasm VMs get fully initialized before Listener is considered warmed up and it is time to drain the old listener

At a high level this proposal makes sense to me and I think we should move it into a doc which also covers ECDS (sorry I was calling it FCDS before) as well as #12183. See also: https://docs.google.com/document/d/1OT7L5GbsJ03njPsSscZVnh-bz-t80z4rzdX8a4ICd8A/edit?ts=5f20b724.
A few comments:

Per @lambdai I agree that we need to handle a mode in which we don't start initializing the filter chains at all until they are called. I think that would be a natural extension/part of this proposal.
I still want to be super explicit about error handling and have similar options across all of the different permutations. I think these options basically come down to:
Block until warming is complete or fails.
Possibly specify a default config that can be used in case of failure.
Possibly use the default config for immediate startup until the first warming success.

@yskopets

I assume that you still need all FilterChains, just at a different speed.

I am afraid not. There are several follow ups on top of on-demand feature.There is no concrete roadmap, but
definitely are unblocked once we have on-demand.

We might introduce further parameters to restrict M filter chains could be loaded though we provide N where M < N.
Retention policy which unload filter chains when the filter chain idle timeout is hit.

@mattklein123
From what we discussed offline I think what you want is

upon on demand filter chain, provide a fallback strategy(e.g close connection if filter chain cannot be loaded in X second). The idea is that we cannot sit and hope user would provide a the best config at ECDS.
Clearly document the expected behavior, such as
a) listener will fail the request if the timeout in on-demand is lower than the timeout of providing default filter chain in ECDS
b) listener will use the default filter chain if the timeout in demand is greater than the timeout of providing default filter chain in ECDS
List recommended strategy for users.
Is that all?

@yskopets
I feel you may want to add the asynchronous factory interface before introducing the failure callback at init manager.
The failure callback at init manager is an optimization it could be done lagter.
Currently we can simulate the failure calling ready() but provide a not-working stuff such as NotReadyTransportSocket.  https://github.com/envoyproxy/envoy/blob/a5d56f5c2e027fa3a3965487a06035de272648df/source/extensions/transport_sockets/tls/ssl_socket.cc#L28

@lambdai yes. Please look at https://github.com/envoyproxy/envoy/blob/5fb236b4c3376e2b761d540a54b4772d620740aa/api/envoy/config/core/v3/extension.proto#L48-L56 and discuss with @kyessenov. We need to be consistent with how we handle failure across all of these features.


YTDL: No longer accepting Spotify

It is the copyright apocalypse and Spotify's terms of service no longer allow scraping songs off their API.  As such YTDL no longer can grab spotify links so I have adjust our documentation and updated our YTDL version to account for these changes.

BTW because YTDL doesnt actually use the Spotify mp3's we could interpret the spotify data ourselves and then give YTDL a query with the artist and title but that is an issue for another day


How to disable pan bounce on frame edges?

I would like to allow for a hard stop when zoomed and hitting the edge of the frame, is it currently possible?
https://github.com/user-attachments/assets/4f8acb10-87bc-47b7-a8ff-b063a43120ec
Thank you!

Unfortunately, it's not possible at the moment, it was added to make sure the image is not moved out of the view with no way of accessing it anymore