[Egress] npm dropping output traffic

Is this a request for help?: YES

Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE

Which release version?: v1.0.32

Which component (CNI/IPAM/CNM/CNS): NPM

Which Operating System (Linux/Windows): Linux
For Linux: Include Distro and kernel version using "uname -a": Linux aks-agentpool-37083126-vmss000000 4.15.0-1069-azure #74-Ubuntu SMP Fri Feb 7 17:22:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Which Orchestrator and version (e.g. Kubernetes, Docker): Kubernetes 1.15.5

What happened:
Setup
+--------------------+        +--------------------+
|     AKS Cluster    |        |   Azure Firewall   |
|       Subnet       |        |       Subnet       |
|     (np rules)     |        |     (fw rules)     |
+--------------------+        +--------------------+
                     |        |
                     |        |
                     +--+RT+--+

Important: please note Azure Firewall rules are created before Network Policies rules.

  I mean, I don't get a bad address but connections can't be established (it results in timeout).
# wget -S https://****.vault.azure.net
Connecting to ****.vault.azure.net (<IP_ADDRESS>:443)
wget: can't connect to remote host (<IP_ADDRESS>): Connection timed out
...

# iptstate
Filters: src: <IP_ADDRESS>
Source                      Destination                Prt State       TTL
<IP_ADDRESS>:60845            <IP_ADDRESS>:53               udp               0:00:24
<IP_ADDRESS>:45526            <IP_ADDRESS>:53               udp               0:00:24

Important: note the missing established connection against <IP_ADDRESS>:443

What you expected to happen:
based on the premise Network Policies should be for east-west traffic, while NSG/FW deals with north-south traffic, I would've expected my pods can establish a connection with any external resources allowed by the Azure Firewall without Network Policies interference.
That being said, after applying the deny-all it is ok to drop all connections between pods, but I would've expected to be able to open connections against Azure resources that are blessed in my Azure Firewall.

How to reproduce it (as minimally and precisely as possible):

create an AKS cluster
limit egress using Azure Firewall
apply all above Network Policies in order
within any pod in the backend-dev namespace, try to resolve using wget to any internet facing resource

Anything else we need to know:
I could workaround it by creating an egress rule (allow all or a specific one) that takes care of external traffic. But ideally, I'd prefer not to duplicate rules in Azure Firewall and Network Policies.
Also, I'm almost sure I had tested it with v1.0.27 before and it was working properly. But I could be wrong.

Hi @ferantivero, are you trying to allow all egress traffic from backend-dev via udp 53? In your example you're only allowing traffic to all-namespaces via udp 53, which does not include external traffic.
Here's an example that allows all egress via udp 53 from backend-dev
apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: dns-dev
  namespace: backend-dev
spec:
  egress:
  - to:
    ports:
    - port: 53
      protocol: UDP
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

@jaer-tsun please let me give try on this.

@jaer-tsun, In short, it didn't work (Short and Full updates below). But please take your time, no hurries at all :)
Please let me answer your question here: are you trying to allow all egress traffic from backend-dev via udp 53?

Not just that, I've done that already, and it is working properly based on my understanding. What I do-not-want is Network Policies to interfere with north-south traffic even after deny-all network policy is created, as I expect Network Policies to only take care of Pod-To-Pod connections, it means within the AKS Cluster subnet.

But please let me remark why I conceptually expect such thing, from "What you expected to happen" section, please let me re-share what is the original intent:

based on the premise Network Policies should be for east-west traffic, while NSG/FW deals with north-south traffic, I would've expected my pods can establish a connection with any external resources allowed by the Azure Firewall without Network Policies interference.

That being said, after applying the deny-all it is ok to drop all connections between pods, but I would've expected to be able to open connections against Azure resources that are blessed in my Azure Firewall (e.g Azure KeyVault).

Short update
please find a screenshot below on how the proposal change has impacted at IpTable level

and yet, my pods can't reach external Azure resources (e.g. Azure KeyVault).
Full update

just tested the proposal and I didn't work
still no external access, my pods are still facing Connection timed out trying to reach Azure KeyVault
Please let me share how the object looks like in k8s after creation, IpTables, and result after applying the proposed Network Policy

Core DNS (take 2 using proposal change)
this is the new allow all egress via udp 53 from backend-dev expecting being able to ACCEPT OUTPUT for UPD 53. This way, we don't get a bad address result. But also, we are able to reach external services (internet facing) that are allowed by our Azure Firewall.
# kubectl get networkpolicy dns-dev -n backend-dev -o yaml

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"dns-dev","namespace":"backend-dev"},"spec":{"egress":[{"ports":[{"port":53,"protocol":"UDP"}],"to":null}],"podSelector":{},"policyTypes":["Ingress","Egress"]}}
  creationTimestamp: "2020-03-17T21:43:34Z"
  name: dns-dev
  namespace: backend-dev
spec:
  egress:
  - ports:
    - port: 53
      protocol: UDP
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

IpTables after allowing DNS resolution (take 2)
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
AZURE-NPM  all  --  anywhere             anywhere
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain AZURE-NPM (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AZURE-NPM-INGRESS-PORT  all  --  anywhere             anywhere
AZURE-NPM-EGRESS-PORT  all  --  anywhere             anywhere
AZURE-NPM-TARGET-SETS  all  --  anywhere             anywhere

Chain AZURE-NPM-EGRESS-PORT (1 references)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain match-set azure-npm-2662673593 src /* ALLOW-ALL-TO-UDP-PORT-53-FROM-ns-backend-dev */
AZURE-NPM-TARGET-SETS  all  --  anywhere             anywhere             match-set azure-npm-2662673593 src /* ALLOW-ALL-FROM-ns-backend-dev-TO-JUMP-TO-AZURE-NPM-TARGET-SETS */

Chain AZURE-NPM-EGRESS-TO (0 references)
target     prot opt source               destination

Chain AZURE-NPM-INGRESS-FROM (0 references)
target     prot opt source               destination

Chain AZURE-NPM-INGRESS-PORT (1 references)
target     prot opt source               destination

Chain AZURE-NPM-TARGET-SETS (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             match-set azure-npm-2662673593 src /* DROP-ALL-FROM-ns-backend-dev */
DROP       all  --  anywhere             anywhere             match-set azure-npm-2662673593 dst /* DROP-ALL-TO-ns-backend-dev */

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  <IP_ADDRESS>/21         anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             <IP_ADDRESS>/21         /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination

Result after applying the proposed Network Policy for Dns  (not expected 👎)
# wget -S https://*****.vault.azure.net
Connecting to *****.vault.azure.net (<IP_ADDRESS>:443)
wget: can't connect to remote host (<IP_ADDRESS>): Connection timed out

# iptstate
Source                      Destination                Prt State       TTL
<IP_ADDRESS>:53762            <IP_ADDRESS>:53               udp               0:00:19
<IP_ADDRESS>:49801            <IP_ADDRESS>:53               udp               0:00:19

Important: please note it still missing the established connection against <IP_ADDRESS>:443

Wouldn't you have to allow egress to tcp 443? not sure if I'm misunderstanding something here...

I'm still not completely sure what you're trying to do, but...
If you want external traffic you'll need to add a rule for it
apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: dns-dev
  namespace: backend-dev
spec:
  egress:
  - to:
    ports:
    - port: 53
      protocol: UDP
  - to: []
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Can you tell me what you're trying to restrict/allow?

thanks for sharing that @jaer-tsun, the thing is I see that as a workaround. Please take a look at the section above: “Anything else we need to know”
Regarding,

Can you tell me what you're trying to restrict/allow?

Egress
restrict: all
allow: udp 53 only
Expected
After applying such restrictive Network Policy rules, I can still reach for instance Azure KeyVault (external Azure resource).
The reason I expect this is because it is south traffic and I want that to be managed exclusively by Azure Firewall and do-not-duplicate my firewall rules at l Network Policy level: https://docs.microsoft.com/en-us/azure/virtual-network/kubernetes-network-policies#planning-security-for-your-kubernetes-cluster
Actual
I can not access Azure KeyVault even allowing that in the Azure Firewall.
What am I missing here, I mean conceptually speaking?

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: dns-dev
  namespace: backend-dev
spec:
  egress:
  - to:
    ports:
    - port: 53
      protocol: UDP
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

The above policy only allows egress via UDP 53 so that should satisfy

Egress
restrict: all
allow: udp 53 only

When you make a call to azkv, it's correctly dropping the traffic.
If you want the external traffic to be dropped by az firewall instead of NPM, you may need to configure ipblock to allow that external traffic and block the rest.

first of all @jaer-tsun thanks for your support 💪
When you say,

When you make a call to azkv, it's correctly dropping the traffic.

Based on my understanding this doc is recommending us to plan the exact opposite. Even more  “if” it is correct to drop external traffic (south), we‘d need to duplicate egress rules at Azure Firewall and Network Policy level.
Regarding,

If you want the external traffic to be dropped by az firewall instead of NPM, you may need to configure ipblock to allow that external traffic and block the rest.

I really do-not-want, please note Network Policy does not manage concepts like Azure Service Tags to deal with Public Ip changes and it just layer 4 (ipblock).
Please note in our solution we have multiple resources like:

azkv
appinsights
azcosmosdb
azredis

It is not practical to maintain these rules in both, Azure Firewall and Network Policies and at the same time we would like to prevent the solution from allowAll workaround:
egress:
- to: []

I’m sure I’m missing something here.

I can see that the doc is making it black and white but:

with ipblock you'd be allowing <IP_ADDRESS>/0 and blocking e.g. {pod subnet}
with other ingress policies you can block traffic from backend-dev or a label in backend-dev for pods that have external access

According to k8s, pods become isolated once network policies are applied on them.
Also, here are some egress examples regarding my response.

under section Subnets and IP Addresses (IPBlocks)
in this example, external is blocked since it's not specified

again thanks for your support @jaer-tsun, and thanks for sharing the resources, I'm pretty much familiarized with them all :)
We would've expected another sort of integration among AKS-NPM-AzFW/NS, and still confused why NPM is blocking north-south, I mean more than just Pod-To-Pod communications.
Additionally, I think it was working differently in v1.0.27. But I might be wrong.
Please feel free to close this issue as by design, unless you consider otherwise.

NPM controls ingress/egress to/from pods and that isn't limited by in cluster or external traffic.
Unfortunately, v1.0.27 introduced a lot of bugs and since then, we've addressed them according to k8s specs to achieve parity with Calico NPM. Also, there weren't any integration with AzFW/NSG taken into account during the design.
thanks @ferantivero for understanding and feel free to contact us for any issues in the future
*closing this issue as npm is working as designed


Using 'use-debounce' will cause number not reaching target

Step to reproduce
function DebounceMN({ value }: { value: number }) {
  const [dValue] = useDebounce(value, 200);
  return <MotionNumber value={dValue ? dValue : 0} />;
}

Expected result
giving value 12345 will result something like 00045

I just pushed a rewrite of MotionNumber called NumberFlow that should fix this: https://number-flow.barvian.me. Please let me know if you're still experiencing issues 👋


Validate all hat/concentrator combinations on MNTD

Acceptance criteria

Try on RAK ROCK Pi varying concentrator: Nebra sx1301, sx1302, sx1303
Swapping Nebra hat
Swapping hat and all concentrator variants
Check for

Witnesses
Beacons
Button starts Bluetooth advertisement
Diagnostics page only lists LTE as error
Can pair with app

I thought it should be validated with all RAK concentrators, not the Nebra ones. My bad :sweat_smile:

@marvinmarnold the use case for this is to largely support the default concentrator. I don't think there is a big use case for using this with replacement concentrators.

Ya you are right @vpetersson. I think I created this in response to this comment. Now that the ROCK Pi issue is resolved, I don't think this adds any value and can be closed.

closing in favour of #376


Support of multi OrientDB environment

@krichter722  :
It'd be great if the database connection would be configurable (including a switch for connecting to the (or even multiple) embedded instance(s)). I can't think of any other option I'm missing to configurable at runtime in the web frontend, but I formulated the title as abstract as possible.

Idea is the following:
There is master orientdb database with whole set of users who can have access to an environment. Every user can have one to many child databases. Databases can be shared among some subset of users (group rights). User can transparently navigate to a child db after initial login.

I'm far from being an expert of orientdb features. Is a "child database" a special feature? What should I think of? A cluster sharing deduplication, a nested database (can't imagine that)?
Regardless of the question above, I'd say managing database instances (ownership, location, etc.) with the client is overkill if these matters are managed by the database itself. Instead provide assistence for database creation, moving and ownership/permission management - think of a postgresql server and a pgadmin client; I'd do the separation of responsibility exactly like those two. Implementing a mapping between Orienteer users database users shouldn't be underestimated, although you know the orientdb code quiet well. Managing different database with different separate connection parameters (which can be stored in Orienteer) is much easier and allows the same features as "transparently navigate to a child db".
The idea of a master database is then obsolete. If you use an orientdb instance to store Orienteer settings (haven't checked how it's actually done, but it might change anyway), I wouldn't expose it to the user at all.

Any news on that? I'm willing to take a look into Orienteer development again.

Hi,
The main idea of this project is to make a data warehouse (actually data lake), so no "dropdown" list for selection of DBs as it commonly is for DB cliens, was not assumed.
Also I have idea to build a SaaS for Orienteer. To be able to implement that, I'm going to use a concept that I mentioned before: child DB - are OrientDB instancies ordered by clients and parent DB is for management of whole SaaS services.
If you can also change Orienteer in a way as you want to have it: that will be cool. So Orienteer will cover interest of 2 categories: 1) clients who want to have a Data Lake for their business 2) clients who just need to have Data Management tool

If you can also change Orienteer in a way as you want to have it: that will be cool. So Orienteer will cover interest of 2 categories: 1) clients who want to have a Data Lake for their business 2) clients who just need to have Data Management tool

I have to admit I didn't get in touch with the terms "Data warehouse" and "Data lake" so far (learned the exact definition today), but I think both approaches 1) and 2) are perfectly compatible and code can be widely reused. My comment 4 above applies to 2) then and the idea of master and child databases is perfect for 1).

Yes, but there is 1 small thing which limits using Orienteer as OrientDB client: to be able operate as Data Warehouse Orienteer should change schema of a DB for its own needs. So after referencing Orienteer to arbitrary DB - that DB will be changed.


Lazy cyclic dependencies are no longer possible in v2.0

Description
Since v2.0, it is no longer possible to have a cyclic dependency where one of the dependencies is lazy resolved.
this results in the following error:

error DIE001: A cyclic dependency has been found Pure.DI.Example.Dependency <-- System.Lazy<Pure.DI.Example.IDependency2> <-- System.Func<Pure.DI.Example.IDependency2> <-- Pure.DI.Example.Dependency2 <-- Pure.DI.Example.Dependency.

In v1, code was generated to resolve the lazy dependency lazily.
Example
public class Dependency : IDependency
{
    public Dependency(Lazy<IDependency2> dependency2) { }
}

public class Dependency2 : IDependency2
{
    public Dependency2(IDependency dependency) { }
}

public partial class Composition
{
    public static void Setup() =>
        DI.Setup(nameof(Composition))
            .Bind<IDependency>().To<Dependency>()
            .Bind<IDependency2>().To<Dependency2>()
            .Bind<IService>().To<Service>().Root<IService>("Root");
}

Repository
https://github.com/Yves-Be/Pure.DI.LazyCircularDependency

Unfortunately, it's a compromise to make the generation faster. I will try to find a solution. Right now there is an easy way to solve this problem:
Bind<Func<TT>>().To(_ => new Func<TT>(Resolve<TT>))

so your example might look like this:
using Pure.DI;

var service = new Composition().Root;

return;

public class Dependency : IDependency
{
    private readonly Lazy<IDependency2> _dependency2;
    public Dependency(Lazy<IDependency2> dependency2) => _dependency2 = dependency2;
}

public interface IDependency
{
}

public class Dependency2 : IDependency2
{
    private readonly IDependency _dependency;
    public Dependency2(IDependency dependency) => _dependency = dependency;
}

public interface IDependency2
{
}

public interface IService
{
}

public class Service : IService
{
    private readonly IDependency2 _dependency2;

    public Service(IDependency2 dependency2) => _dependency2 = dependency2;
}

public partial class Composition
{
    public void Setup() =>
        DI.Setup(nameof(Composition))
            .Bind<Func<TT>>().To(_ => new Func<TT>(Resolve<TT>))
            .Bind<IDependency>().To<Dependency>()
            .Bind<IDependency2>().To<Dependency2>()
            .Bind<IService>().To<Service>().Root<IService>("Root");
}

Unfortunately, in this case performance degrades and PerResolve lifetime stops working.
So this behavior will not be fixed at the moment. It is recommended to avoid cyclic dependencies.

@Yves-Be could you please try this version

That seems to work, up until you inject IDependency in the Service.
I believe it is good enough for my case though.

@Yves-Be yeah, I got it, I'll fix it so it's fully functional.

Fixed in 2.0.13


npm install not working

Please help
npm install  is not working not working
it give below error :

D:\vue-element-admin-master>npm install
npm WARN deprecated<EMAIL_ADDRESS>This project has been renamed to 'tasksfile'. Install using 'npm install tasksfile' instead.
npm WARN deprecated<EMAIL_ADDRESS>core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated<EMAIL_ADDRESS>This project has been renamed to @pawelgalazka/cli . Install using @pawelgalazka/cli instead
npm WARN deprecated<EMAIL_ADDRESS>Package no longer supported. Contact<EMAIL_ADDRESS>for more info.
npm WARN deprecated<EMAIL_ADDRESS>This project has been renamed to @pawelgalazka/cli-args. Install using @pawelgalazka/cli-args instead
npm WARN deprecated<EMAIL_ADDRESS>CircularJSON is in maintenance only, flatted is its successor.
npm WARN deprecated<EMAIL_ADDRESS>use String.prototype.padStart()
npm WARN deprecated<EMAIL_ADDRESS>Please upgrade to kleur@3 or migrate to 'ansi-colors' if you prefer the old syntax. Visit https://github.com/lukeed/kleur/releases/tag/v3.0.0\ for migration path(s).
npm ERR! code ENOENT
npm ERR! syscall spawn git
npm ERR! path git
npm ERR! errno ENOENT
npm ERR! enoent Error while executing:
npm ERR! enoent undefined ls-remote -h -t<EMAIL_ADDRESS>npm ERR! enoent
npm ERR! enoent
npm ERR! enoent spawn git ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR!      C:\Users\Admin\AppData\Roaming\npm-cache_logs\2019-12-23T14_38_12_530Z-debug.log

从错误来看应该是你的git本地环境配置有误


Panic

OS: Arch Linux
Version: compile from git
Compile argument : cargo build --release
backtrace:
./tap
thread 'main' panicked at 'cannot sample empty range', /home/chromer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134:9
stack backtrace:
   0:     0x7f40d5682f61 - <unknown>
   1:     0x7f40d56a7b1f - <unknown>
   2:     0x7f40d5680179 - <unknown>
   3:     0x7f40d5682d75 - <unknown>
   4:     0x7f40d56842a3 - <unknown>
   5:     0x7f40d5684034 - <unknown>
   6:     0x7f40d5684829 - <unknown>
   7:     0x7f40d56846e1 - <unknown>
   8:     0x7f40d56833c6 - <unknown>
   9:     0x7f40d5684472 - <unknown>
  10:     0x7f40d5444f73 - <unknown>
  11:     0x7f40d5445003 - <unknown>
  12:     0x7f40d5487090 - <unknown>
  13:     0x7f40d54e8970 - <unknown>
  14:     0x7f40d54c6d8b - <unknown>
  15:     0x7f40d54fe3c9 - <unknown>
  16:     0x7f40d545f0ed - <unknown>
  17:     0x7f40d54d29a3 - <unknown>
  18:     0x7f40d5493569 - <unknown>
  19:     0x7f40d567a65b - <unknown>
  20:     0x7f40d545f1e5 - <unknown>
  21:     0x7f40d5027cd0 - <unknown>
  22:     0x7f40d5027d8a - __libc_start_main
  23:     0x7f40d5445615 - <unknown>
  24:                0x0 - <unknown>

UPDATE: Same panic with 0.4.6 release.
(https://github.com/timdubbins/tap/releases/download/v0.4.6/tap_0.4.6-x86_64-unknown-linux-gnu.tar.gz)

Hi, thanks for raising this issue.
Can you try running tap in a directory that contains audio files? Or supply a path as an argument?
I think this panic is caused by the way we're queuing a random track (for the 'random album' and 'random' shortcuts) - if tap is run in a directory that contains no audio this would panic.
Let me know if this solves your issue. If it does I'll add some error handling for this case. If not I will take a closer look and try and reproduce

Hi,

Can you try running tap in a directory that contains audio files?

No panic by this way, but:
The behaviour of tap is somehow weird, i put it in a folder containing musics with tracks and folders (mp3, m4a, webm, ogg), but it onlt detects and play the first folder and the tracks inside it and ignores every other files.
Let me show via screenshots:

supply a path as an argument

Same panic:
$ ./tap Pop/
thread 'main' panicked at 'cannot sample empty range', /home/tim/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Full backtraces:
$ RUST_BACKTRACE=full ./tap Pop/
thread 'main' panicked at 'cannot sample empty range', /home/tim/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.8.5/src/rng.rs:134:9
stack backtrace:
   0:     0x55b0a2c9774a - std::backtrace_rs::backtrace::libunwind::trace::h9a6b80bbf328ba5d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55b0a2c9774a - std::backtrace_rs::backtrace::trace_unsynchronized::hd162ec543a11886b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55b0a2c9774a - std::sys_common::backtrace::_print_fmt::h78a5099be12f51a6
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x55b0a2c9774a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha1c5390454d74f71
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55b0a2cb5caf - core::fmt::write::h9ffde816c577717b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1254:17
   5:     0x55b0a2c94b25 - std::io::Write::write_fmt::h88186074961638e4
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
   6:     0x55b0a2c97515 - std::sys_common::backtrace::_print::h184198273ed08d59
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x55b0a2c97515 - std::sys_common::backtrace::print::h1b4d8e7add699453
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x55b0a2c98bbe - std::panicking::default_hook::{{closure}}::h393bcea75423915a
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:269:22
   9:     0x55b0a2c98965 - std::panicking::default_hook::h48c64f31d8b3fd03
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:288:9
  10:     0x55b0a2c9911e - std::panicking::rust_panic_with_hook::hafdc493a79370062
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:691:13
  11:     0x55b0a2c98fd2 - std::panicking::begin_panic_handler::{{closure}}::h0a64bc82e36bedc7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:580:13
  12:     0x55b0a2c97bb6 - std::sys_common::backtrace::__rust_end_short_backtrace::hc203444fb7416a16
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
  13:     0x55b0a2c98d72 - rust_begin_unwind
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
  14:     0x55b0a2a75753 - core::panicking::panic_fmt::h0f6ef0178afce4f2
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  15:     0x55b0a2a757ed - core::panicking::panic::h0ead933cb8f56d66
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
  16:     0x55b0a2a96f43 - tap::utils::random::ha777dd80d3b24636
  17:     0x55b0a2aad900 - tap::player::player::Player::randomized::h24ed4756dfa58c73
  18:     0x55b0a2af213f - tap::data::UserData::new::h8f0fca1e0c699fef
  19:     0x55b0a2afd8d9 - tap::app::run::h7a5f14fa257f482e
  20:     0x55b0a2aaee0d - tap::main::hfdee168564c8bdbd
  21:     0x55b0a2b155a3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5a4f0f554398fb82
  22:     0x55b0a2a86c79 - std::rt::lang_start::{{closure}}::h9d287f8170869061
  23:     0x55b0a2c8f4be - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hb1327dc2ef3fecdf
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
  24:     0x55b0a2c8f4be - std::panicking::try::do_call::h4044173225fe83dd
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  25:     0x55b0a2c8f4be - std::panicking::try::hd8a722c09d156a53
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  26:     0x55b0a2c8f4be - std::panic::catch_unwind::hd2ca07971cf0119b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  27:     0x55b0a2c8f4be - std::rt::lang_start_internal::{{closure}}::h26d89d595cf47b70
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
  28:     0x55b0a2c8f4be - std::panicking::try::do_call::hf47aa1aa005e5f1a
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  29:     0x55b0a2c8f4be - std::panicking::try::h73d246b2423eaf4e
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  30:     0x55b0a2c8f4be - std::panic::catch_unwind::hbaaeae8f1b2f9915
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  31:     0x55b0a2c8f4be - std::rt::lang_start_internal::h76f3e81e6b8f13f9
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  32:     0x55b0a2aaf045 - main
  33:     0x7fb7a3427cd0 - <unknown>
  34:     0x7fb7a3427d8a - __libc_start_main
  35:     0x55b0a2a75cd5 - _start
  36:                0x0 - <unknown>

Hey, this should be fixed with the latest commit.
I've also modified the search to include any audio files that are not in subdirectories of the search root (so All too well Daughter and Joo wo in your example directory will know be visible). These files appear under the name of the search root in the fuzzy-finder.
Let me know if this now works as expected.

Tested 0.4.7-git , issue fixed, thanks for your efforts.


Removing mention of max worker nodes per PM

also applies to 1.4/master, based on PM decision, thanks

Thanks @lparis!


get text by language fails

Populated my cache with the default berkley db but when I do
eng = get_etexts('language','en')
or get_etexts('language','eng')
or get_etexts('language','En')

The query returns the empty frozenset.
Is this a known bug?

The language information is stored as dcterms:language in the Project Gutenberg metadata which recommends RFC 4646 to specify the languages, so tags like en, fr, fi, de, etc. (which you used) should be the correct way to query the data-set. Doing a quick grep through the RDF files confirms that you executed the right query, so this must be a bug.

I can reproduce this, adding bug label.

I found the root cause of this. The language metadata extractor was implemented incorrectly back in #50. The language field in the RDF is typed as RFC 4646 so we need to reflect this in the extractor.
Example RDF:
<dcterms:language>
  <rdf:Description rdf:nodeID="Ndc98bbe6c7bb4281b75c633906a61a97">
    <rdf:value rdf:datatype="http://purl.org/dc/terms/RFC4646">nl</rdf:value>
  </rdf:Description>
</dcterms:language>

The important part here is the datatype="http://purl.org/dc/terms/RFC4646". We'll need to specify this when converting the queried value into RDF. This can be done via the datatype argument to the Literal constructor.

@raduangelescu: Thanks so much for reporting this. The bug should now be fixed. Could you please install the latest gutenberg version from the repository and let me know if the problem is fixed for you? If it's not, please feel free to re-open the issue.


Slow performance returning to window

Issue Type: Bug
Within the last few days, I've been having some fairly slow performance returning to the VSCode window from other windows. Looking at the task manager, I notice a CPU spike for code when I return to the code window. I have disabled extensions and even made sure to close all open tab. The performance spike is enough to cause audio playing in the background (I have  tried both the Spotify desktop application as well as YouTube in a chrome tab) to stutter like my CPUs are completely clocked out. Moving around the window is slow, and when switching back to another window, it's still fairly slow for a few seconds before whatever VS Code was doing finishes. I usually only notice this kind of user experience slow down when I'm compiling a heafty application that takes all 32 of my logical cores.
I am running on the Dev Insider channel of Windows 11. I currently have Build 22499.rs_prerelease.211105-1550. I'm using 2-folder workspace that I've been using for several months without issues.
This issue is also seen on Insider build 1.63.0-insider commit bedf867. I also put together a video capture of it happening with both stable and insider builds with the task manager showing a bunch of processes being kicked off when it does occur.
https://user-images.githubusercontent.com/549323/142494188-c7903cdb-94fb-43f7-8866-36888357600f.mp4
NOTE: Another interesting datapoint: returning to August's 1.60.2 does not resolve the issue. This must be related to the interaction of VSCode with some other app that got updated (Git or Windows are the only two that come to mind). Git for Windows from https://git-scm.com is at version 2.33.0.
Might be related to #137397
VS Code version: Code 1.62.3 (ccbaa2d27e38e5afa3e5c21c1c7bef4657064247, 2021-11-17T08:11:14.551Z)
OS version: Windows_NT x64 10.0.22499
Restricted Mode: No

System Info

Item
Value

CPUs
AMD Ryzen 9 3950X 16-Core Processor             (32 x 3493)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
63.91GB (41.84GB free)

Process Argv
--crash-reporter-id 4c41e6da-9652-41d9-a1cb-f219ff67a4d1

Screen Reader
no

VM
0%

Extensions: none
A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
pythontb:30283811
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30384385
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
dockerwalkthru:30377721
bridge0723:30353136
pythonrunftest32:30373476
pythonf5test824:30373475
javagetstartedt:30391933
pythonvspyt187:30373474
vsqsis300:30402085
vsaa593:30376534
vssld246:30401890

Upgrading Git for Windows to 2.34.0 does not resolve the issue.

Can you tell which process is getting the CPU spike (the full command line args)? Is it different if you start with --disable-extensions?

I guess you answered the extensions question already. But if you can figure out which process it is, we can get a profile of that process.
How long have you seen this? Could it be related to the win11 dev channel?

@roblourens I've only seen this for maybe a few days, so it very much could be related to the Win11 Dev channel, but it should be noted I don't see this issue occur in any other application so there may be a new compatibility issue here (or still possible to be a bug on Windows's side).
Since my last post, I also updated Windows 11 to the insider build 22504, but the problem is still around.
Figuring out which process has a spike is difficult because as seen in the video, lots of new processes are initiated during the spike, and they all seem to be smaller CPU percentages. Here's just one screenshot of the video showing multiple processes adding up to 0.6% CPU usage, but task manager is reporting VSCode as a whole is using 8% (note that 3.1% is ~100% of 1 logical core on my machine, assuming all 3.1% is coming from a single process in a single thread):

My eyeballs seem to say Console Window Host has the most processes created with the most ~1%, but the multiple code.exe processes do seem to take a heavy hit.
Any recommendations on how I can measure a more technically correct answer? I'm willing to build VSCode from source if necessary (or whatever else is needed to get to the bottom of this) as this is a big frustration with my day-to-day activities.
I'm going to try disabling the built in git extension as well as use powershell as my default terminal instead of git bash to see if that provides better performance

Can you get profiles of the extension host and renderer process?

EH:

Run the "Show running extensions" command
In the top right corner, there is a sort of Record button to reload with the profiler on
Repro the issue, go back to that page, save the profile

Renderer process

Help > Toggle Developer Tools
Use the Performance tab to get a profile

@lszomoru maybe git is doing something wrong here, or maybe that is normal

These were two separate tests since it was less easy to reproduce. Both are with a barebones 1.63.0-insider build with built-in extensions enabled (no other extensions). The replication in both is usually in the last 5-10 seconds. Looks like the EH profile is showing some (potentially?) long times for spawn from the git extension. Disabling Git did seem to have much better performance
Profile-20211118T164000.zip

It should be noted that the workspace I'm using has 2 git repositories which collectively have 11 submodules, totally 13 different repos to check. I'm going to try to proceed with Auto Fetch disabled as I'm guessing that's what's triggering for things to run. I'm not sure why a simple child process call to fetch git data would cause some system-wide slow downs, but 🤷

While I was having great performance last night with autofetch disabled for the whole workspace, today, I've reproduced the issue a handful of times :/ I'm mostly operational though, but do let me know if you need any more data or need me to try out an Insider build. Happy to help however I can.

Just as an update on this one, I realized that I had installed Incredibuild roughly around when I may have started having this issue. Since that works by hooking into the kernel to intercept tasks that are parallelizable and distribute the bytecode to other machines, I now have a running hypothesis it's basically slowing most processes including these simple git processes that the git extension makes.
I uninstalled Incredibuild, did a restart to ensure it was out of the kernel, and now will see if the problem arises with VSCode's Git enabled and Autofetch enabled. As of now, I have seen lots of processes open like before when returning to the window after a few minutes (I'm guessing after the auto fetch 180s timer), but it no longer hogs enough CPU to make my music stutter or notice any other performance issues.
I'll keep monitoring this and report any findings. I'll also contact the Incredibuild team to see if they have any thoughts.

Unfortunately while the issue seemed to go away at first, eventually I was able to reproduce this issue multiple times with Incredibuild completely uninstalled (I never reinstalled it after it was uninstalled). I'm still waiting to hear back from the Incredibuild team to see if they have any thoughts on the situation. It should also be noted that I would hope that this isn't a Disk IO issue as both of my drives (where VSCode is installed and where the workspace is located) are faster NVMe drives (Samsung 970 EVO 1TB M.2 NVMe and Samsung 980 PRO 2TB M.2 NVMe respectively).

Ok, I have some more concise info now:

"The issue" is defined as having the below symptoms

Background music stutters/crackles as if 100% CPU is being used (but it's not)
System input (Windows cursor and keyboard input) has high latency (cursor hangs and jumps to another location, keyboard input takes awhile [0.5+ seconds] to be processed)
The symptoms are temporary, usually start when I see a lot of child processes spawn from VS Code

The issue goes away after a system (Windows) restart
The issue comes back after several hours of having VS Code open (I'm guessing it doesn't matter if VS Code is open, but I haven't tested it otherwise)
After I start noticing the issue, it usually happens every time I come back to the window after a few minutes (mainly trying to get all repos to elapse their 180s autofetch timer)
Restarting the extension host does not resolve the issue
Closing all windows of VS Code and opening again does not resolve the issue
The issue is somewhat reproducible (no noticeable mouse/keyboard input latency, only a single crack is heard with music); I am completely okay with these symptoms on my machine, but it may indicate issues for others with less powerful machines
While it's possible Incredibuild might make the symptoms appear sooner, the issue reproduces without it
The above is tested with the Git extension enabled, autofetch is enabled (180s timer)
Chrome starts having weird load issues

Here's my gut feeling analysis after looking at the Git extension code:

The fact that Windows restarting is the only thing resets symptoms and things work fine 1-2 (more maybe? my testing wasn't that fine) makes me think there's something new with Windows 11 Dev Channel builds (or normal Windows 11) that is causing this issue. I'm not sure if WIndows or VS Code is at fault yet. I would have to test the issue on Windows 10 on another machine, then install Windows 11 and test again. It would be pretty prohibitive to take my current machine and do a clean install down to Windows 10 to see if it produces there.
The issue is exacerbated with more Git repositories in the same workspace. The code checks to see if a single Repository is idle before doing this system things, but here's no global mutex to prevent 13 updateStateModel calls happening simultaneously on 13 repos. Having some mutex to prevent these from updating simultaneously would probably be enough of a bandaid to call this "resolved" for now. There's definitely no reason to have all of these actions done all at once (which it shows there's been some work to spread out spawning processes on a single Repository
There's probably a more complicated solution which would prevent even a single stutter in background music, but it's likely not worth the time

I will be away for a week so my responses will be heavily delayed, and I won't be able to do more testing until 12/1

@lszomoru Just an update on this one:

I'm still using VS Code 1.62.3
Windows 11 insider updated from 22504 to 22509 on 12/1
I've reinstalled Incredibuild
I did not purposely do any driver updates, but I can't speak for anything that was automatically updated
I've tried leaving VS Code open over night (I used to not be able to go 8 hours without the issue start showing up)
Git extension's Autofetch is still enabled

While I'm still seeing the CPU spikes, I believe this last Windows update has prevented/fixed a bug such that these spikes from causing lagging/stuttering/halting in other system-level stuff (i.e. audio, human input devices).
So outside of monitoring the Task Manager, my symptoms have disappeared, likely due to the Windows 11 insider 22509 update. I'm not sure if that necessarily means that the issue is resolved; there are still improvements that could be made to the autofetch logic described in my last comment. However, it's no longer "breaking" for my specific setup. I'll leave it up to you @lszomoru if this issue should be closed for now; I can always reopen (or open a new one) if the issues return.

@seesemichaelj, thank you very much for the update. I will close the issue for now but please feel free to reopen it in case the issue reoccurs. I will also be looking at potential optimisations to reduce resource consumption for workspaces with multiple repositories.


Fails to match using --matching

Hi,
I had two releases with the following pattern

arm64_musl.tar.gz
arm71_gnu.tar.gz

When running ubi with --debug it found cpu architecture of type arm
It then said it had two matches and could not determine which to choose.
Adding a --matches arm64 or --matches arm71 to the command returns the error could not find any assets containing our --matching string.
Just browsing at the code, could it be that the vector at L415 is empty because it did not match 64 bit at L416 and it is trying to match against an empty vector at L439.
Should the matches string not be matched against all remaining assets if all else fails?

Thanks for the bug report! This is definitely a bug. I have a fix in progress.

This should be fixed in v0.0.26.

Awesome 👌


How do we character ranges of the clusters

Right now, when we call predictor.predict() we get the clusters as a list of lists, and the cluster heads along with their token indices. Is it possible to :

Get the cluster heads as character ranges? Meaning that  'cluster_heads': {'Momofuku Ando': [4, 5], 'Osaka': [12, 12],  'instant noodles':
[9, 10], 'Many students': [22, 23], 'Nissin': [18, 18]}}, instead of the token/word indices like [4,5], [12,12], etc. can we get the character ranges
Alternatively, can we get a separate variable that maps the token indices to tokens? Something like ['Do', 'not', 'forget', 'about'....] .
I tried looking at how the text is tokenized but couldn't exactly get that from the code. Basically for my application I need to check whether a coreference appears in a particular character range, and would like to do that accurately (with the best way to do that being using the character range)

That would be possible. @Masboes, this would be a good first issue to pick up.

in spacy you could use the token.idx property to get the token's first character index. Then token.idx + len(token) to get the last character index.
Is it useful ?

@sudarshansivakumar @shmouelsamares you can see a working example here.


Add backgroundColor option to tile layer, fix #35

thanks for the contribution!

You're welcome :)


L2-226: Stake neuron

Motivation
Create Neuron functionality.
Changes

Add createNeuron to GovernanceCanister.
Update getNeurons to expect principal.

Tests

New test for GovernanceCanister.getNeurons.
New test for GovernanceCanister.createNeuron.

@max-dfinity @ielashi please review

@peterpeterparker maybe also?

Looking good! Left some comments, which are mostly issues in the old code that I think are worth fixing now that it'll be in a more widely-used library.

@ielashi ready for another review 😄

@ielashi ready for another review. I moved the TransferError types to types/errors.ts I hope that's ok.

@ielashi ready for another review! Thanks!

@mstrasinskis please review

@ielashi thanks for the chat! I updated the PR, you will notice no changes in the ledger nor ledger types files.
Ready for another round!


styled-components theming await for loading

Hi. I'm creating a theming wrapper with styled-components and zustand(hook for persist local-storage, currTheme). My problem is although I get the correct theme from zustand but, on the first render, styled-components don't show the theme correctly.
I found this blog post which had my exact workaround to this issue. We created "isLoaded" state and await content to render. then set it to "true" in on mount useEffect and render content(isLoaded && <></>).
Of course, it worked but why? and this is not a good idea I think, because we block the whole rendering and I lose some benefit in SSR. I'm using nextjs with injecting styles to head. Am I on the correct path or any help?
@kitten thanks for always helping me <333

The more I think about it, there is nothing to do about it. We can move it to _document but I don't know how effective it would be and there would be any difference in performance and first content paint.


Add some logging about missing results on startup.

This logging tells me what things homu is still looking for
so I can restart them.

In general, you can find the pull request Homu is waiting for by looking at the PENDING state in the queue. Is there a particular reason why this information is also needed in the command line interface? As the things displayed on the command line interface is only shown by the ssh'ing user, I would like to limit its usage to purely debug purposes.

Closing this since it's old and irrelevant to my current work. I am finding it infuriating that homu doesn't log what it's thinking. Every once in a while homu seems to lose messages from buildbot and becomes unable to recover, and I have no idea what homu is thinking about the state of the world - what PR is it waiting on, how do i get it to move on?

Sorry for the harsh critique. I was debugging a stuck homu at the time.

Sorry for the inconvenience, @brson. It must have been an irritating experience.
I find it worrisome that the Homu queue is continuously being stuck. IMO in most cases posting @bors retry on the PENDING PRs would be sufficient, because the losses of the "done" messages from GitHub and Buildbot make Homu wait for them indefinitely. So it should be retried. (Or you should let GitHub and Buildbot send the packets again, but it would be more or less harder.) But I guess the problem the Rust project is having is something different? How did you resolve the problem this time?
More logging will be definitely useful. In fact, some logging feature had been added by @huonw, but it seems to be not enabled by default. I will tackle this problem soon.
Again, sorry for the annoyance! I think I was too lazy on this matter. If you experience another problem again, please don't hesitate to bug me. I'll try to be there.

@barosl I have never @bors retry in this situation because i don't want to go through another multi-hour build when the build did complete.
The problem we had yesterday we think turned out to be a bug in our buildbot deployment: homu was requesting builds but buildbot wasn't doing them because of a misconfiguration. So ... false alarm. Very sorry :bow:

@brson Ah, I see! That's fortunate for Homu's side. :stuck_out_tongue_closed_eyes:
I had an idea for the problem you're describing. Homu is now capable of reading the build states from the GitHub statuses. It was primarily done to support Travis in more general fashion (previously Travis -> Homu, but now Travis -> GitHub -> Homu), but I think it could be also applicable to our current Buildbot deployment.
We may add a Buildbot configuration to post its build results to GitHub directly, and Homu just retrieves the statuses from GitHub. In this way Buildbot will look more "natural" in the GitHub interface, because everyone can see what's going on in Buildbot from the PR page. Also, it would contribute to separation of concerns, as Buildbot only deals with GitHub, and Homu only looks at GitHub. Plus it would help debugging issues like this.

@barosl ah, that sounds like a neat simplification, yeah!


(WIP) Remove excess checking and make tagging earlier.

The tags can't be passed to create() in aws-sdk-v1 but we can move the call closer to creation.
This also reduces the duplicate status checks.

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3911/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3912/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3913/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3914/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3919/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3920/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3921/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3922/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3923/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3924/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3926/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3935/

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/3940/


RichTextLabel "Visible Characters" setting does not hide bullet characters for BBCode lists

Tested versions

Reproducible in: Godot v4.2.1.stable

System information
Godot v4.2.1.stable - Windows 10.0.19044 - GLES3 (Compatibility) - NVIDIA GeForce GTX 1660 SUPER (NVIDIA; <IP_ADDRESS>76) - AMD Ryzen 3 3200G with Radeon Vega Graphics (4 Threads)
Issue description
I am trying to create a textbox with a typewriter effect using the "visible_characters" parameter of a RichTextLabel. Each paragraph of dialog is an item on a list made with BBCode, for aesthetic purposes. The trouble is that the bullets of the list still appear even when visible_characters is low and therefore they shouldn't be visible yet.
I don't know what causes this behaviour, and I can't find anything in the documentation that indicates that this is intended.
Hopefully this screenshot of the included MRP will help you see the problem:

Steps to reproduce
Simply run the textbox scene and the bullets should appear immediately despite visible_characters being set to 0 at first.
Minimal reproduction project (MRP)
dialogbox.zip

Should be fixed b y https://github.com/godotengine/godot/pull/90472


Open File: unable to open a file with spaces in name if absolute path is provided

Steps to Reproduce:

open quick open (Cmd+P)
paste the absolute path of an existing file that contains spaces (for me: C:\Program Files\Microsoft VS Code Insiders\resources\app\out\vs\code\electron-main\main.js)

=> no results shown to open

I was teaching my students about workflows for fixing bugs in VSCode, and used this bug as an example.  I wrote them a guide at https://github.com/humphd/vscode/tree/good-first-experience-issue-42726#walkthrough-fixing-a-bug-in-visual-studio-code.
I was going to do a PR, but it looks like one of them beat me to it, so I'll let @aushakou continue.

Good findings 👍

Doesn't work for me. Tested on Win10 and Ubuntu

@roblourens @chrmarti this probably got lost when moving to RipGrep for search. The idea is that a full absolute path of an existing file should always return a result, at least from quick open (not sure about other use cases). Let me know if this should be baked into the file-search or if we should add this to the picker as a special case and let the search service behave as before.

This works when there are no spaces in the path on Mac. So the code is still there, we just need to fix it.

@chrmarti can you point me to the code?

Here: https://github.com/Microsoft/vscode/blame/master/src/vs/workbench/services/search/node/fileSearch.ts#L569
The path gets there without the spaces due to: https://github.com/Microsoft/vscode/blame/master/src/vs/base/parts/quickopen/common/quickOpenScorer.ts#L314

@chrmarti thanks, my fix was not complete it seems.


feat(QueryBuilder): add timeout(ms, options?)

This PR adds the function timeout(ms, options?) to the QueryBuilder. It's possible to set timeouts for specific queries.
The timeout feature was first implemented in Knex on Feb 15, 2016 (https://github.com/tgriesser/knex/commit/6dfe653c42002e706504d8f56285cf5b500ff4e2#diff-2ee33e306d47d27f6349fbefe16e28e0).
This PR is related to the issue #905

Coverage decreased (-0.02%) to 95.165% when pulling cd7566ee97439e7f2ac9f8d5506679a89163ef8b on hlibco:master into 42dba9a6c48fb2ea1bfbc29b75fd1143f9b26b19 on Vincit:master.

Thank you for working on this. I'm willing to merge this PR. but each feature must have tests and documentation. A simple integration test case will be enough and a link to knex documentation from the objection docs (like other knex methods have been linked).

@koskimas Thanks for reminding me about tests and docs. I sent a new commit taking them into account and also the comment of @kibertoad - It's my first time working with Knex and Objection, let me know if I've placed something in the wrong place.

@hlibco I added some notes. There were couple of unneeded methods, and I added an idea for a better test. When those are fixed, I'm ready to merge this. Thanks again!

I'll add the missing test myself then...


dhara_nand_is_free returns true at I/O error

Thanks for the nice use case of the dhara library, it was really helpful!
Found a small bug here:
https://github.com/aloebs29/flash_management/blob/fcd5c533b9a14afd4b84f954e5ac8a2556997f88/src/dhara/nand.c#L84
This line should be is_free = false;

I'm happy to hear it was helpful! Thanks for catching the error.


Skip REST function call when eventSource is null

Change summary:

Trigger REST function call only when eventSource is NOT null ('USER' or 'BLOCK')
Move value changed event emission logic to db side

cc: @jakepyo
Related issues:

https://github.com/ainblockchain/ain-blockchain/issues/1138

Thanks for the review. Now merging..


Periodic CPU hogging

Brief Description: The server loads a CPU core 100% for a few minutes periods. After about 10 hours of running, CPU is fully loaded almost constantly. During this time the server stops to respond to any connections. Finally one has to kill the 'python' process and restart the OB server.
Operating System (OS and version): Debian Linux 8.4
OpenBazaar version: Server 0.1
Hardware: Hyper-V virtual machine on 12-core Intel Xeon @ 2.6 GHz with one physical core and 8 GB RAM assigned to the VM.
Reproducible (Always / Almost Always / Sometimes / Rarely / Couldn't Reproduce): Always
Steps to reproduce:

Run the server.
Observe CPU load.

Observed Behavior: Constant 100% CPU load in a few minutes periods, longer after OpenBazaar server is running for more than 10 hours. During these periods the server does not respond to a remote controlling client, also no P2P connections are accepted.
Expected Behavior:
Additional info (links, images, etc go here):

+1
Operating System (OS and version): Ubuntu 14.04.4 LTS
OpenBazaar version: Server 0.1
Hardware: Digital Ocean VPS 1 GB Memory / 30 GB Disk
Reproducible (Always / Almost Always / Sometimes / Rarely / Couldn't Reproduce): Always
Steps to reproduce:

Run the server.
Observe CPU load.

Observed Behavior: After running for awhile (ex. overnight), I'm no longer able connect via the client unless the OB server is bounced.

Yep, working on it right now!

The same was on 1.1.0 release  on raspberry pi. I had to restart pi to get connection. Python service load was 100%, no connection to the server.
Did 'git pull' command and updated server part a day ago.
Now is ok. 24 hours - works fine, connection is ok.

I have the same problem with OSX (10.11.4). This is a critical bug for me as I can not leave my laptop server running non-stop at 100% cpu usage.

Same on Debian 8 in plesk VPS

Some code was committed today to the server repo. Preliminary indications are that this problem is fixed with a bonus significant performance improvement.

It works fine after update on raspberri pi 3.  no more 100% cpu load. Works faster fro sure. Thanks!

I'm gonna let my nodes run for the rest of the day. If they're fine, we should close this issue.

Working fine after one day.

Same here

No problems on OSX after 24 hours.

Ok. I'm closing this issue. Great job @cpacia


Make event IDs persist until the event is resolved

This pull request makes Sensu event IDs persist, for one or more event occurrences, until the event is resolved. This closes https://github.com/sensu/sensu/issues/1196
This makes event IDs useful, beyond just an event occurrence identifier.

:+1:

I'm excited about this change. I can't wait to see what people build with this.
/cc @solarkennedy and @joshzitting

I need this, this is great!


Unable to build on Threadripper Ubuntu Proxmox VM

I'm trying to build AutoGPTQ on a Ubuntu 22.04 VM running through Proxmox on a Threadripper with an RTX 3090.
pip install -v . yields the following error:
Using pip 23.3.1 from /home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/site-packages/pip (python 3.10)
Processing /home/nrkbeta/dev/projects/AutoGPTQ
  Running command python setup.py egg_info
  Traceback (most recent call last):
    File "/home/nrkbeta/dev/projects/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 1475, in <module>
      gen_module_search(args.r, args.p, [2,3,4])
    File "/home/nrkbeta/dev/projects/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 1418, in gen_module_search
      gen_and_compile(n,m,t,n,mb,tb,1,mu,tu,p,u,bits=bits, gs=gs, module=True)
    File "/home/nrkbeta/dev/projects/AutoGPTQ/./autogptq_extension/qigen/generate.py", line 800, in gen_and_compile
      subprocess.check_output(["./autogptq_extension/qigen/mmm", f"{n}", f"{m}", f"{t}", f"{bits}", f"{gs}"])
    File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 421, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 526, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['./autogptq_extension/qigen/mmm', '1', '4096', '4096', '4', '-1']' died with <Signals.SIGILL: 4>.
  Traceback (most recent call last):
    File "/home/nrkbeta/dev/projects/AutoGPTQ/setup.py", line 103, in <module>
      subprocess.check_output(["python", "./autogptq_extension/qigen/generate.py", "--module", "--search", "--p", str(p)])
    File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 421, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 526, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['python', './autogptq_extension/qigen/generate.py', '--module', '--search', '--p', '32']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/nrkbeta/dev/projects/AutoGPTQ/setup.py", line 105, in <module>
      raise Exception(f"Generating QiGen kernels failed with the error shown above.")
  Exception: Generating QiGen kernels failed with the error shown above.
  Generating qigen kernels...
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/nrkbeta/miniconda3/envs/axolotl/bin/python -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize

  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)

  __file__ = %r
  sys.argv[0] = __file__

  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"

  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/home/nrkbeta/dev/projects/AutoGPTQ/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' egg_info --egg-base /tmp/pip-pip-egg-info-g9fixjz3
  cwd: /home/nrkbeta/dev/projects/AutoGPTQ/
  Preparing metadata (setup.py) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Runningpython autogptq_extension/qigen/generate.py --module --search --p 32 manually gives the following output:
1 4096 4096 1 1024 32 1 16 16 32 1 4 -1
Before failing with:
Traceback (most recent call last): 4 -1
  File "/home/nrkbeta/dev/projects/AutoGPTQ/autogptq_extension/qigen/generate.py", line 1475, in <module>
    gen_module_search(args.r, args.p, [2,3,4])
  File "/home/nrkbeta/dev/projects/AutoGPTQ/autogptq_extension/qigen/generate.py", line 1418, in gen_module_search
    gen_and_compile(n,m,t,n,mb,tb,1,mu,tu,p,u,bits=bits, gs=gs, module=True)
  File "/home/nrkbeta/dev/projects/AutoGPTQ/autogptq_extension/qigen/generate.py", line 800, in gen_and_compile
    subprocess.check_output(["./autogptq_extension/qigen/mmm", f"{n}", f"{m}", f"{t}", f"{bits}", f"{gs}"])
  File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/nrkbeta/miniconda3/envs/axolotl/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./autogptq_extension/qigen/mmm', '1', '4096', '4096', '4', '-1']' died with <Signals.SIGILL: 4>.

I'm also not able to make it work by running the pre built pip package.
Any hints?

Thank you, could you run python -c "import platform; print(platform.machine())" and give the result here?
In the meantime you can try to compile with DISABLE_QIGEN=1 pip install -v -e . from the main branch (https://github.com/PanQiWei/AutoGPTQ/pull/471).

Hi fxmarty,
A fresh git pull and DISABLE_QIGEN=1 did the trick! Thanks!
I came into this through axlotl, which has torch 2.0 specified as a dependency. I had to upgrade to 2.1 to get DISABLE_QIGEN=1 pip install -v -e ., as well as reinstall flash_attn in order to get axlotl working afterwards. Might be of use to others.


Return default value if the input is None

Initial Checks

[X] I have searched Google & GitHub for similar requests and couldn't find anything
[X] I have read and followed the docs and still think this feature is missing

Description
Let's take this simple model as an example
>>> from pydantic import BaseModel
>>>
>>>
>>> class Foo(BaseModel):
...     bar: list[int] | None = [0,1,2]
...     baz: list[str] | None = ["a", "b", "c"]
...
>>> Foo()
Foo(bar=[0, 1, 2], baz=['a', 'b', 'c'])

>>> Foo(bar=[1,2], baz=["a","b"])
Foo(bar=[1, 2], baz=['a', 'b'])

>>> Foo(bar=None, baz=None) # Explicitly pass `None`
Foo(bar=None, baz=None) # Current Behavior

>>> Foo(bar=None, baz=None) # Explicitly pass `None`
Foo(bar=[0, 1, 2], baz=['a', 'b', 'c']) # Feature request: Return default value if the input is `None`, possibly a model setting in ConfigDict?

There already seems to be some demand for this:

https://github.com/pydantic/pydantic/issues/1851
https://stackoverflow.com/questions/63428454/can-i-make-a-default-value-in-pydantic-if-none-is-passed-in-the-field-without-us

Affected Components

[ ] Compatibility between releases
[X] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[ ] JSON Schema
[ ] Dataclasses
[X] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

@Ravencentric,
I think it probably makes sense to use a custom validator for this. For example:
from pydantic import BaseModel, ValidationInfo, field_validator

class Foo(BaseModel):
    bar: list[int] | None = [0,1,2]
    baz: list[str] | None = ["a", "b", "c"]

    @field_validator('bar', 'baz')
    @classmethod
    def check_alphanumeric(cls, v: str, info: ValidationInfo) -> str:
        if v is None:
            return cls.model_fields[info.field_name].default
        return v

print(repr(Foo()))
# Foo(bar=[0, 1, 2], baz=['a', 'b', 'c'])

print(repr(Foo(bar=[1,2], baz=["a","b"])))
# Foo(bar=[1, 2], baz=['a', 'b'])

print(repr(Foo(bar=None, baz=None))) # Explicitly pass `None`
# Foo(bar=[0, 1, 2], baz=['a', 'b', 'c'])

I haven't seen an abundance of demand for this feature. If there's significant community interest, we could reconsider making this a flag on a field, but for now I think the field validator approach (a change in user code) makes the most sense!

@sydney-runkle
Thank you for the example.
Can we do the same in BeforeValidator? I love the Annotated + Validator pattern as it provides stronger reusability. But I couldn't access the default value information in the validator function in this case.
Specifically I'd like to do something like this.
def validator(v: Any, info: ValidationInfo) -> Any:
    print(
        info
    )  # ValidationInfo(config={'title': 'MyClass'}, context=None, data={}, field_name='field')
    if v is None:
        default = get_default_somehow()
        return default
    return v

class MyClass(BaseModel):
    field: Annotated[int, Field(default=0), BeforeValidator(validator)]

MyClass.model_validate({"field": None})

Although I can explicitly add BeforeValidator like following, writing default value twice looks a bit weird.
def null_default_validator(default: Any) -> BeforeValidator:
    def validator(v: Any) -> Any:
        if v is None:
            return default
        return v

    return BeforeValidator(validator)

class MyClass(BaseModel):
    field: Annotated[int, Field(default=0), null_default_validator(default=0)]

MyClass.model_validate({"field": None})

I have to say that I would have wanted this feature.
Here is my usecase - I'm getting a serialized api request to my service - and the sender doesn't have the option to exclude undefined fields, so they have to send them as nulls which get serialized to None in python.
The thing is - that I already have a default value which I want this field to take in the case it is None.
Right now the default value works only for the case it haven't been sent(undefined).
I want an option to configure this behavior so it'll use the default on None as well - it will give me more freedom.
Otherwise what I need to do is to manually populate the field in the validator - or in usage always check if the field is not None which is redundunt.

@yotaro-shimose and @tsalex1992 - Just in case you might find it helpful, our team encountered a slightly similar issue and came up with another possible solution.
Sadly, it doesn't fit with the Annotated + Validator pattern but it is fairly reusable. With your Foo example above, for instance:
# some/path/to/utils.py
def generic_validator_function_with_default(class_ref, value, info):
    return (
        value
        if value is not None
        else get_default_value_from_field_config(class_ref, info)
    )

def get_default_value_from_field_config(class_ref, info):
    field = class_ref.model_fields[info.field_name]
    return field.default_factory() if callable(field.default_factory) else field.default

# some/path/to/models/foo.py
from some.path.to.utils import generic_validator_function_with_default

class Foo(BaseModel):
    bar: list[int] | None = [0,1,2]
    baz: list[str] | None = ["a", "b", "c"]

    @field_validator('bar', 'baz')
    @classmethod
    def handle_field_default(cls, value, info):
        return generic_validator_function_with_default(cls, value, info)

@saurookadook
Thank you for your suggestion. I love that example as it shows good reusability.
As your workaround says, it might be good to have reference to the class to be wrapped in function validators. Hopefully it works with python Annotated syntax so that pydantic maintainers can implement it.

Having this syntax would be great for shared library code as well
Is there anything one can do to help out on this issue?

@MarkusSagen,
I'd be happy to review a PR with support for this via a field flag if the API is pretty clean, given that there seems to be a decent amount of requests for this. I think we want to do this on the field level and not the config level, but I'm not sure...

In the API I'm dealing with, I needed it globally on every model so a model_config level would be appreciated

Thank you @sydney-runkle
Do you have any pointers for which file the changes would be placed in
I've not yet worked with the internals of Pyadntic

Being able to set defaults like this would also be useful for the cases when using ConfigDict(from_attributes=True) mode, which is going to remove the need to validate each field which could possibly be None  in the database.

After a bit of tinkering, I arrived independently arrived at a similar solution to @saurookadook which is added to our common BaseModel that contains our configuration and util functions:
    @field_validator("*", mode="before")
    @classmethod
    def not_none(cls, v, val_info):
        """
        Convert None to Default on optional fields.
        Why doesn't Pydantic have this option?
        """
        field = cls.__fields__[val_info.field_name]
        if v is None and (default := field.get_default(call_default_factory=True)) is not None:
            return default
        return v

This could also be broken out into a function and called with _not_none = field_validator("my_field", "my_field2")(not_none) or used inside a WrapValidator, but in our case any Optional field can potentially be populated with None when we'd rather have our default values.
I'd still rather have a Field(replace_none_with_default=True) option though.

Here's another solution.
`from pydantic import BaseModel, model_validator
from typing import Optional
class TestModel(BaseModel):
name: str
age: Optional[int]=0
@model_validator(mode='before')
@classmethod
def remove_none(cls, data):
    if isinstance(data, dict):
        return {k: v for k, v in data.items() if v is not None}
    return data

data = {"name": "test", "age": None}
model = TestModel(**data)`

See also this: https://github.com/pydantic/pydantic/discussions/8585
But for that, we would need to have access to the model's class inside ValidationInfo.

After some studies I came to this result, it works, but it is extremely inelegant, it would be great if pydantic already did this by default, having to create this subclass is not the best thing in the world as I have to duplicate this in all my python packages, or create another python package just for this code below, either way, it's not elegant
from typing import Dict, Any
from pydantic import BaseModel
from pydantic_core import PydanticUndefinedType

class CustomBaseModel(BaseModel):
    @model_validator(mode='before')
    def set_defaults_for_none(cls, values: Dict[str, Any]):
        fields = cls.model_fields
        for field_name, field_info in fields.items():
            value = values.get(field_name)
            default_value = (
                field_info.default
                if not isinstance(field_info.default, PydanticUndefinedType)
                else None
            )
            default_factory = (
                field_info.default_factory
                if not isinstance(
                    field_info.default_factory, PydanticUndefinedType
                )
                else None
            )

            if value is None:
                if default_value is not None:
                    values[field_name] = default_value
                elif default_factory is not None:
                    values[field_name] = default_factory()

        return values

@Ravencentric,

I haven't seen an abundance of demand for this feature. If there's significant community interest, we could reconsider making this a flag on a field, but for now I think the field validator approach (a change in user code) makes the most sense!

I typically use Pydantic schema classes in GPT prompts to ensure the response is in JSON format. However, when the GPT model doesn't find a value for a field, it initializes the field with None instead of using the default value. This issue suggests the need for a flag in the field class to handle such cases appropriately.

Semi-related, the example in this section: https://docs.pydantic.dev/latest/concepts/json/#partial-json-parsing

Going back to the @Ravencentric's original example, IMO the current behavior is actually preferable as written because the type annotation marked the field as optional (list[int] | None). In that case it makes sense that None should be an allowed value that you can explicitly pass in.
However I think in most cases if you have a default value to use for a field then you don't want that field to be optional.
For example:
class Foo(BaseModel):
    bar: list[int]= [0,1,2]

Foo()
# Foo(bar=[1,2,3])

Foo(bar=None)
# raises ValidationError

In that case it would be preferable if it used the default value instead of raising an error.
The model_validator workaround that others mentioned does work, but that also raises the question why doesn't a mode="before" field validator work? That part is very unexpected that the None type validation runs before the validator when the validator was supposed to happen before.

Based on the example suggested by @sydney-runkle, this seems to be working well and should contend everyone in this thread :slightly_smiling_face: :
from pydantic import BaseModel, field_validator
from pydantic_core import PydanticUseDefault

class Foo(BaseModel):
    bar: list[int] = [0,1,2]
    baz: list[str] = ["a", "b", "c"]

    @field_validator("*", mode="before")
    @classmethod
    def none_to_default(cls, v):
        if v is None:
            raise PydanticUseDefault()
        return v

assert Foo(bar=None, baz=None) == Foo()

Hey @sydney-runkle, I'd like to work on this. I agree with @dcosson's suggestion, so I made some local changes to pydantic-core, and this is what I have so far:
from typing import Optional
from pydantic import BaseModel

class Foo(BaseModel):
    bar: list[int] = [0, 1]
    baz: list[int] = [2, 3]
    daz: list[int] = [4, 6]
    caz: Optional[list[int]] = [0, 0]
    faz: Optional[list[int]] = [7, 8]

print(Foo(bar=[9, 0], daz=None, caz=None))
# bar=[9, 0] baz=[2, 3] daz=[4, 6] caz=None faz=[7, 8]

I think an API like the following would work well for everyone,
class Foo(BaseModel):
    # explicitly `True`, overrides `Config`
    bar : list[int] = Field(default=[1,2], none_as_default=True)

    # explicitly `False`, overrides `Config`
    baz : list[int] = Field(default=[1,2], none_as_default=False)

    # behaviour depends on `Config`
    bar : list[int] = Field(default=[1,2])

    class Config:
        # default `True`
        none_as_default = False

While raising PydanticUseDefault() works well for us in practice, this is now causing errors in our test suite:
pydantic_core._pydantic_core.SchemaError: Uncaught UseDefault error, please check your usage of default validators.

We tend to avoid adding new parameters/flags to the core API of Pydantic (fields, config, model_dump_*/model_validate_* methods) if this can be implemented relatively easily using validators. Moreover, one could then request I want to use the default value if MyNoneTypeSingleton is provided, making the none_as_default not extensible.
Here is a simple way to implement such a behavior:
from pydantic import AfterValidator, BaseModel
from pydantic_core import PydanticUseDefault

from typing import Annotated, TypeVar, Any

def default_if_none(value: Any) -> Any:
    if value is None:
        raise PydanticUseDefault()
    return value

T = TypeVar('T')

DefaultIfNoneV1 = Annotated[T, AfterValidator(default_if_none)]
DefaultIfNoneV2 = Annotated[T | None, AfterValidator(default_if_none)]

class Foo(BaseModel):
    bar: DefaultIfNoneV1[list[int] | None] = [0, 1, 2]
    # or:
    # baz: DefaultIfNoneV2[list[int]] = [0, 1, 2]

Foo(bar=None)
#> Foo(bar=[0, 1, 2])

However PydanticUseDefault should be documented, it’s not the case today.

Sorry about raising this when things are already being implemented, but is this really a valid feature? I stand with @dcosson's comment and to me this is like breaking type checking. Allowing None when a field is not optional will never work in a strongly typed language and the point of pydantic is to provide such a guardrail in typing from what I understand. Also, how does one deal with type-checking like mypy or pylance with this feature?
Default values set like the following should be sufficient, essentially identical to having default arguments for the initialiser:
class Foo(BaseModel):
    bar: list[int]= [0,1,2]

Foo()

Even if we really want to go with this feature, that none_as_default config value should default to False to keep the existing behaviour.

@changhc Only if you use Pydantic as some kind of dataclass in your code and ignore the other important feature of Pydantic: JSON serialization/deserialization.
Sometimes, you want to deserialize a structure where fields can be null in the JSON — which isn't always generated by your own service/code —, but you don't want to always have to handle the possible None values in your code (e.g. only handle list[str] instead of list[str] | None).
So, in your code, the values always match your type hints and only the frontier between your code and the JSON input handle the None value.

https://github.com/pydantic/pydantic/pull/11060 is a complete rewrite of the validators documentation, and PydanticUseDefault is now documented in the utilities types.

While raising PydanticUseDefault() works well for us in practice, this is now causing errors in our test suite:
pydantic_core._pydantic_core.SchemaError: Uncaught UseDefault error, please check your usage of default validators.

@dromer this error happens when no default value is provided for a field.
Marking the issue as resolved. Please let us know if PydanticUseDefault does not fulfill your needs.

@dromer this error happens when no default value is provided for a field.

That is not the case in our code. We did explicitly set a default value and expected it to be used.
It works in production but our test suite fails with this error on the same object.
I will try to collect a minimum test case and create a new ticket later.


Checkbox isn't working inside a Dialog

Dear Evergreen-Team,
First of all, I would like to thank you for your hard work. I have long searched for a suitable UI framework for React, which I have now found. Thank you for that.
Currently I use the version v4.0.0-39
Unfortunately, the checkbox does not work inside a dialog. I can click the checkbox, but it remains empty. In the linked SandBox.io Project it works, but inside Chrome Version 69 or Safari, it's doesn't work, why?

How can the problem be solved?
Moreover, how can I use the component FormField with validation? (I haven't found anything yet)
A third question: How can I set the autofocus to the first TextInputField? (I don't want the Cancel button in the top left in focus)

Hi @ITforHome. Thanks for giving Evergreen a try and posting your issue here.
The checkbox is working for me in Chrome Mac OS X, but you should consider a controlled usage of it. Not sure why it's not working. Could you provide your system OS and confirm this behavior with a gif?
Autofocus bug
It seems you have found a bug with the autoFocus/autofocus prop not being passed properly on the TextInput inside of the TextInputField. I recommend using tabIndex={0} for now until we get this fixed.
No need to nest inside FormField
There is no need to nest TextInputField inside FormField. Take a look at my example here: https://codesandbox.io/s/m9opjm10qx
I am not getting the example to load glamor CSS properly and seeing this results in CodeSandbox. Which is something I need to take a closer look at:

@jeroenransijn Thanks for your quick answer.
Currently I use MacOS Mojave 10.14 and Chrome Version 69.0.3497.100 (that's the last version of Chrome)

AutoFocus Bug
Where exactly do I have to insert the tabIndex={0} in my dialog so that the Cancel button is not in focus?
FormFields
Thanks for your example Project on SandBox.io. But except for the import statement from FormField, I can't  find anything about FormField, Validation and ValidationMessages in the file. Can it be that you have linked me to another project?
I would be very happy if you could show me how I could add a formfield with validation and messages to my dialog

Yes, you have right. I don't know, why we have CSS Issues on SandBox.io. But this is the same file in my own Project in Visual Studio Code as Environment...

I have updated the CodeSanbox (https://codesandbox.io/s/m9opjm10qx) to show where to put tabIndex={0}

FormField is mainly a implementation detail, you often won't use it directly. TextInputField is a combination of TextInput and FormField already. Note that all components are presentational, and there is no out-of-the-box form validation — you will need to manually manage that in your application.
TextInputField
The TextInputField component combines a TextInput with a label and optional
description, validationMessage and hint.
Label and description
<TextInputField
  label="Default text input field"
  description="This is a description."
  placeholder="Placeholder text"
/>

A hint is under the text input
<TextInputField
  label="Default text input field"
  hint="This is a hint."
  placeholder="Placeholder text"
/>

Required text input field
<TextInputField
  id="ids-are-optional"
  label="A required text input field"
  required
  description="This is a description."
  placeholder="Placeholder text"
/>

Invalid field
<TextInputField
  isInvalid
  required
  label="A required text input field"
  description="This is a description."
  validationMessage="This field is required"
/>

Controlled usage
The TextInputField component works the same as using input directly.
Use e.target.value to get the value of the component on change.
<Component initialState={{ value: '' }}>
  {({ state, setState }) => (
    <TextInputField
      label="A controlled text input field"
      required
      description="This is a description."
      value={state.value}
      onChange={e => setState({ value: e.target.value })}
    />
  )}
</Component>

I think I know why the examples are not completely working — the package.json has v3 of Evergreen as a dependency:

@jeroenransijn Thank you for your detailed answer. Only the question remains, why the checkbox doesn‘t work. Maybe you already have an idea for the cause?

@jeroenransijn Update: I solved the issue. I have forgotten to set a controlled usage for this.

How can i set another TextColor on the Cancel Button in the Dialog Footer. I want to set the TextColor to grey and not to blue... But the Apperance "Minimal" should be kept, how can i do that?

@jeroenransijn For the FormValidation, i use the tutorial: "Form Validation in React" to implemente this. But i have trouble with that, to implemente this to my React-App. I am aware that it takes time to help me, but I would be very grateful for it because I have little knowledge about that. Because the files are too large to insert here, I created SandBox.io project. This is exactly the same project that I use myself:

Could you help me, to solve my Problem? maybe you could use the completed form validationas an example of how to bring form validation and the evergreen UI together...

@jeroenransijn Update: I solved the Issues with the FormValidation... Do you want to use that as an example, how to use FormValidation with the Evergreen-UI TextInputField Component?

@ITforHome currently not possible to change the text color of a minimal button without a hack.
As for the form validation, It might be a useful example to have in codesandbox.io!

@ITforHome closing this since it seems you resolved your issue :)


tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed

tensorflow==2.2.0
tensorflow-addons==0.10.0
keras-bert>=0.81.0
The above are some installation package versions，when I run "python train_am.py", error "tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed " happended.
Thanks for your help.

Maybe your GPU is occupied


stucked on windows

os: Windows
I just intall cargo install lsd but stuck when call lsd after install.
it give users a bad first experience
after 1 years, the command outputs:
❯ lsd
lsd: .\ModifiableWindowsApps: 访问控制列表(ACL)结构无效。 (os error 1336).

really bad first impression

Potentially related to #375


Form Recogniser not giving ideal output of tables consisting symbols (checkmarks and crosses in this case)

This issue is for a: (mark with an x)
- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I had added in a PDF document which had the below table.

Now the result which I got from Form Recogniser was used in the predocs.py script which converted the table to a HTML table as shown below -
Table 4 - Mother, Son, Daughter in Law

Ownership of	Income Of

Mother	Son	Mother + Son	Mother + Son + Daughter in Law	Son + Daughter in Law

Mother	:selected:	X * :selected:	:selected:	X * :selected:	X* :selected:

Son	:selected:	V :selected:	:selected:	V :selected:	V :selected:

Mother + Son	:selected:	:selected:	:selected:	:selected:	:selected:

Mother + Son + Daughter in Law	:selected:	:selected:	:selected:	:selected:	:selected:

          * - Yes, if Son is sole child

My question is, from where is that "V" coming from? Is there something we can do so that the "V" which is coming can be avoided?
Any log messages given by the failure

NA

Expected/desired behavior

Where V has come in the table parsed by form recogniser, ":selected:" should have come.

OS and Version?

Linux (Ubuntu 22.04).

azd version?

azd version 1.2.0 (commit 99ea7577f0df0df2ba34b677da189fafba18c0f7)

Versions

NA

Mention any other details that might be useful
NA

@devashish-gopalani-cognoai Does that V happen when you use Form recognizer in the portal as well? Assuming so, please post here: https://learn.microsoft.com/answers/tags/440/document-intelligence
If it only happens with form recognizer called by our script, let us know.

@pamelafox yup, it is happening over there too. I'll raise it over there. Closing this issue.


Further file and directory name sanitization needed

Env: Ubuntu, Python 2.7.10
venture-001 has sections with forward slashes in their names, such as "Grading / Notation", "Week 1 / Semaine 1", "Final Project / Project final", and so on.
The script seems to work okay for most of the sections, but fails on "Quizzes / Quizz". Here's the trace:
Downloading Quizzes...
Downloading Quizzes / Quizz
Traceback (most recent call last):
  File "dl_all.py", line 335, in <module>
    quiz_info = get_quiz_info(session, i[0], i[1])
  File "dl_all.py", line 175, in get_quiz_info
    render(session, os.getcwd()+'/'+category_name)
  File "dl_all.py", line 44, in render
    f = open(path+'.html', 'wb')
IOError: [Errno 2] No such file or directory: u'/home/username/Documents/coursera-downloads/venture-001/Quizzes / Quizz.html'

Should be an easy fix to replace the / with a dash or something.

I am getting the same error.
Env: Windows,  Python 2.7.10
How do I solve this issue?

You can just rewrite the beginning of get_quiz_info function like this:

def get_quiz_info(session, url, category_name):
    session.get(url)
    wait_for_load(session)
    safe_category_name = category_name.replace('/', '_')
    render(session, os.getcwd()+'/'+safe_category_name)

This project has been deprecated for a while. Just doing some cleaning.


Error running npm init

I am attempting to run npm init svelte@next my-app, but I am getting the following error:
npx: installed 5 in 1.991s
internal/modules/cjs/loader.js:1174
      throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: C:\Users\kkelly\AppData\Roaming\npm-cache\_npx\72016\node_modules\create-svelte\bin.js
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1174:13)
    at Module.load (internal/modules/cjs/loader.js:1002:32)
    at Function.Module._load (internal/modules/cjs/loader.js:901:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
    at internal/main/run_main_module.js:18:47 {
  code: 'ERR_REQUIRE_ESM'
}

Please use the provided issue template, which includes information about your Node version. I suspect you're using one that's too old and lacks the proper ESM support.

You are correct.  Outdated Node.  Thanks.

Please add a short little note about this on the homepage and/or the getting started section of the docs, just to save some of us from having to revert here to see the problem...


Improve path handling for URLs with or without trailing slash

Enhance path assignment to support URLs with or without a trailing slash, ensuring consistent path formatting.

Thanks!


Visual Studio Code: Lua.workspace.library (settings.json) cannot handle Environment Variables?

Describe the bug
"Lua.workspace.library": [
"${env:CUSTOM_PATH}/Lua Library/"
],
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):

OS: Windows
Client: VSCode

Additional context
Add any other context about the problem here.
Provide logs
Find the log at https://github.com/sumneko/lua-language-server/wiki/Default-log-path
    workspace = {
        checkThirdParty = true,
        ignoreDir = {
            [1] = ".vscode",
        },
        ignoreSubmodules = false,
        library = {
            [1] = "${env:CUSTOM_PATH}/Lua Library/",
        },
        maxPreload = 1000000,
        preloadFileSize = 1000000,
        useGitIgnore = true,
        userThirdParty = {
        },
    },

Proposed fix: #795


feat(#14): postbox

Build is not passing due to a missing schema generation process.
But that is not related to this feature/PR so Im merging this!

Build is not passing due to a missing schema generation process.
But that is not related to this feature/PR so Im merging this!

Thanks!