Allow the use of hostNetwork=true for pilot so webhooks can be accessing using different a different CNI in EKS and other cloud based k8s distros

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
I am currently trying to deploy istio alongside Weave CNI on an EKS cluster. This is primarily because I like istio, but also need to use Weave as I understand it is one of the only CNIs that allow Pods to communicate using multicast internally.
I have had istio working fine using the aws vpc cni, but since moving to weave, I get the following error when trying to setup the gateway:
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": Address is not allowed

As I understand it, this is a limitation with EKS as weave cannot be installed on the control plane and therefore cannot reference pods within the cluster unless they expose their port on the host network.
I am not sure whether this is a simple change to the helm chart to deploy the pilot pod with hostNetwork: true within the spec, but I get the error:
a"2023-06-29T17:43:32.095327Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2023-06-29T17:43:32.095354Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1111127) is up-to-date. No change required.

It might be a really simple fix... it might not.. and I appreciate it might be an edge case... but it would be good to get this working alongside other cni's in cloud based k8s distros...
Describe alternatives you've considered
AWS CNI does not work due to it not being supported... I am going to make the wild and dangerous assessment that this is going to be the same with any CNI provider in something like EKS.
Affected product area (please put an X in all that apply)
[ ] Ambient
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[X] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context
None

I don't think we should do this. Running as hostNetwork is not safe and I don't support encouraging it. You can do this on your own if you want -- and are willing to deal with the repercussions. See https://github.com/istio/istio/issues/34021 for prior discussion and link for how to to do it.

Hey @howardjohn, thanks for the quick response! I do understand, but it makes it kind of difficult to adopt anything that doesn't chain to the AWS CNI... Our cluster isn't publicly facing and we already have to use this on the OperatorHub Deployment and Metrics Server as they have a similar issue.
If it is as simple as that - would you accept a PR which would highlight in the values file that setting this value to 'true' incurs risk? If possible it would be good to bake this into the helm chart itself as I'd like to do the deployment through argocd (as part of the cluster bootstrap) and post-render is a bit difficult to achieve in this way.

I guess the wider question is, what is the recommended way of installing Istio on EKS if users opt for choosing a different CNI than provided AWS VPC CNI.

As far as I know it is the only way... EKS uses their own plugin to run over the underlying VPC and therefore this is what both the control plane and the nodes would use.
If you remove the AWS CNI from the nodes, then the only way to expose it is to use hostNetwork=true as I understand it.

Yeah, it's also quite a popular strategy for a lot of organisations to use a more advanced and flexible CNI like Calico etc, so if the only way of getting Istio to work with it is by modifying the hostNetwork option, then I guess that's just because of the way EKS mandates it. It's still not ideal, and I can see why it's not recommended by Istio itself, but it is somewhat the only way of running a custom CNI on EKS.

I have the same problem and patched istiod deployment manually setting hostNetwork=true. Unfortunately, this didn't fix the problem. Any advice what else to try? I see two errors now, one in istiod log:

error	controllers	error handling /istio-validator-istio-system, retrying (retry count: 5089): webhook is not ready, retry	controller=validation
another in replacsets trying to inject the proxy:
replicaset-controller  Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

so have you opened the ports on the nodes? I dont think 443 is enabled by default depending on the kubernetes distro.

check this out - https://istio.io/latest/docs/setup/install/external-controlplane/#set-up-a-gateway-in-the-external-cluster you can put istiod behind an istio gateway so the mutated webhook can use this new endpoint url via istio gateway
Can you try this approach to see if it works @geoffo-dev ?

Note: i'm recommending this as I know this approach for other cloud providers when Istiod runs outside of cluster

The same problem with EKS + cilium.
This situation makes it difficult to operate istio in non-stock systems. Of course, I can patch the pilot's deployment, but this creates certain overhead costs for supporting such a solution.

I don't think we should do this. Running as hostNetwork is not safe and I don't support encouraging it. You can do this on your own if you want -- and are willing to deal with the repercussions. See https://github.com/istio/istio/issues/34021 for prior discussion and link for how to to do it.

@howardjohn, sorry but I really don't understand about what repercussions you are talking about. In EKS users decide which ports are allowed from EKS control plane to nodes using security groups. Since EKS control plane is NOT directly available for users except API, allowing another random ports from control plane to nodes are secure.
Sad reality is that If you have big EKS clusters using default VPC CNI is not an option. This CNI have many restrictions / issues and only option is use Cilium/Calico/Weave. Moreover hostNetwork mode approach is a widely adopted solution in most popular Kubernetes projects which is using webhooks in case of using these CNIs:

cert-manager - https://github.com/cert-manager/cert-manager/blob/main/deploy/charts/cert-manager/values.yaml#L477-L486
Karpenter - https://github.com/aws/karpenter/blob/main/charts/karpenter/values.yaml#L52-L53
kube-prometheus-stack https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L2323-L2326
AWS Load Balancer Controller - https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/helm/aws-load-balancer-controller/values.yaml#L261-L264
metics server - https://github.com/kubernetes-sigs/metrics-server/blob/master/charts/metrics-server/values.yaml#L69-L75

These rules as also adopted in very poplar Terraform module for EKS boostrap - https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/node_groups.tf#L137-L172
For sure, hostNetwork must set to false by default but users should have opportunity to override this depend on their environment. Because if the user touches this option he usually knows what he is doing.
I really hope that Istio maintancers reconsider this or let me know if we can just open a PR for this.
Thanks!

@howardjohn sure I can, as well as take responsibility and maintain own fork of chart to avoid limitations. But we are talking here about wide adopted approach for a very popular problem and I really don't understand the motivation why it doesn't want to be included in official helm chart.

hostNetwork is an operational and security risk. I don't know the motivation behind other projects adopting it, but to me personally I don't want to in any way encourage users to run in that setup.

If you must run on an effectively broken Kubernetes distribution, I would put the webhook behind a load balancer that is accessible to the control plane before resorting to hostNetwork. But again, you don't have to convince me as you can simply override the value :-)

hostNetwork is an operational and security risk. I don't know the motivation behind other projects adopting it, but to me personally I don't want to in any way encourage users to run in that setup.

While I understand the concerns associated with using the hostNetwork parameter, it's crucial to recognize that its application is driven by the specific needs of users. Your recommendation against this approach doesn't change the fact that teams are resorting to using hostNetwork in EKS + Cilium/Calico stacks to address issues arising from hooks in cloud Kubernetes clusters.
The key point here is that the absence of official support for hostNetwork doesn't change the fact that users persist in using it. Instead, it introduces a challenge where this workaround becomes a necessary "kludge" to the Istio Helm-chart, for teams employing cloud Kubernetes with non-standard CNI.
As a potential compromise, consider adding support for a configurable dictionary in the securityContext parameter.
This adjustment won't explicitly endorse hostNetwork use but could significantly simplify Istio deployment for users facing challenges in this issue.
Your consideration of this suggestion is appreciated.

@igor-nikiforov can you pls explore and let us know if you can expose mutating webhook server to istio ingress gw and use that approach instead in EKS?

@linsun, thank you for addressing our question.
It seems that, in this situation, a more cost-effective solution for us would be to fork the Istio Helm chart and introduce support for the hostNetwork parameter, enabling Istio to function correctly in EKS. While this would lead us to diverge from the official Helm chart for Istio, it aligns with a similar approach seen in other official Helm charts utilizing hooks (such as karpenter, cert-manager, prometheus, etc.).
The solution involving the addition of an extra component in the form of a web server (or Ingress), solely to facilitate Istio's deployment in EKS, appears redundant and more cumbersome to maintain.
Unfortunately, abstaining from adding this parameter to the official Istio Helm chart does not eliminate the use of hostNetwork; it merely complicates the support and operation of Istio in EKS with a non-standard CNI plugin. Those who use hostNetwork for hooks in EKS are likely to persist with its usage simply because it is the most straightforward solution and is supported in almost all Helm charts where hooks are present.

hello just FYI,  even setting host network: true did not help

@howardjohn @linsun
My intent is to push users towards using a proxy in front of Istiod rather than using hostNetwork.  I 100% believe in this as well, however, I'm currently don't see if feasible with fedora 34 distro, kubeadm, and cilium.
This documentation https://istio.io/latest/docs/setup/install/multicluster/primary-remote_multi-network/ Only works if I expose the primary istiod w/ host network.

hello just FYI, even setting host network: true did not help

Not sure where you are deploying it, but you might have to open the port on the nodes as well to allow the traffic through.

from another host run nmap -p (nodeport port#) host, gateway needs to be deployed to open those. (which i find weird)

I appreciate that you want to try and keep this solution as secure as possible - I am a little frustrated that we have had to resort to the use of hostNetwork for a number of applications. However we only have to do this because that we are unable to control the networks on the K8s hosts which is a result of using cloud provided K8s solutions - not istio's problem I understand!
@howardjohn as you mentioned there are many ways to do this and as you say we could use kustomize to do this... There is a bit of faff to do, but fully appreciate that you can. Whichever way we choose, I understand doing it this way would considered bad... but unless we can control the network between both the masters and workers (which is unlikely) it is really the only option. We could implement a proxy, but I am sure that would come with its own challenges.
I think what we recognise through is that many people need to use different CNIs with Cloud enabled K8s distros...  CertManager, External Secrets, OPA Gatekeeper, metrics server, nginx ingress, etc all offer the hostNetwork option.
I also appreciate there are 100s of other requests - but this one came with a PR which implemented it...

@howardjohn, ootb argocd has no ability to kustomize a helm chart.
Further, during kubecon my boss was assured that this contribution would be 'welcomed'.
I was doing a quick search before I reached out, but I was going to propose using hostPort which would enable the webhook to be exposed but the other ports to not be (to the best of my knowledge anyways).

there is a related request that is opened with AWS. Please upvote.

I found that my issue was from not allowing traffic through port 15017 and updating the cluster firewall as described here solved the problem.

Here's the code that I'm using to get this working in ArgoCD:
https://gist.github.com/KrisJohnstone/e263e5dfdf3a6ec29cbdf822992eaf01

Nice thank you @MartinKaburu and @KrisJohnstone for reporting back that updating firefox can fix this.
cc a few parties who may be interested: @nmnellis @ilrudie @danehans

Nice thank you @MartinKaburu and @KrisJohnstone for reporting back that updating firefox can fix this. are you still required to set hostNetwork to true or not for AWS cloud?
cc a few parties who may be interested: @nmnellis @ilrudie @danehans

Think you have misunderstood the problem or @MartinKaburu's comment has thrown you off. (I'm also assuming that firefox was meant to be firewall and was a brain fart).
If your security groups are too restrictive then traffic to the istio pod would be blocked and thus webhooks calls would fail. Given it's Martins first post in the thread and the context of his post I'm guessing hes using VPC CNI or a CNI that makes use of AWS ENI, they effectively share the same networking concepts. The advantage is that you do not need to use hostNetworking or hostPort functionality. There are a number of disadvantages with either.
Alternatively, if you choose to remove the default CNI (VPC CNI) and install another CNI that doesn't utilise AWS ENI then because Calico (insert CNI here) pods cant be run on the master nodes, changes need to be made to items such as webhooks to enable communication with control plane nodes.
The common pattern that's suggested is using hostNetwork: true there is three(?) main issues with this:

By enabling hostNetwork the pods networking is indistinguishable from that of the host.
Security issues around abstract sockets (this was fixed in containerd but there are other processes on the host that might still run them).
k8s network policies don't apply to resources running at the host level.

This is why I was suggesting using hostPort instead of hostNetwork. In doing so the pods aren't run at a host level so thereby negating 1, 2 and partially 3. Instead traffic is fowarded traffic from the host to the container.
AFAIK this represents the best middle ground.

Thanks @KrisJohnstone, sorry about typing fireworks instead of firewall rules, it was on Sunday night. :-(
Thank you for clarifying the issue is not with the default EKS CNI, but with different CNI and the hostPort vs hostNetwork. I agree hostPort seems a very nice middle ground as long as we can guarantee that port (15017 here) is not used by anything else on the hosts where istiod is placed.

cc @stevenctl as you are looking into chaining CNIs and EKS

Here's the code that I'm using to get this working in ArgoCD: https://gist.github.com/KrisJohnstone/e263e5dfdf3a6ec29cbdf822992eaf01

@KrisJohnstone The link seems to be broken. Please share your solution.

At the end we ended up with the exact same change proposed by @geoffo-dev
We were forced to substitute aws cni with calico due to the fact that we cannot increase the subnets cidrs. (which happens to be a common issue when multiple teams are involved)
We discarded  the hostPort hypothesis since the port should be opened only on the node where the pod is scheduled and this is suitable only when you have daemon sets talking with the underlying host.
We also tried to put an nginx ingress controller in front of the the istio control plane however the ingress hostname should be public and this is not nice either.
If the hostNetwork is not going to be supported I think it should be defined how to workaround this issue (as @ChrisJBurns already said earlier)


How to adjust points width

Hi,
How can I adjust the points width?
I have tried the following code with no luck:
  .strengthBar[_ngcontent-c2] .point[_ngcontent-c2] {   	width: 40px !important;   }

I needed to set ViewEncapsulation.None


Pdf reader

experimental prototype to read and discard header, footer from PDF.
Other issues need to be open, in order to divede the text in chapters


Highlight collaborators

Fixes #1.

One litte strange behavior, though: https://github.com/fhemberger/contribution-report/blob/feature/collaborators/index.js#L127
ISO date of '2016-01-01T00:00:00.000Z' turns into '"2016-01-01T00:00:00.000Z"' on subsequent calls of getPage. I don't know why this is happening, maybe you have an idea.

Dunno about the date thing ¯\_(ツ)_/¯
LGTM tho. Mergin.


Develop

DRY via Lib class
cleanup and minor adjusments/fixes
CS
100% BC - no test has been modified.

Feel free to use it.
Otherwise you may disregard.

@dereuromark Take a look at this https://github.com/burzum/cakephp-imagine-plugin/commit/59da54f7f095a656cbe52284bd11ac130c24242d
I've completley refactored this into it's own class that can be used with a fluid interface now. :)
$processor = new ImageProcessor();
$processor->open('foo.jpg')
    ->flip([/*...*/])
    ->rotate([/*...*/])
    ->thumbnail([/*...*/])
    ->save('foo-changed.jpg');

Awesome shit :)

This has been merged into the 3.0 branch.


Failed to initialize NVML: GPU access blocked by the operating system

Windows Version
Windows 11 PRO Version 10.0.22000.2176
WSL Version
Versione WSL: <IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
6.1.38-microsoft-standard
Distro Version
Ubuntu 22.10
Other Software
No response
Repro Steps
Hello to everyone.
I tried to check my nvidia driver version within ubuntu 22.04 installed on the WSL2 subsystem installed on Windows 11,however, i got the error "Failed to initialize NVML: GPU access blocked by the operating system". Hope to get support from you on this issue.
On Windows 11 host os it works :

# nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 536.40                 Driver Version: 536.40       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1060 3GB  WDDM  | 00000000:01:00.0  On |                  N/A |
| 57%   41C    P2              25W / 120W |    710MiB /  3072MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 Ti   WDDM  | 00000000:02:00.0 Off |                  N/A |
| 31%   40C    P8              21W / 250W |     74MiB / 11264MiB |      0%

on Ubuntu 22.04 guest os it doesn't :
# nvidia-smi

Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

BUT :
# nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

# python3 --version
Python 3.10.7

Within ubuntu 22.04 I have installed cuda-12.2 using the instructions below :
# wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin

# sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600

# wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb

# sudo dpkg -i cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb

# sudo cp /var/cuda-repo-wsl-ubuntu-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/

# sudo apt-get update

# sudo apt-get -y install nvidia-cuda-toolkit

My PC has 2 graphic cards :

NVIDIA GeForce GTX 1060 3GB
NVIDIA GeForce RTX 2080 Ti

Below you can see the nvidia driver version installed on WIndows 11 for both the GPUs :

I have also added the line below to /home/marietto/.bashrc and to /root/.bashrc :
export PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
and I did :
# source /home/marietto/.bashrc
# source /root/.bashrc

Expected Behavior
# nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 536.40                 Driver Version: 536.40       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1060 3GB  WDDM  | 00000000:01:00.0  On |                  N/A |
| 57%   41C    P2              25W / 120W |    710MiB /  3072MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080 Ti   WDDM  | 00000000:02:00.0 Off |                  N/A |
| 31%   40C    P8              21W / 250W |     74MiB / 11264MiB |      0%

Actual Behavior
Failed to initialize NVML: GPU access blocked by the operating system
Diagnostic Logs
No response

Fixed installing the "<IP_ADDRESS>-microsoft-standard-WSL2" kernel.

Fixed installing the "<IP_ADDRESS>-microsoft-standard-WSL2" kernel.

How to install?

What about to use qemu + hyperV to virtualize Linux on top of Windows 11,instead of WSL2 ? I did that with FreeBSD as guest,but it can be done also with Linux :
https://forums.freebsd.org/threads/how-to-virtualize-freebsd-14-release-as-a-vm-on-top-of-windows-11-using-qemu-hyperv.93158/#post-652021


Allow adding of rolling Tool Kits.

It would make sense to be able to roll for tools instead of skills. :)

I second this. That was fast I was about to write about it.

I'd like to see this too, or have a tool/tool kit required (like an ingredient) but not destroyed on failure.

I will add this next.
(@iotech-fvtt)
(you broke your tools) will not come with this. That is to specific for mainstream so this seems to be more a feautre for advanced users when i invent the crafting macro you then should be able to manipulate the result in a way that produces what you want.

done 0.5.x


[Maybe] Improve how the Settings plugin does the directory listing

While writing test cases I discovered that the settings plugin will cause will cause configuration cache misses.
If you apply the settings plugin and compile the same module 2 times in a row you will get:
Calculating task graph as configuration cache cannot be reused because directory '.' has changed.
So because the Settings plugin just operates on Files instead of using valueSources doing about anything can cause a configuration cache miss, especially deleting or creating the build directory

so the problem does not lie that much in the settings plugin generally but more in the "directory" functionality. Since directory list list all files including the build directory this functionality can be a bit problematic

In my practical experience this is not problematic. The main use of the configuration cache is running the same build locally several times in a row. The content of the '.' directory usually does not change while doing so. I use this or similar approaches to automatically include projects in several projects and it works well with the configuration cache enabled.
I don't know how this could be done in a different way. But if there is, please share a concrete suggestions.
(Otherwise, I would close this issue.)

Thank you for the context. In that case, I am happy to accept a contribution that improve this through a value source that filters out reasonable entries in the listing. Maybe this should be configurable, but I think having a reasonable default exclusion that always applies would also be a good first step. I think a reasonable default would be:

everything that is not a folder
the build folder
folders starting with . (hidden folders)


Fix incorrect delete in MXExecutorReshape exception handling (#13376)

Cherrypicking #13376 to 1.4.x branch.

Merged
@mxnet-label-bot remove [pr-awaiting-review]


Ignore CACHE_MISSes in GradleceptionWithJavaMaxLts

We observed some consistent CACHE_MISS alerts from GradleceptionWithJavaMaxLts builds, which should be ignored.

The merge queue build has started. Click here to see all failures if any.


Add some coverage tests and set no cover for impossible statements

Codecov Report

Merging #25 into master will decrease coverage by 0.80%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #25      +/-   ##
==========================================
- Coverage   99.26%   98.46%   -0.81%
==========================================
  Files          17       17
  Lines        1500     1495       -5
==========================================
- Hits         1489     1472      -17
- Misses         11       23      +12

Impacted Files
Coverage Δ

src/viztracer/main.py
98.40% <ø> (+0.74%)
:arrow_up:

src/viztracer/prog_snapshot.py
99.43% <ø> (-0.01%)
:arrow_down:

src/viztracer/functree.py
100.00% <100.00%> (+2.19%)
:arrow_up:

src/viztracer/viztracer.py
94.49% <0.00%> (-5.51%)
:arrow_down:

src/viztracer/report_builder.py
95.31% <0.00%> (-4.69%)
:arrow_down:

src/viztracer/simulator.py
96.77% <0.00%> (-3.23%)
:arrow_down:

src/viztracer/flamegraph.py
97.59% <0.00%> (-2.41%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d5e0aee...838c5e4. Read the comment docs.


Yank all broken versions from crates.io

https://crates.io/crates/libz-sys contains quite a few versions that do not compile with recent Rust versions, they should be yanked.

0.0.1:

error: expected identifier, found `"pkg-config"`
 --> /home/klausi/.cargo/registry/src/github.com-1ecc6299db9ec823/libz-sys-0.0.1/src/build.rs:1:14
  |
1 | extern crate "pkg-config" as pkg_config;
  |              ^^^^^^^^^^^^ expected identifier

error: aborting due to previous error

error: Could not compile `libz-sys`.

0.1.0

0.1.1 does not compile because pkg-config v0.3.0 does not compile, so we probably also need to yank broken versions there.

Thanks for the report! I commented over on the Cargo PR though in that I don't think this is quite the right solution, so I'm going to close this.


RFC: Buffered Read Until

https://github.com/ponylang/rfcs/blob/master/text/0013-buffered-read-until.md

@lisael you interested in tackling this one?

Of course! I take it.

@lisael I've invited you to join the ponylang organization as a contributor without write access. That will allow committers to list you as an assignee for any issues you decide to take on (such as this one). Once you accept the invite, leave a note here and I'll add you as the assignee.
Thanks for the RFC and thanks for implementing it.

Invitation accepted. Thanks!


media download stalls after a while

Can you post your config?

I am about to implement #2 which might help

Maybe we also need #60?

Can you post your config?

Build a data_generator as described in the README.md using species_id.json as "speciesKey" and providers.json as "datasetKey"
launch a gbif_dl.io.download() on the data_generator with low nb_workers and tcp_connections (let's say 2 or 4)
it will stall after a few thousand images

providers.json.gz
species_id.json.gz

Okay, thanks... will check. I guess the first thing we would need to add is to provide a way to write out proper log files so that we can check for API or download errors. Hold on an let me try out some stuff

import gbif_dl
data_generator = gbif_dl.dwca.generate_urls(
     "10.15468/dl.bx6xeq",
     dwca_root_path="dwcas",
     label="speciesKey"
)
gbif_dl.io.download(
    data_generator,
    root="./imgs",
    random_subsets={'train': 0.95, 'val': 0.05},
    nb_workers=100,
    retries=1,
    loglevel="INFO"
)

@jclombar do you have observed this also from a smaller query (that I can test again)?

@jclombar I observed a simular issue as you did when interrupting the internet connection (thus break the request session). Maybe that's the issue and I will try to find a way to resume the session now....

Thanks. Tracking it now

Hi @Christian-lyc I think we need to update the docs....
gbif_dl/stores/dl_async.py
Here is the download function now

Hi, thank you so much for your help, but there's another error occurs. I only download 1000 images, but it shows no space, which shouldn't be like that.
download(data_generator,root="/export/data/yliu/dataset",random_subsets={'train': 0.95, 'val': 0.05},nb_workers=20, retries=1,loglevel="INFO")
                              Traceback (most recent call last):

File "", line 1, in
File "/export/home/yliu/.local/lib/python3.8/site-packages/gbif_dl/stores/dl_async.py", line 319, in download/s]
return run_async(
File "/export/home/yliu/.local/lib/python3.8/site-packages/gbif_dl/utils.py", line 63, in run_async
return runners.run(func(*args, **kwargs))
File "/export/home/yliu/.local/lib/python3.8/site-packages/gbif_dl/runners.py", line 104, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/export/home/yliu/.local/lib/python3.8/site-packages/gbif_dl/stores/dl_async.py", line 209, in _download_from_asyncgen
async for batch in chnk:
File "/export/home/yliu/.local/lib/python3.8/site-packages/aiostream/stream/transform.py", line 87, in chunks
async for first in streamer:
File "/export/home/yliu/.local/lib/python3.8/site-packages/aiostream/stream/create.py", line 34, in from_iterable
for item in it:
File "/export/home/yliu/.local/lib/python3.8/site-packages/gbif_dl/generators/dwca.py", line 47, in dwca_generator
with DwCAReader(dwca_path) as dwca:
File "/export/home/yliu/.local/lib/python3.8/site-packages/dwca/read.py", line 97, in init
self._directory_to_clean, self._working_directory_path = self._extract()
File "/export/home/yliu/.local/lib/python3.8/site-packages/dwca/read.py", line 448, in _extract
extracted_dir = self._unzip_or_untar()
File "/export/home/yliu/.local/lib/python3.8/site-packages/dwca/read.py", line 427, in _unzip_or_untar
tmp_dir = mkdtemp()
File "/usr/lib/python3.8/tempfile.py", line 497, in mkdtemp
_os.mkdir(file, 0o700)
OSError: [Errno 28] No space left on device: '/tmp/tmpyk2gmouo'


PR for Nth day of the Week on Mth Occurrence

This PR adds the feature of executing a job on nth Day of the Week. Now, the DoW spec can be of the form, N#M where N is the Day of the Week, Sunday(0) to Saturday(6), and M is the occurrence number, 1 to 4 or L for Last week.
For example,

if the current day is Mon Jun 1 01:00 2020 and the cron spec is 1 1 * 6 4#3 then the next execution date will be Mon Jun 18 01:01 2020.
if the current day is Mon Jun 1 01:00 2020 and the cron spec is 1 1 * 6 4#L then the next execution date will be Mon Jun 25 01:01 2020.

Hi @robfig,
I've not updated the doc and README. I'll do it once the Code Review is complete.


Problems rendering embedded videos

Hello,
I've been having difficulties capturing screenshots of sites like Twitter. When I try, I get screenshots that have boxes that say "We cannot play the video in this browser. Please try a different web browser." - see attached image. I've tried playing with enabling plugins, html5, and changing the user agent, to no success.
Additionally, when encountering embedded videos elsewhere, such as NYT, a similar problem arises where there's an empty box where there should be an embedded video.
Any help is appreciated

@alleycat08 Is it possible that you meant to create an issue for Splash, and not for scrapy-splash (which simply offers integration between Splash and Scrapy)? Or does this happen only when using scrapy-splash, and not when using Splash directly?
Also, have you checked Splash’s FAQ section about website rendering issues?


Bugfix/4559

Related Issues

Related https://github.com/DivanteLtd/vue-storefront/issues/4559
Short Description and Why It's Useful

Screenshots of Visual Changes before/after (If There Are Any)

IMPORTANT NOTICE - Remember to update CHANGELOG.md with description of your change
Contribution and Currently Important Rules Acceptance

[ ] I read and followed contribution rules

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


ExtractionResult is not flexible enough for some extractors

Some extractors are non-local: for different root eclasses, they extract different sub-terms for the same eclass.
However, the current extraction gym does not allow picking two different nodes for the same eclass.
As a result, the global_greedy_dag extractor sometimes extracts cycles because it chooses conflicting nodes when extracting different roots.

I have a simple solution: let's just allow a single root per benchmark.
If you want to benchmark something with multiple roots, just create multiple benchmarks

Perhaps there's a more general problem with cycles and DAG extractors- if you allow them to extract a particular term, the problem goes away. However, if they must pick a single term per eclass cycles can occur


Feature/add language switcher issue#49

General
closes #49

Add possibility to change the app language

Type

ℹ  What types of changes does your code introduce?

👉 Put an x in the boxes that apply

[ ] Fix
[x] Feature

Describe the changes here (summarized)

Add new app language switcher component.
Add new label to i18n files.

@Jadhielv Check this please! :smiley:


Invalid nuget version number listed for Microsoft.DotNet.BuildTools

This line of code:
https://github.com/Microsoft/visualfsharp/blob/4598abfdd07eab392f76e7670af4d35305415a12/packages.config#L4
references version '1.0.25-prerelease-00085' which does not exist.  This causes a build of the proto-compiler to fail.  Commenting out this line allows the build of the proto-compiler to succeed.

Here is a link to the Nuget Package for Microsoft.DotNet.BuildTools: https://www.nuget.org/packages/Microsoft.DotNet.BuildTools/


add details into internal validation

Initial purpose to change - add details into YamlConfig validation.
And in addition, more details were added into the project internal error raise.
Please check DetektGenerateConfigTask, because semantic was changed: from now actual error is provided to gradle. It means, that --stacktrace gradle option is able to show detekt's stack.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@2f68430). Click here to learn what that means.
The diff coverage is 79.77%.

@@            Coverage Diff            @@
##             master    #2347   +/-   ##
=========================================
  Coverage          ?   82.82%
  Complexity        ?     2160
=========================================
  Files             ?      352
  Lines             ?     6077
  Branches          ?     1112
=========================================
  Hits              ?     5033
  Misses            ?      472
  Partials          ?      572

Impacted Files
Coverage Δ
Complexity Δ

...ain/kotlin/io/gitlab/arturbosch/detekt/cli/Main.kt
36% <ø> (ø)
6 <0> (?)

...etekt/formatting/wrappers/ParameterListWrapping.kt
100% <ø> (ø)
4 <0> (?)

...ch/detekt/formatting/wrappers/MaximumLineLength.kt
88.88% <ø> (ø)
5 <0> (?)

...tlab/arturbosch/detekt/cli/config/InvalidConfig.kt
100% <ø> (ø)
1 <0> (?)

...turbosch/detekt/formatting/wrappers/Indentation.kt
100% <ø> (ø)
4 <0> (?)

.../io/gitlab/arturbosch/detekt/cli/runners/Runner.kt
92.3% <ø> (ø)
12 <0> (?)

...itlab/arturbosch/detekt/api/internal/Validation.kt
33.33% <0%> (ø)
0 <0> (?)

...b/arturbosch/detekt/api/internal/FailFastConfig.kt
36.36% <0%> (ø)
5 <0> (?)

...osch/detekt/api/internal/KotlinEnvironmentUtils.kt
84.9% <0%> (ø)
0 <0> (?)

...ch/detekt/api/internal/ValidatableConfiguration.kt
100% <100%> (ø)
0 <0> (?)

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f68430...e40dd8c. Read the comment docs.

@BraisGabin, fixed. Apologise for that.


Java to Azure migration doc should include Spring Boot Apps migration to Azure Container Apps

Java to Azure migration doc should include Spring Boot Apps migration to Azure Container Apps
See also https://github.com/MicrosoftDocs/azure-dev-docs/issues/825

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3830ec12-c308-1d59-07c8-be295d8db7a3
Version Independent ID: b4439660-4af0-12eb-408b-ec9a181ec505
Content: Migrate Spring Boot applications to run on Azure Kubernetes Service
Content Source: articles/java/migration/migrate-spring-boot-to-azure-kubernetes-service.md
Service: azure-java
GitHub Login: @KarlErickson
Microsoft Alias: manriem

The migration overview now includes coverage of Azure Container Apps. See https://learn.microsoft.com/en-us/azure/developer/java/migration/migration-overview#migration-guidance.


add MacOS support

see #4

[x] x86_64
[ ] M1 ARM

For 1.0.0 and 2.0.0, I've been using Ghidrathon on mac with no problems running a bunch of my own scripts. All it takes is a 10-line patch to let Ghidrathon find jep.cpython-310-darwin.so (for other python versions 310 may need to change) instead of libjep.so. I'm not sure whether #4 is larger because it handles lots of special cases? But if desired, maybe I can convert that patch to a basic PR and #4 can continue its extended work and be merged later.

@Cnly this sounds great and much appreciated! Yes please open a PR w/ the aforementioned changes independent of the changes being worked in #4. Ideally we'd handle all Python versions supported by Ghidrathon. Would you be able to test another Python version or two to see how the filename is affected? Unfortunately we don't have a Mac environment on our end for testing.

I'm on M1 ARM. I don't think the x86 solution works for me but here's what i've been doing:

Create a virtual environment with conda. activate it
install python 3.10 with pip.
pip install jep
look in the environment's site-packages for jep's library

it should be libjep.jnilib

overwrite the configure_jep_native_binaries.py to use libjep.jnilib and copy into a folder named mac_arm_64 instead of linux_x86_64
build with gradle per usual instructions (with venv activated)
after the build is done, extract the resulting extension .zip file to the Extensions/Ghidra/Extensions folder in the Ghidra home directory
go into the extracted extension's folder. find the mac_arm_64 folder and rename libjep.jnilib to jep.dll
follow rest of install instructions

haven't had time to automate all of this but it should be straightforward if anyone needs a quick and dirty method.

I'm on M1 ARM on Ventura, I just did the following and Ghidrathon worked out of the box for me:
diff --git a/build.gradle b/build.gradle
index 59b952f..4cc1ae4 100644
--- a/build.gradle
+++ b/build.gradle
@@ -29,7 +29,7 @@ if (project.hasProperty("PYTHON_BIN")) {
 	pythonBin = project.getProperty("PYTHON_BIN")
 }
 else {
-	pythonBin = "python"
+	pythonBin = "python3"
 }

 // we need to install Jep; this requires C++ build tools on Windows (see README); we need to define
@@ -73,4 +73,4 @@ if (ghidraInstallDir) {
 else {
     throw new GradleException("GHIDRA_INSTALL_DIR is not defined!")
 }
-//----------------------END "DO NOT MODIFY" SECTION-------------------------------
\ No newline at end of file
+//----------------------END "DO NOT MODIFY" SECTION-------------------------------

Then I built with Gradle and Ghidra 10.3 and Ghidrathon worked out of the box!

I'm on M1 ARM on Ventura, I just did the following and Ghidrathon worked out of the box for me:
diff --git a/build.gradle b/build.gradle
index 59b952f..4cc1ae4 100644
--- a/build.gradle
+++ b/build.gradle
@@ -29,7 +29,7 @@ if (project.hasProperty("PYTHON_BIN")) {
 	pythonBin = project.getProperty("PYTHON_BIN")
 }
 else {
-	pythonBin = "python"
+	pythonBin = "python3"
 }

 // we need to install Jep; this requires C++ build tools on Windows (see README); we need to define
@@ -73,4 +73,4 @@ if (ghidraInstallDir) {
 else {
     throw new GradleException("GHIDRA_INSTALL_DIR is not defined!")
 }
-//----------------------END "DO NOT MODIFY" SECTION-------------------------------
\ No newline at end of file
+//----------------------END "DO NOT MODIFY" SECTION-------------------------------

Then I built with Gradle and Ghidra 10.3 and Ghidrathon worked out of the box!

Did this work on an m1 mac though? I am on an m1 mac mini and discovered this change on my own, which allowed it to build successfully. I can get it installed/added into ghidra, but when launching it, it pukes complaining about architecture types.
/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so: dlopen(/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so, 0x0001): tried: '/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (no such file), '/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
java.lang.UnsatisfiedLinkError: /Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so: dlopen(/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so, 0x0001): tried: '/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (no such file), '/Users/austinc/.ghidra/.ghidra_10.3_PUBLIC/Extensions/Ghidrathon/os/mac_arm_64/libjep.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
	at java.base/java.lang.Runtime.load0(Runtime.java:755)
	at java.base/java.lang.System.load(System.java:1953)
	at jep.MainInterpreter.initialize(MainInterpreter.java:126)
	at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
	at jep.Jep.<init>(Jep.java:133)
	at jep.Jep.<init>(Jep.java:128)
	at jep.SubInterpreter.<init>(SubInterpreter.java:63)
	at ghidrathon.interpreter.GhidrathonInterpreter.<init>(GhidrathonInterpreter.java:124)
	at ghidrathon.interpreter.GhidrathonInterpreter.get(GhidrathonInterpreter.java:264)
	at ghidrathon.GhidrathonConsoleInputThread.run(GhidrathonConsoleInputThread.java:62)

---------------------------------------------------
Build Date: 2023-May-10 1508 EDT
Ghidra Version: 10.3
Java Home: /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home
JVM Version: Eclipse Adoptium 17.0.7
OS: Mac OS X 13.0.1 aarch64

we need to be careful in this thread about which architecture is being used. macOS has (usually) nice x86 compatibility via Rosetta, so it's easy to mix up whether we're loading java/python/ghidra in x86 or arm mode.
for example, in @austinc3030's traceback above, we can see:
.../os/mac_arm_64/libjep.so: dlopen(.../os/mac_arm_64/libjep.so, 0x0001):
  tried: '.../os/mac_arm_64/libjep.so'
  (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')),

which suggests that the jep.dll is built for x86 (so maybe python3 x86 was invoked to build jep.dll) but ghidra and java are running on arm64.
unfortunately, im still pretty new to macOS so i dont have the specific commands needed to confirm the architectures being used (but i suspect file /path/to/executable will show it).
i'll try to reproduce as well and report back.

I found that rpath is not set in libjep.so (at least my environment), so after I fix to set rpath, it worked on my environment (M2 MBA 2022, Ventura 13.5).
% otool -L $HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so
$HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so (architecture x86_64):
        @rpath/Python3.framework/Versions/3.9/Python3 (compatibility version 3.9.0, current version 3.9.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
$HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so (architecture arm64):
        @rpath/Python3.framework/Versions/3.9/Python3 (compatibility version 3.9.0, current version 3.9.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
% find / -name 'Python3'
...
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Python3
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Python3
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Python3
...
% install_name_tool -add_rpath /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Library/Frameworks/ $HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so
% otool -L $HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so
$HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so (architecture x86_64):
        @rpath/Python3.framework/Versions/3.9/Python3 (compatibility version 3.9.0, current version 3.9.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
$HOME/.ghidra/.ghidra_10.3.2_PUBLIC/Extensions/Ghidrathon-2.2.0/os/mac_x86_64/libjep.so (architecture arm64):
        @rpath/Python3.framework/Versions/3.9/Python3 (compatibility version 3.9.0, current version 3.9.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

I'm now on Apple Silicon and the latest commit 7489831 already works for me out of the box. My gradle command is
gradle -PGHIDRA_INSTALL_DIR=/opt/homebrew/Caskroom/ghidra/10.3.3-20230829/ghidra_10.3.3_PUBLIC -PPYTHON_BIN=/opt/homebrew/bin/python3

and it seems that libjep.so is also correctly built:
$ file os/mac_x86_64/libjep.so
os/mac_x86_64/libjep.so: Mach-O 64-bit bundle arm64

Oh..., on my environment, your approach did not work.
% gradle -PGHIDRA_INSTALL_DIR="${GHIDRA_PATH}" -PPYTHON_BIN="$(which python3)"
% file $HOME/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so
/Users/XXX/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64]
/Users/XXX/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so (for architecture x86_64):   Mach-O 64-bit bundle x86_64
/Users/XXX/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so (for architecture arm64):    Mach-O 64-bit bundle arm64

Can't load library: /Users/XXX/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so
java.lang.UnsatisfiedLinkError: Can't load library: /Users/XXX/.ghidra/.ghidra_10.3.3_PUBLIC/Extensions/Ghidrathon-3.0.1/os/mac_x86_64/libjep.so
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
	at java.base/java.lang.Runtime.load0(Runtime.java:755)
	at java.base/java.lang.System.load(System.java:1953)
	at jep.MainInterpreter.initialize(MainInterpreter.java:126)
	at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
	at jep.Jep.<init>(Jep.java:133)
	at jep.SharedInterpreter.<init>(SharedInterpreter.java:60)
	at ghidrathon.interpreter.GhidrathonInterpreter.<init>(GhidrathonInterpreter.java:72)
	at ghidrathon.interpreter.GhidrathonInterpreter.get(GhidrathonInterpreter.java:232)
	at ghidrathon.GhidrathonConsoleInputThread.run(GhidrathonConsoleInputThread.java:66)

---------------------------------------------------
Build Date: 2023-Aug-29 1442 EDT
Ghidra Version: 10.3.3
Java Home: /Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home
JVM Version: Oracle Corporation 17.0.8
OS: Mac OS X 13.5.2 aarch64

After applying my rpath patching, Ghidrathon 3.0.1 works expectedly.

This should be addressed in https://github.com/mandiant/Ghidrathon/commit/32f3969e09a5d29197cabbc28d93dc9c572d7e45 - please reopen if you continue to have issues after this is released.


Warning: bind(): React component methods may only be bound to the component instance

I'm using<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>versions.
When adding a ZoomControl which has an onControlClick callback, React gives an error:

The error is coming from this line: https://github.com/alex3165/react-mapbox-gl/blob/1bd531359d89f3c5d4eb97285c380b68e6761ebf/src/zoom-control.js#L103 where the component calls .bind for the parent component's (MyMap) method.
MyMap.js:
import React from 'react';
import ReactMapboxGl, { Layer, Feature, Marker, ZoomControl } from 'react-mapbox-gl';

const MyMap = React.createClass({
  getInitialState() {
    return {
      mapCenter: [0, 0],
      zoom: 2,
      style: 'mapbox://styles/mapbox/light-v9',
    };
  },

  render() {
    const { state, props } = this;

    return (
      <div className="MyMap" style={{width: 500, height: 700}}>
          <ReactMapboxGl
            center={state.mapCenter}
            zoom={[state.zoom]}
            style={state.style}
            accessToken="pk.eyJ1IjoiZmFicmljOCIsImEiOiJjaWc5aTV1ZzUwMDJwdzJrb2w0dXRmc2d0In0.p6GGlfyV-WksaDV_KdN27A"
          >
            <ZoomControl
              zoomDiff={1}
              onControlClick={this._onControlClick}
            />
          </ReactMapboxGl>
      </div>
    );
  },

  _onControlClick(map, zoomDiff) {
    const zoom = map.getZoom() + zoomDiff;
    this.setState(() => ({ zoom: zoom }));
  }
});

export default MyMap;

Can you try to modify your MyMap.js to :
            <ZoomControl
              zoomDiff={1}
              onControlClick={this._onControlClick.bind(this)}
            />

That doesn't help. It just adds a new warning:
Warning: bind(): You are binding a component method to the component. React does this for you automatically in a high-performance way, so you can safely remove this call. See AlvarMap

The warning comes because ZoomControl component is messing with MyMap component's method.
zoom-control.js, row 103:
onClick={onControlClick.bind(this, map, zoomDiff)}>

There the onControlClick method is MyMap's method: MyMap._onControlClick. React warns because you shouldn't be rebinding the this to a new component.

I have published a patch (0.19.1) where I have replaced the bind with an arrow function, let me know if it fix your issue.

I tested with version 0.19.1 but the problem still persists.

Can you send me a link to your project ? or a link to a jsfiddle ?

The affected line was changed in a previous patch and we don't have an example available, to reproduce the issue, thus I need to close this for now 😄

Sorry for taking so long. I don't know how I tested it before but I can now confirm that bump from 0.18.0 -> 0.19.1 fixed this issue.
A reproducible example of the bug: https://github.com/kimmobrunfeldt/react-mapbox-gl-bug/blob/master/src/MyMap.js


Closing this PR with minimal review.


Unexpected return from gamma_q function

When the magnitude of the second argument to the Jmat.gamma_q
function is large enough, the return value becomes complex while it
appears that the correct asymptotic behavior is to approach zero.
ex:
Jmat.gamma_q(1,15) returns 3.058805404521081e-7
Jmat.gamma_q(1,16) returns
1.1253517473441832e-7+1.2246467991473532e-16i
Jmat.gamma_q(10, 87) returns 1.84297022087776e-14
Jmat.gamma_q(10, 88) returns 7.193232413504669e-43i

Should be fixed by commit ff8a84912c8108f2a8fad7ba2f214895825da9dc
Thanks for reporting this!


Schema-ddl Java interop

I'm trying to call Schema-ddl from Java, but I am running into an issue with the FlatSchema case class. Namely, it doesn't appear to be accessible from java. I wondered if this was merely because it lives inside of a package object rather than a plain old package. Do you think it's remotely possible that changing com.snowplowanalytics/package.scala from
package com.snowplotanalytics
//...
package object schemaddl {
//...
}

to
package com.snowplowanalytics.schemaddl
//...

might make those case classes available in Java without breaking anything? Is there any other way to use RedshiftDdlGenerator (which relies on being able to access FlatSchema) from Java?
Thanks!

Hello @nihakue. I'll investigate this and give you answer tomorrow.

@nihakue yes, you're right problem is that classes defined in package object cannot be accessed from Java. I'm scheduling moving these case classes to separate package for next release.

@chuwy Thanks, that's good to know. When is your release goal for v0.3.0?
In the meantime, do you have any advice for using RedshiftDdlGenerator from Java? Otherwise I will try to implement a patch locally.

I cannot promise exact dates, but I have a feeling that it will be released before end of January.
As a whole interop thing, Java support never was our target and we use schema-ddl just from Scala code, primarily in Schema Guru. I decided to move these classes just because Java interop is nice thing to have if that could be implemented easely, but as I said we never used it from Java and thus I have no advices, just suppose everything should work smoothly.
Can I ask (just out of curiosity), in what project are planning to use schema-ddl?

@chuwy I built local versions of schema-ddl and schema-guru with that change. All your tests pass, and schema/redshift ddl generation both seem to work still. The only effect seems to be the improved Java interop, but I haven't done any extensive testing of my own. I will let you decide how best to proceed.
Unfortunately the project isn't open source, and I can't say too much about it. But we're using schema guru to generate a schema/redshift ddl for a dataload into Redshift :)

Cool use case @nihakue !

:) Sorry!


Multiple countdown clocks in one row share the same deadline

When I try to add 3 countdown clocks in one dashboard row with different deadlines
Then the same deadline is set for all three clocks

Bug can be expanded to multiple countdowns on a panel.  Calling duplicate breaks it up as does saving the panel at which point different end times can be set on each.

Thanks @bmagistro for that extra info that helped me recreate the problem.

Published a new version 0.0.9 with a fix for this.


Site not working

I only see a white screen when trying to view wttr

curl  doesn't work too.
> GET /San+Francisco HTTP/1.1
> Host: v2.wttr.in
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 28 Dec 2023 18:20:31 GMT
< Content-Length: 0
<
* Connection #0 to host v2.wttr.in left intact

It was working just fine yesterday but today it's not working.

curl wttr.in is also not working, it just doesn't do nothing when i type it in the terminal

I think it is back up again!

Now it seems the queries to the weather servies is out of compacity...

down again

Back online again


HOTFIX: Init stream-stream left/outer join emit interval correctly

Follow up to KAFKA-10847 (https://github.com/apache/kafka/pull/10917).
The above fix intended to reduce the emit frequency to save the creation cost of RocksDB iterators. However, we incorrectly initialized the "timer" with timestamp zero, and thus, the timer was always in the past and we did try to emit left/outer join result too often and got a throughput of only 500 record/sec.
This PR fixes the initialization of the emit interval timer. After re-running the benchmark, we determined that a default emit interval of 1000ms provides the best performance of 12K rec/sec, so this PR also changes the emit frequency to 1000ms.

@kkonstantine I think we should try to get this into 3.0 release. Thoughts?
Call for review @guozhangwang @spena

Merged to trunk and cherry-picked to 3.0 branch.


js tests weren't being run

Codecov Report

Merging #34 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #34   +/-   ##
=======================================
  Coverage   26.58%   26.58%
=======================================
  Files           6        6
  Lines         395      395
=======================================
  Hits          105      105
  Misses        290      290

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 993657f...a7bcfcf. Read the comment docs.

Should we also add coverage reporting for the TS to codecov? It posts a huge comment for every PR, so we might as well have that show something useful.

@vidartf sure, not sure if codecov will be happy with mixed language though.

@timkpaine I have that set up on a few projects (e.g. here https://codecov.io/gh/vidartf/jupyter-vtk-datawidgets). Works nicely.


uploading images to memory

When running the multi-stream scenario the number of queries is 270336, but the number of
images in the imagenet data set is 50000 so how do the upload to memory should work

do i need to upload only 50000 images and loadgen will generate query index with wraparound?
do 2. i need to load 270336 images with replication?

in the mlperf.conf file the offline scenario don't include min query count how many images should be loaded (assuming i can process 10000 images per second using resnet50?

found the right configuration


Support for version 2.y.z of mysql2

Currently @google-cloud/trace-agent only supports 1.y.z mysql2.
https://github.com/googleapis/cloud-trace-nodejs/blob/6f118e9b7fe2194d938bfb3a6e1a8aa763a77b3b/src/plugins/plugin-mysql2.ts#L17
However, the current major version of mysql2 is 2.y.z.
https://www.npmjs.com/package/mysql2
mysql2 includes TypeScript type definitions and other more useful features. Developers will want to use the newer version of mysql2.

@opentelemetry/instrumentation-mysql2 already supports 2.x. Could you use that instead?


Arduino IDE compiling error

Arduino IDE v. 1.8.13
ESP32  v. 1.0.6 selected ESP32 Dev Module
weird issue
C:\Users\USER\OneDrive\Desktop\portal_calendar\portal_calendar.ino: In function 'bool startWifi()':
portal_calendar:94:40: error: no matching function for call to 'WiFiClass::waitForConnectResult(int)'
     if (WiFi.waitForConnectResult(10000) != WL_CONNECTED) {
                                        ^
In file included from C:\Users\USER\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi\src/WiFi.h:32:0,
                 from C:\Users\USER\OneDrive\Desktop\portal_calendar\portal_calendar.ino:1:
C:\Users\USER\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi\src/WiFiSTA.h:56:13: note: candidate: uint8_t WiFiSTAClass::waitForConnectResult()
     uint8_t waitForConnectResult();
             ^
C:\Users\USER\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi\src/WiFiSTA.h:56:13: note:   candidate expects 0 arguments, 1 provided
Found multiple libarys for "WiFi.h"
 Used: C:\Users\USER\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\libraries\WiFi
 Not used: C:\Users\USER\OneDrive\Dokumente\arduino-1.8.13\libraries\WiFi
exit status 1
no matching function for call to 'WiFiClass::waitForConnectResult(int)'

It looks like you have version 1.0.6 of the esp32 boards package installed, this project needs 2.0+. Is there an update available for that package in your boards manager?

Turns out I had a old board list (url to json file) that was last updated to esp32 version 1.0.6.
By searching for a up-to-date list, replacing the url in the Arduino IDE settings and updating boards i've got the version 2.0.6.
Compiles now fine. Thanks wuspy


ImportError: DLL Load Failed While Importing QtCore

Followed the install instructions, although my main.py is in the /src directory, so CD /src then python -m main.py.
PS C:\Tools\Spotifyte\src> python -m main.py Traceback (most recent call last):   File "C:\Python310\lib\runpy.py", line 187, in _run_module_as_main     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)   File "C:\Python310\lib\runpy.py", line 110, in _get_module_details     __import__(pkg_name)   File "C:\Tools\Spotifyte\src\main.py", line 6, in <module>     from qfluentwidgets import (NavigationBar, NavigationItemPosition, MessageBox,   File "C:\Python310\lib\site-packages\qfluentwidgets\__init__.py", line 17, in <module>     from .components import *   File "C:\Python310\lib\site-packages\qfluentwidgets\components\__init__.py", line 1, in <module>     from .dialog_box import *   File "C:\Python310\lib\site-packages\qfluentwidgets\components\dialog_box\__init__.py", line 1, in <module>     from .color_dialog import ColorDialog   File "C:\Python310\lib\site-packages\qfluentwidgets\components\dialog_box\color_dialog.py", line 2, in <module>     from PySide6.QtCore import Qt, Signal, QPoint, QRegularExpression, QSize ImportError: DLL load failed while importing QtCore: The specified procedure could not be found.
Windows 11 fully updated, Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] on win32
What can I do to help troubleshoot?  I am really eager to try this out.

Hey @y0himba
This's a QT venv error and there's nothing wrong with the code. To solve this,

Create a new virtual environment (do not import existing packages from the old env)
Install all libraries freshly

This's a known issue in PyQt6/PySide6/PyQt5. As of now, they haven't released any fixes. Only creating new environments fix it. If you want help regarding creating environments,

Creation of Virtual Env (Src: Python)
How to Set Up a Virtual Environment in Python (Src: FreeCodeCamp)

Thanks
@rohankishore

Thank you so much! Working now.


mod: update go version to go1.20.8 and update deps

Please take a look.

:green_heart: Build Succeeded

Buildkite Build
Commit: 9b7f3f96fd6057a4017e411312500b3caad69e83


Chore: Typos fixed in files of doc/source/concepts/ folder

[X] I have added the tests to cover my changes.
[X] I have updated the documentation and CHANGELOG accordingly.
[X] I have read the CONTRIBUTING document.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@SusheelThapa please close this one as well.


In-line help in add project tab can not close by clicking other places.

In-line help in add project tab can not close by clicking other places.

Fixed with PR #1223


Avoid reserved Codepoints U+FE00...U+FE0F and U+FE20...U+FE2F

The following codepoints are reserved and should be avoided:

U+FE00...U+FE0F are Variation Selectors - see https://codepoints.net/variation_selectors
U+FE20...U+FE2F are Combining Half Marks - see https://codepoints.net/combining_half_marks

In my project I have 3380 SVGs icons. I am not explicitly setting codepoints. Codepoints U+F101 (DEFAULT_START_CODEPOINT) ... U+FE3F are automatically allocated by fantasticon. However, these codepoints collide with the reserved codepoints and create rendering issues.
Here's a repro makeicons.sh script:
#!/bin/bash

# Generate 3380 SVG icons (solid triangle)
mkdir -p icons
((DEFAULT_START_CODEPOINT=16#f101))
for ((i=0;i<3380;i++))
do
  echo '<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path d="M16 8L24 24 L 8 24z"/></svg>' > $(printf 'icons/icon-%x.svg' $((DEFAULT_START_CODEPOINT+i)))
  if ((i % 100 == 0)); then echo "$((i+1)) icons created ..."; fi
done
echo "$i total icons created"

# Generate my-icons.js
cat > my-icons.js <<EOF
module.exports = {
  name: 'my-icons',
  fontHeight: 500,
  normalize: true,
  inputDir: './icons',
  outputDir: './dist',
  fontTypes: ['ttf'],
  assetTypes: ['css', 'json', 'html'],
  formatOptions: {
    json: {
      indent: 2
    }
  },
  getIconId: ({
    basename,
    relativeDirPath,
    absoluteFilePath,
    relativeFilePath,
    index
  }) => {
    return basename;
  }
};
EOF

# Generate my-icons.ttf
mkdir -p dist
fantasticon --config ./my-icons.js

dist/my-icons.html below shows that icons in U+FE00...U+FE0F are blank and icons in the U+FE20...U+FE2F` are not centered but left jusitifed.

If https://github.com/tancredi/fantasticon/issues/527 was implemented, it could help dodge the problematic codepoints.

Workaround, insert into the codepoints dictionary object reservation for the affected codepoints, e.g.
module.exports = {
  name: 'my-icons',
  fontHeight: 500,
  normalize: true,
  inputDir: './icons',
  outputDir: './dist',
  fontTypes: ['ttf'],
  assetTypes: ['css', 'json', 'html'],
  formatOptions: {
    json: {
      indent: 2
    }
  },
  codepoints: {
    // U+FE00...U+FE0F Variation Selectors - see https://codepoints.net/variation_selectors
    'variable-selector-00': 0xfe00,
    'variable-selector-01': 0xfe01,
    'variable-selector-02': 0xfe02,
    'variable-selector-03': 0xfe03,
    'variable-selector-04': 0xfe04,
    'variable-selector-05': 0xfe05,
    'variable-selector-06': 0xfe06,
    'variable-selector-07': 0xfe07,
    'variable-selector-08': 0xfe08,
    'variable-selector-09': 0xfe09,
    'variable-selector-0a': 0xfe0a,
    'variable-selector-0b': 0xfe0b,
    'variable-selector-0c': 0xfe0c,
    'variable-selector-0d': 0xfe0d,
    'variable-selector-0e': 0xfe0e,
    'variable-selector-0f': 0xfe0f,

    // U+FE00...U+FE0F Variation Selectors - see https://codepoints.net/variation_selectors
    'combining-half-marks-00': 0xfe20,
    'combining-half-marks-01': 0xfe21,
    'combining-half-marks-02': 0xfe22,
    'combining-half-marks-03': 0xfe23,
    'combining-half-marks-04': 0xfe24,
    'combining-half-marks-05': 0xfe25,
    'combining-half-marks-06': 0xfe26,
    'combining-half-marks-07': 0xfe27,
    'combining-half-marks-08': 0xfe28,
    'combining-half-marks-09': 0xfe29,
    'combining-half-marks-0a': 0xfe2a,
    'combining-half-marks-0b': 0xfe2b,
    'combining-half-marks-0c': 0xfe2c,
    'combining-half-marks-0d': 0xfe2d,
    'combining-half-marks-0e': 0xfe2e,
    'combining-half-marks-0f': 0xfe2f,
  },
  getIconId: ({
    basename,
    relativeDirPath,
    absoluteFilePath,
    relativeFilePath,
    index
  }) => {
    return basename;
  }
};


How to reduce the installation package after electron integrates @journeyapps/sqlcipher ?

Installation package after electron integrates @journeyapps/sqlcipher is too large（102MB）.  Package after electron integrates sqlite3 just 60+MB. How to reduce package after integrates @journeyapps/sqlcipher ?
electron version: 10.1.5
@journeyapps/sqlcipher version: 5.2.0

you could delete some folders and files after your compile,
electron only need  these files ,maybe less?
├── binding.gyp
├── lib
│   ├── binding
│   │   └── napi-v6-darwin-x64
│   │       └── node_sqlite3.node
│   ├── index.js
│   ├── sqlite3-binding.js
│   ├── sqlite3.js
│   └── trace.js
├── package-lock.json
├── package.json
├── sqlite3.js

When using a tool like electron-builder, you can configure it to exclude files. I'm using something like this:
files:
  - '**/*'
  - '!src/**'
  - '!**/*.{tlog,lastbuildstate,pdb,obj,lib,map,exp,vcxproj,vcxproj.filters}'
  - '!**/sqlcipher/deps'
  - '!**/binding/node-*'

The package size should be reduced in v5.3.0.

The package size should be reduced in v5.3.0.

Thank you very much


Updated AutoAdmittedUsers description in New/Set-CsTeamsMeetingPolicy.md

Related to https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/7665.

Docs Build status updates of commit 81db077:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 9065bdc:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@tseward

@tiburd Please merge this PR. Thanks

Docs Build status updates of commit 82df341:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 2f0960c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit aca8007:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

skype/skype-ps/skype/New-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
:white_check_mark:Succeeded
View (skype-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Installer installs unnecessary (other platform files)

perhaps a bug in build process (?)
npm run electron, -> generate installer, run installer ->
c:\Program Files\Video Hub App 3
size: 792 MB (830,843,345 bytes)
(resources folder contain some node modules for all platforms, I believe it should be there, it can all be 4x smaller)

I'm not a fan of the size of the installed app and would love for it to be smaller.
I'm unsure about how to go about decreasing the cruft; any suggestions welcome 👍
If you happen to notice some specific examples where removing just 1 folder we get a good size decrease, perhaps we could target those specifically.

(resources folder contain some node modules for all platforms, I believe it should be there, it can all be 4x smaller)
sorry typo, my bad, it should read: I believe it should NOT be there

If you remove node modules for those platforms not in use (I just left win32-x64 in my case), you'll save 500MB on installation
example:
c:\Program Files\Video Hub App 3\resources\app.asar.unpacked\node_modules@ffmpeg-installer\linux-ia32
c:\Program Files\Video Hub App 3\resources\app.asar.unpacked\node_modules@ffprobe-installer\linux-ia32\

In fact, I just checked V2 installation did not have those other platform modules in it, it appeared there recently, something changed in the build process

Great catch! I'll definitely see what I can do.
I know we specifically treat the ffmpeg and ffprobe in a special way:
https://github.com/whyboris/Video-Hub-App/blob/main/node/main-extract.ts#L28
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path.replace('app.asar', 'app.asar.unpacked');

And the dependency should only install the OS-relevant files 🤔 ...
As far as I can tell (going by vague memory), I think the npm run electron will simply copy the node_modules as part of the compilation/build process. So in theory I could delete some specific folders before-hand to reduce size 🤔
This Issue will be on the check-list before releasing 3.0.0 ✅

@lighthunt -- I'm unsure what happened on your end, but on mine everything seems fine 🤔
Consider deleting node_modules and running npm install again 🤔
My current node_modules is about 800MB, when I build the installer, it comes out to around 100MB, and the installed directory is under 300MB 🤷
I'll keep this issue in mind - to check file sizes before releasing 3.0.0 but I'm closing this issue at the moment. Please feel free to comment 👍

This is so weird ;-) Those 10 day ago, I've been building both V2 and main in the same git folder with 'npm clean-install' in between checkouts ... and then run the build few times (checkout main) to make sure I do it right.
Few days ago when building and testing translations the size was again under 300M installed, so I have just assumed you or someone has fixed the issue already... weird.
(I don't know, I may have cleaned some more caches (npm in home ... but I think I did this even before V2 build ... ), or who know what happened, I think node_modules were over 1.1GB those days .. ) If it was a false alarm, sorry about that

I'll keep this issue in mind - to check file sizes before releasing 3.0.0
Good idea on the sanity check before release.  (may be be automatic in the build ... size <300 ? with warning, you may always update the number later )

Great idea on an automatic check! I'm unsure how to set something like that up without doing some research.
Share your thoughts if you have something in mind already.
My guess is it's not too-hard. Just checking the /release folder for size is enough. Though without having node run a specially-written script I don't yet know how to do it 🤔
While this won't give a warning, maybe running du -hcs ./release/ would be enough -- a simple reminder about the folder size? 🤔

Thanks @lighthunt -- I implemented a first draft of this. If you happen to know a better way, please let me know 🤝
#535


Saved model and then looked at the output of the same batch after loading, got different results!

Description
(Brief description of the problem in no more than 2 sentences.)
Get different results from model which is saved and from loaded model.
I tried differen methods for saving and loading, all showed wrong output.
           model.forward(batch)
           out1 = model.get_outputs()[0].asnumpy()
           print('saving')
           arg, aux = model.get_params()
           all_layers = model.symbol.get_internals()
          _sym = all_layers['fc1_output']
          _arg = {}
          for k in arg:
              _arg[k] = arg[k]
          model.save_checkpoint(prefix, 1, save_optimizer_states=True)
          sym, arg_params, aux_params = mx.model.load_checkpoint(os.path.join(args.pretrained, "model"), args.pretrained_epoch)
          model.set_params(arg_params, aux_params)
          model.forward(batch)
          arg_ckpt,_ = model.get_params()
          out2 = model.get_outputs()[0].asnumpy()
          print(f"diff_output:{np.abs(out1-out2)}")

@mxnet-label-bot add [question, training]

@KhurramPirov
It the code just pseudo code or it's complete code?
It looks you like you are running all batched data from data_iter for the initial forward,
But you only forward once for the last batch after you saved. If you are not processing the same input, the result will not be the same.

@KhurramPirov
It the code just pseudo code or it's complete code?
It looks you like you are running all batched data from data_iter for the initial forward,
But you only forward once for the last batch after you saved. If you are not processing the same input, the result will not be the same.
@frankfliu
It is a  sample of the test code. I am checking for each batch once and then look at the output, it is all in one loop with keyboard interruption. Not the last batch, sorry did not insert interruption and right indents.

Make sure you are saving and loading the model from the same path. I've tested the code below and it is working fine.
for batch in data_iter:
    model.forward(batch)
    out1 = model.get_outputs()[0]
    break
model.save_checkpoint('model', epoch = 0)
sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0)
model.set_params(arg_params, aux_params)
model.forward(batch)
out2 = model.get_outputs()[0]
print(f'diff_output:{nd.abs(out1 - out2)}')

Output is all zeros.

I checked the path, I do it inside batch_callback and model.fit, can it be
the case(seems no), but nevetheless.
пт, 12 июл. 2019 г. в 13:03, Rishik Mourya<EMAIL_ADDRESS>
Make sure you are saving and loading the model from the same path. I've
tested the code below and it is working fine.
for batch in data_iter:
model.forward(batch)
out1 = model.get_outputs()[0]
break
model.save_checkpoint('model', epoch = 0)
sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0)
model.set_params(arg_params, aux_params)
model.forward(batch)
out2 = model.get_outputs()[0]print(f'diff_output:{nd.abs(out1 - out2)}')
Output is all zeros.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/15504?email_source=notifications&email_token=AH2IALW3CR6MZP532CKQ5ALP7BJFXA5CNFSM4H7SKOHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZZKIOA#issuecomment-510829624,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH2IALTCG3VBAW5DDH2PFTLP7BJFXANCNFSM4H7SKOHA
.

--
Khurram Pirov
skype: matrixrextended
mob: +7-916-439-53-76

It doesn't work inside batch_callback.
пт, 12 июл. 2019 г. в 13:03, Rishik Mourya<EMAIL_ADDRESS>
Make sure you are saving and loading the model from the same path. I've
tested the code below and it is working fine.
for batch in data_iter:
model.forward(batch)
out1 = model.get_outputs()[0]
break
model.save_checkpoint('model', epoch = 0)
sym, arg_params, aux_params = mx.model.load_checkpoint('model', epoch = 0)
model.set_params(arg_params, aux_params)
model.forward(batch)
out2 = model.get_outputs()[0]print(f'diff_output:{nd.abs(out1 - out2)}')
Output is all zeros.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/15504?email_source=notifications&email_token=AH2IALW3CR6MZP532CKQ5ALP7BJFXA5CNFSM4H7SKOHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZZKIOA#issuecomment-510829624,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH2IALTCG3VBAW5DDH2PFTLP7BJFXANCNFSM4H7SKOHA
.

--
Khurram Pirov
skype: matrixrextended
mob: +7-916-439-53-76


Add travis

Runs tests, builds example, and verifies formatting.
Hopefully, we will also be able to run the dart analyzer in the future, but that's currently blocked by https://github.com/flutter/flutter/issues/10015.
Fixes https://github.com/flutter/flutter/issues/9918.

https://github.com/flutter/plugins/pull/46 and https://github.com/flutter/plugins/pull/45 have to be submitted first to make Travis happy.


The component picker does not always use the correct icons for components

The component API can now specify a number of icons for a component. But it seems like at least in the Preferred component picker, these don't always show up:
(eg the typography ones should have Text icons but do not - but the Row and Column ones do have the correct icon).

Also for the advanced picker:

The problem is related to the specific component annotations in the project. The problematic preferred child components of Column are registered like so
{
    component: 'Title',
    moduleName: '/app/components/Components',
    variants: AntdTitleVariants,
  },
  {
    component: 'Paragraph',
    moduleName: '/app/components/Components',
    variants: AntdParagraphVariants,
  },
  {
    component: 'Text',
    moduleName: '/app/components/Components',
    variants: AntdTextVariants,
  },

The component picker uses the existing component registrations to look up the icons for the preferred children. However, we don't have corresponding component registrations for Title, Paragraph or Text, and since we can't find an icon for those, we fall back to the component icon.
Title, Paragraph and Text use the components from antd, which we wrapped into TypographyTitle, TypographyParagraph and TypographyText respectively, and we have component registrations for those components, so updating Title, Paragraph and Text to Title, Paragraph and Text use the components from antd, which we wrapped into TypographyTitle, TypographyParagraph and TypographyText in the preferred children section fixes the icons problem.


bintray.com will be deprecated and libraries are unreachable.

Static library URL documented in the readme ('https://hyperledger-org.bintray.com/aries/') is unreachable and returns forbidden.

Link is working again.

The issue is still persisting, could you please provide an alternate link for downloading static libraries required for iOS and Android


bot doesnot run File "pokecli.py", line 35, in

File "pokecli.py", line 35, in
import six
ImportError: No module named six
Something went wrong and the bot needed to be restarted. Please investigate the cause.
Waiting for 46 seconds, press a key to continue ...

install the module six ; https://pypi.org/project/six/


Add 'alert' command support

For use as part of a build process, when the build script is watching files and running in background in some console, it would be useful to be able to send an alert message if something goes wrong in the build.
With the build console in the background (personally I have an editor on my first monitor and browser on the other, with no place left for console) the user has no way of knowing the build is failing and therefore why the reload is not happening. An alert fixes that with a very visible notification whenever I make a build-failing error.
I tried to get that into a pull request here: #86 . I'm not very proficient in OSS and Github though, so feel free to correct me in any way.

@veproza hey, thanks for this. I think you did a pretty great job with this commit. Your refactoring didn't break the tests and this is one part of the protocol I've been meaning to implement anyway. I'll see if the test suite for this needs some updating and then I'll merge this in.

Fixed by #86


chore: most recent code review changes

upstream changes from most recent code review.

Added Ian's suggestions.

Removed stale type default of boolean in API docs, which has already been applied upstream.

I did a file compare against matching upstream files for index.js, utils.js, and README.md. Just picked up one non-whitespace difference and added a suggestion on the PR to catch up:

https://github.com/nodejs/node/pull/42675/files#r862629844


Time management utilities

Problem
For some use cases, particularly when Smart Contract are based on time or blocks' height conditions, it is useful to conveniently "wait" for a given timestamp or block.
Moreover, when things run on a LocalNet in DevMode it is possible to "fast forward" both time and blocks.
It would be very useful if such methods could have a nice UX (a less boilerplate code).
Solution
Add to AlgorandClient and LocalNet (or some better place) convenient methods to fetch time/blocks, wait for time/blocks conditions, jump to time/block (in DevMode).
Proposal
Add to the AlgorandClient the following methods:

getLastRound: returns the last committed round from Algod
getLatestTimestamp: returns the latest UNIX timestamp committed with the last round from Algod
waitUntilRound: is a "sleep" conditions that alts the execution until a given round is committed (it could leverage getLastRound)
waitUntilTimestamp: is a "sleep" conditions that alts the execution until a given UNIX timestamp is committed (it could leverage getLatestTimestamp)

In order to speed up tests when things run LocalNet (in DevMode), the following methods can be added (not in AlgorandClient, since it's DevMode specific):

blockWarp: generate blocks to jump directly to a given blocks' height (blocks are instantly generated as soon as a transaction is committed in DevMode)
timeWarp: set a proper time offset to commit a block with the desired UNIX timestamp.

Specifically for timeWarp I'd suggest the following implementation to jump directly to toTimestamp:

Get the latestTimestamp with getLatestTimestamp
Set the timestamp offset with the Algod method to: toTimestamp - latestTimestamp
Generate a block to commit the desired offset (this could use blockWarp with a single block generation)
Reset the timestamp offset to 0

Pros and Cons
Pros: speed up test, less boilerplate code, nicer UX.
Cons: none.
Dependencies
Algorand SDK and LocalNet in DevMode

My suggestion for the design of this would be a new NetworkManager class is added (similar to AppManager, AssetManager, etc.) with the following signatures from AlgorandClient:

algorand.network.getLastRound()
algorand.network.getLatestTimestamp()
algorand.network.waitUntilRound()
algorand.network.waitUntilTimestamp()
algorand.network.localNet.blockWarp()
algorand.network.localNet.timeWarp()

We could also proxy or maybe even move some of the methods on ClientManager like isTestNet() and potentially proxy some of the localnet specific things in other classes like localNetDispenser on AccountManager as one of a number of examples.

All of them should wrap the underlying algosdk types, similar to getById in AssetManager and getInformation in AccountManager and do the similar treatment like convert round numbers to bigint etc. to follow AlgorandClient conventions.


MINOR: fix instable KafkaStreamsTest

set streams state.dir to test-dir (default /tmp is not reliable)

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Call for review @dguy @bbejeck @guozhangwang

@mjsax could you update the title to point to the newly created ticket created by @tedyu ?

@mjsax
Thanks for the fix.

@guozhangwang Done.

LGTM. Merged to trunk and 1.0


Delete unresolve conflict text

example of unresolve conflict existing :  https://id.reactjs.org/docs/getting-started.html#practical-tutorial


[BUG] System.ArgumentException: An item with the same key has already been added

When trying to install latest forge 1.20.1 got this error
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentException: An item with the same key has already been added. Key: Name
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at CmlLib.Utils.JarFile.GetManifest()
   at CmlLib.Core.Installer.Forge.Installers.ForgeInstaller.startProcessor(JToken processor, Dictionary`2 mapData)
   at CmlLib.Core.Installer.Forge.Installers.ForgeInstaller.StartProcessors(JArray processors, Dictionary`2 mapData)
   at CmlLib.Core.Installer.Forge.Installers.ForgeInstaller.MapAndStartProcessors(JObject installProfile, String installerDir)
   at CmlLib.Core.Installer.Forge.Installers.FNewest.Install(String installerDir)
   at CmlLib.Core.Installer.Forge.Installers.ForgeInstaller.Install(ForgeInstallOptions options)
   at CmlLib.Core.Installer.Forge.MForgeNoAds.Install(ForgeVersion forgeVersion, ForgeInstallOptions options, Boolean forceUpdate) in C:\Users\bronemishka\Desktop\git\minecraft-modpack-loader\ModPackLoader\Core\Utils\MForgeNoAds.cs:line 93
   at ModPackLoader.Core.UltimateInstaller.InstallMinecraft() in C:\Users\bronemishka\Desktop\git\minecraft-modpack-loader\ModPackLoader\Core\UltimateInstaller.cs:line 63
   at ModPackLoader.ViewModels.LoaderViewModel.InstallModpack() in C:\Users\bronemishka\Desktop\git\minecraft-modpack-loader\ModPackLoader\ViewModels\LoaderViewModel.cs:line 43
   at CommunityToolkit.Mvvm.Input.AsyncRelayCommand.AwaitAndThrowIfFailed(Task executionTask)
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_0(Object state)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---

Environment:

OS: win 11
CmlLib.Core Version:3.3.10
.NET Core 7

Additional context
Tried to investigate the problem and found that the reason could be with the IForgeInstallerVersionMapper, so probably somewhere there is the place fix should be done. From my side I just modified MForge to not show the ad.

update CmlLib.Core.Installer.Forge 0.0.3


mkmf.rb can't find header files for ruby

# gem install redis-dump
Fetching: yajl-ruby-1.3.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing redis-dump:
        ERROR: Failed to build gem native extension.

    current directory: /var/lib/gems/2.3.0/gems/yajl-ruby-1.3.1/ext/yajl
/usr/bin/ruby2.3 -r ./siteconf20180302-17663-4bv1w.rb extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/yajl-ruby-1.3.1 for inspection.
Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/yajl-ruby-1.3.1/gem_make.out

I had to install the following packages to make it work:
apt install ruby-dev make gcc


Dilato: Core: User could drop & drag a no permission file to another folder successfully without any error messages

Steps:

Launch Brackets
Open a folder which contain a no read and write permission file
Drag and drop this no read and write permission file to another folder

Result: User could drop & drag a no permission file to another folder successfully without any error messages
Expect: There should throw an error message, please see the screenshot for the expect result.

This issue reproduce on Windows, Mac, Linux(Ununtu & Debian) systems
Test build:
Linux: 1.13.17668
Windows & Mac: 1.13.17666-1019

looks similar to https://github.com/adobe/brackets/issues/14269


opni enhancement proposal template and vale style guide

can you also add example commends to run to apply vale so everyone can just follow that

can you also add example commends to run to apply vale so everyone can just follow that

added https://github.com/rancher/opni/blob/processes/vale.md


[WIP] feature: and pouch copy in client and daemon side

Ⅰ. Describe what this PR did

copy file from container
copy file to container

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

I only add some initial comment. For this valuable part, I think we need to spend more time and patience to make this move on. @knightXun
We will definitely help to make it work. Let us carry on. Thanks a lot.

Any update on this? @knightXun

Codecov Report

Merging #2182 into master will decrease coverage by 0.49%.
The diff coverage is 1.73%.

@@            Coverage Diff            @@
##           master    #2182     +/-   ##
=========================================
- Coverage   64.78%   64.28%   -0.5%
=========================================
  Files         208      208
  Lines       16727    16900    +173
=========================================
+ Hits        10837    10865     +28
- Misses       4532     4687    +155
+ Partials     1358     1348     -10

Flag
Coverage Δ

#criv1alpha1test
32.73% <1.73%> (-0.09%)
:arrow_down:

#criv1alpha2test
33.31% <1.73%> (-0.16%)
:arrow_down:

#integrationtest
39.59% <1.73%> (-0.39%)
:arrow_down:

#unittest
23.57% <0%> (-0.27%)
:arrow_down:

Impacted Files
Coverage Δ

daemon/mgr/container.go
51.38% <0%> (-4.77%)
:arrow_down:

daemon/mgr/container_state.go
67.21% <0%> (-20.03%)
:arrow_down:

apis/server/container_bridge.go
69.28% <0%> (-9.24%)
:arrow_down:

apis/server/router.go
92% <100%> (+0.16%)
:arrow_up:

daemon/logger/jsonfile/utils.go
71.54% <0%> (-1.63%)
:arrow_down:

cri/v1alpha2/cri.go
62.96% <0%> (+0.32%)
:arrow_up:

cri/v1alpha1/cri.go
62.56% <0%> (+0.67%)
:arrow_up:

ctrd/container.go
53.82% <0%> (+1.43%)
:arrow_up:

apis/server/utils.go
66.66% <0%> (+4.76%)
:arrow_up:

... and 1 more

now working on e2e test

ping @allencloud

I am afraid that we need to add more details in swagger.yml to define the API side. For adding new APIs, we should add definition in swagger.yml, like https://github.com/alibaba/pouch/blob/master/apis/README.md .
So please help to add that as well. I think we are almost there. Keep going. Thanks a lot. @knightXun

Any update on this? @knightXun
Do you still work on this? 🐸 🦊


Replaced String[] with HashMap<String, String> for person data structure

@okkhoy I missed tutorial 1 but I did these commits before the tutorial and was planning to show them. Will this be acceptable for the 1kLoC LO?
Thanks!

the requirement was to make some enhancement to 1kLoC codebase.
This is a part of a different LO. So I cannot accept it in lieu of the compulsory LO.

BTW, you should create pull requests to nus-cs2103-AY1718S2 repo rather than the nus-cs2103-AY1718S1


On-Screen Touchbar frozen after sleep/close lid - reopen lid

I've been using the onscreen touchbar and it works as expected. For some reason or another, if I lock the screen, close the lid/reopen, or just awaken from sleep, the on screen bar is still there as expected but it is frozen to the last operation it had. The only solution is to exit touchBarServer application and restart.

I'm running into the same problem

We'd like to make a fix for this, but we haven't been able to reproduce this ourselves.
Is there any additional information you can provide? Does it happen every time? Do you have to wait a bit after closing the lid, or can you reopen the lid instantly? Maybe you can provide the output of running this command in the Terminal app:
pmset -g

Or perhaps anything else that you can think of that's specific to your situation and might explain why you're having this issue and we're not. As soon as we're able to reproduce the issue, we can look for a fix or workaround.

Ok so here is what I'm seeing. When I have the MacBook Pro connected to an external monitor as a second screen... and i close the lid and reopen it after lets say 10 seconds of it being closed, the issue doesn't happen. Touch Bar onscreen works! When I disconnect the monitor and simple use the MacBook Pro (Late 2013) and I do the same closing of lid and reopening, then the touchbar is frozen and will not work until i quit the server and restart.
The output:
Currently in use:
standbydelay         10800
standby              1
womp                 1
halfdim              1
hibernatefile        /var/vm/sleepimage
powernap             1
gpuswitch            2
networkoversleep     0
disksleep            10
sleep                1
autopoweroffdelay    28800
hibernatemode        3
autopoweroff         1
ttyskeepawake        1
displaysleep         10
acwake               0
lidwake              1